Uberenv as submodule

[adrienbernede]

Uberenv is being used in more and more projects, especially for CI purpose.

However, the fact that is has to be added in-source is a brake to broader adoption and ease of use.

Adapting uberenv to be usable as a submodule will probably:

• break the current workflow
• require multiple small changes

Still, I think this should be done. If we do it in a separate branch at first, we may have trouble migrating to Spack environments afterward (duplicate the amount of work).

So I’m considering completing my work on environments first, even if some changes in Spack will likely imply adapting uberenv later on, hopefully drastically reducing its scope.

Input welcome

[adrienbernede]

@white238 @kennyweiss @cyrush, any thought about this?
I’ll create an issue to discuss the impact of Greg's PR (spack/spack#15256).

[white238]

I don't see how changing uberenv to a submodule makes it easier to upgrade. They are both essentially locked into a SHA. The only difference is a copy vs a "cd uberenvSubmodule; git checkout develop; git pull". The nice thing about having it directly in the repo is I can put the script (and only the script) exactly where I want it in the repo.

[adrienbernede]

Locked into a SHA?

Copying uberenv in a project implies losing the history, and any information about what version of Uberenv we are using and how to merge in new changes.

I agree that the submodule will apply only to the script (and possibly to some shared spack configurations, like compilers for machines) which limits the interest of the process.

OK, I’ll try to come back better prepared for this battle :)

[white238]

By locking into a SHA, I meant locking into a point in history. Yes i lose git history one way but as a user of uberenv I don't really care about the history that much. If i do care I can go look at the uberenv repo.

Also if i remember correctly uberenv requires the spack_configs and packages directories to at the same level as the script. This could obviously be solved with a simple change in uberenv.

[adrienbernede]

Yes, that’s part of the changes that would need to be done.

I’m going back and forth with this submodule thing. Clearly, some projects don’t like adding another project, in source, and I understand them.

"Uberenv is just a script" and for a long while it hasn’t be changing much. But during the last year, it has changed, and I have it in at least 6 projects.

[cyrush]

@adrienbernede you can use uberenv a a submodule in any project you like, but its a per project decision.

We discussed this for axom, we didn't see a strong benefit, as we would also have to change the directory structure and how it was called if it was a submodule.

[adrienbernede]

@cyrush, in fact, uberenv cannot be used as a submodule as-is, percisely because it requires changes in the directory structure.

For example, the fact that the Spack configuration is in the same directory as the uberenv script prevents from versioning the former in the parent project.

[kennyweiss]

Thanks @adrienbernede. As @cyrush mentioned, we discussed this at a recent axom meeting due to similar concerns and opted not to make the change.

My take is that, if there is a project that wants to use uberenv as a submodule and there are relatively simple changes that can be made to uberenv's structure to support that, we should make those changes. But we should do so in a way that minimizes changes to existing users (e.g. add some knobs that default to the current setting).

[kennyweiss]

@adrienbernede wrote:

So I’m considering completing my work on environments first, even if some changes in Spack will likely imply adapting uberenv later on, hopefully drastically reducing its scope.

This sounds like a good plan to me.

[white238]

A while back @becker33 added a feature at my request i never got around to trying out:

spack config --scope site add spack.yaml
// then some command to remove the user scope

This should remove any scope/config patching we do.

[goxberry]

I'll say that as a user of uberenv, I don't need the commit hash of uberenv.py to use it, but having its commit hash can help when diagnosing build errors that arise when calling uberenv -- specifically, excluding uberenv as a cause of any errors. (This task is something I'm doing now, hence this comment.)

For the purpose of debugging, you may want to suggest in your documentation that users run git hash-object on uberenv.py so they can get the commit hash in this repo corresponding to their project's version of uberenv.py. Suggested use of this command assumes no local modifications to uberenv.py, and additional flags may be needed if a user's local project is configured with things like filters that differ from the configuration of this upstream repo. If a user's project locally modifies uberenv.py (e.g., via something like running a formatter on all Python source in its repo), they may have to roll their own script to get the commit hash of the "closest" upstream version of uberenv.py.

If users can get the commit hash of the most relevant upstream version of uberenv.py, they can probably get at the other information they would otherwise want to get from vendoring in uberenv as a git submodule. Maybe they can't find that information as easily or conveniently, but they could find it nevertheless. Providing a means in your documentation of getting at that commit hash for 80ish% of use cases would require no other project restructuring on your part, but would probably help reduce your support burden.

[white238]

It would be nice when this is done, uberenv could allow multiple packages directory. Something like:

<repo>/someSubmodule/tpls/packages
<repo>/tpls/packages

Then they would just be copied into Spack in order.

[adrienbernede]

@white238 That does not seem straightforward.
Currently one can specify the package directory on command line (with the "arch" part).
Now with submodule, we need to be able to setup the root package directory (without the "arch" part).
Having the later while maintaining compatibility with the first is possible but causes confusion. So adding multiple layers, hmm, I don't know yet.

[adrienbernede]

Oops, I realize I was mixing up "package directory" and "spack config directory" :)
Seems much simpler now :)

[adrienbernede]

Uberenv now usable as submodule.