environment.py: concretize together by default

[haampie]

We have somewhat conflicting defaults, namely view: true and
concretization: separately.

This leads to post-install errors when merging different flavors of the
same package into the default view.

It makes more sense to either disable views or do consistent
concretization by default.

Our docs hint that concretization: separately is aimed at system
administrators, which is likely a small percentage of spack users.
By now many people use environments to just build stuff, given that
repeated concretization is slow, spack install x doesn't even show
what gets built, etc, all better when using environments. And for those
users consistent concretization makes more sense.

[tldahlgren]

... spack install x doesn't even show what gets built, etc,...

That statement surprises me. What is the context where you don't see Installing, Successfully installed, and [+] messages for packages that get built?

[scheibelp]

This reasoning might be poorly explained:

• We've since added stacks to support this (while that isn't specifically a problem this PR needs to address, this docs section should probably point to the section on stacks for this specific use case)
• Admins might not be the only people who benefit: there may be users who want to install separate tools in a single environment, and it may be that those tools cannot be concretized together (i.e. if each root provides a different top-level executable, but each top-level executable might depend on mutually exclusive library configurations)

Do you think it might be worth adding an option similar to --with-view? I think it would be OK to switch the default but IMO we should provide an option so that users can control this from the command line.

[scheibelp]

This reasoning might be poorly explained:

Sorry to clarify: I mean that the documentation, as originally written, is missing details.

[scheibelp]

I think it should be possible to control this default with a new option --concretize-separately (which is false by default - as desired in this PR); I think this would make it easier to maintain scripts which generate environments (when users want to concretize separately).

It also occurred to me we might want to add a temporary warning message to Environment.concretize() when doing concretize_together (only when it fails): right now the concretization: choice isn't explicitly dumped into the config, so users who were depending on that behavior will see errors (but wouldn't know why that started happening).

[haampie]

I don't think we should introduce tons of new features and warnings, that's just overwhelming and the opposite of reducing the steep learning curve for newcomers.

Alternatively and much simpler, we can make spack env create and spack env activate --temp set concretization: together explicitly by default, yet keep separately as a default when loading existing spack.yaml files that do not have concretization: ... set, to be backwards compatible.

[haampie]

@tamara somehow missed your message. But I meant to say: it does not print an overview ahead of time like with a spack add ..., spack concretize, spack install workflow, where the concretize step shows you what is already installed and what should still be installed. Without environments, the alternative is spac spec -I ..., spack install ..., but that has the downside of waiting for concretization twice. So, I think most people would want to use environments all the time, because you concretize just once, and see what gets installed ahead of time.

[scheibelp]

Alternatively and much simpler, we can make spack env create and spack env activate --temp set concretization: together explicitly by default, yet keep separately as a default when loading existing spack.yaml files that do not have concretization: ... set, to be backwards compatible.

That sounds good to me for most cases, but my concern was with users who might be running a nightly CI script which automatically generates the environment. I think this isn't critical since they can instantiate from a simple template like:

spack:
  specs: []
  concretization: separately

so I could see the benefit in not adding more options to the spack env command. However, that still creates a problem where their scripts will suddenly fail without a clear indication of why, so I still think that a debug-level message would be useful in Environment.concretize()

[tgamblin]

adding this to v0.18.0 -- we should do something with this but whether we switch to together or best-effort per #28941 is TBD

[haampie]

This can be 0.19 earliest since it's breaking, #30038 should be 0.18 so this is less breaking.

[haampie]

Closing this, will replace with a new PR that just changes the default of unify