solver: use penalties for variant values

[alalazo]

fixes #51112

On develop nodes in a DAG get a penalty of 1 when a variant value is not the default. This might lead to cases of non-determinism, due to having multiple optimal solutions.

Here we give different penalties to different non-default values, thereby reducing the non-determinism in concretization.

[alalazo]

This bugfix comes with a slight regression:

• Spack: 1.2.0.dev0 (e62ca3b)
• Builtin repo: spack/spack-packages@127fe0c
• Python: 3.12.3
• Platform: linux-ubuntu24.04-broadwell

radiuss.develop.csv
radiuss.pr.csv

[haampie]

Last comment here before Christmas break:

I would still try to avoid the sorted call on the values. Yes, sorted leads to determinism, but since Spack has no clue what the values represent, it is effectively the same as randomization: if the default fabric in libfabric does not work, you get the next one in alphabetical order. If the C standard C17 does not work, you get C99 because it comes next in alphabetical order, and if that doesn't, you get C11. That makes no sense.

However, apparently we do not keep track of the order as defined in package.py but throw values into sets, meaning that the scope of this PR becomes larger.

I think that's worth it, since merging this with sorted and fixing it later means there's an unnecessary concretizer change.

For what it's worth, if someone else wants to review this PR in the meantime and has different opinions, that's fine. I'll catch up early January.