Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various eval-variables fixes and sys-ocaml-system #5829

Merged
merged 8 commits into from Mar 29, 2024

Conversation

dra27
Copy link
Member

@dra27 dra27 commented Feb 11, 2024

#3900 introduced sys-ocaml-cc, sys-ocaml-libc and sys-ocaml-arch which followed from some very early demonstrations I gave of the Windows opam packaging proposals at MSR Cambridge in 2015.

These variables have been quietly sat on new opam 2.1 installations since 2021. This PR addresses three issues:

  • If you upgrade from a 2.0 root to a 2.1, you don't get them (you can see this in ocaml/opam Docker Hub images).
  • At present, they use Unix sh commands, and they don't evaluate at all with the internal Cygwin installation on Windows
  • A bug (which will be fixed in Cygwin 3.5.1) means that if the sh commands are used with an OCaml 5.1.1 compiler without the runtime DLLs (which is VERY easy to do), opam hangs on a popup.

I've therefore:

  • Attempted some initial refactoring to ensure that the format upgrade re-applies eval-variables. As a side-note, I think the presence of eval-variables in opamrc is a mistake - they belong to the repository, not to the configuration, but we are where we are and that's something we can look at improving in opam 3.0
  • Allowed the eval-variables to use the internal Cygwin installation (this is just a missing case)
  • Further tweaked the opam 2.2 versions of these variables not to require Posix tools on Windows. A small side-effect is that because I use -config-var, system compilers must be 4.08+ on Windows, but I think this will be fine (it's not like we have old distributions to worry about here - system compilers will have been set-up on purpose to avoid recompilation).

Finally, while doing the Windows compiler packaging updates, the variables added in #3900 are working beautifully (as expected), but I propose adding sys-ocaml-system so that at some point similar support can be extended to Unix system compilers. sys-ocaml-system is simply the value of system from ocamlc -config and permits precise detection of the system compiler, especially in a mixed-architecture environment. It's in the same kind of area as #5587 - trying to probe the compiler itself, rather than the present very weak assumption of global variables correlating with all runnable compilers (remembering that an arm64 Windows machine is capable of running 4 different architectures...)

@dra27 dra27 added this to For beta2 in Opam 2.2.0 Feb 11, 2024
@dra27 dra27 added this to the 2.2.0~beta2 milestone Feb 11, 2024
@kit-ty-kate kit-ty-kate marked this pull request as ready for review March 4, 2024 13:21
@rjbou rjbou self-requested a review March 4, 2024 13:22
@dra27
Copy link
Member Author

dra27 commented Mar 6, 2024

Thanks for sorting out the tests, @kit-ty-kate. I’m happy with the filtering out, unless it would be better just to sed off the values (i.e. be able to see that the variables were defined, but not worry about their differing values), @rjbou?

I think the only remaining thing is to put the various definitions into OpamEnv and reference them from there in both OpamFormatUpgrade and OpamInitDefaults.

@kit-ty-kate
Copy link
Member

I think the only remaining thing is to put the various definitions into OpamEnv and reference them from there in both OpamFormatUpgrade and OpamInitDefaults.

How does 01576d0 sound?

Comment on lines -841 to +840
let env = raw_env () in
let cyg_env ~env ~cygbin ~git_location =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to not have env as an optional argument ? It permit to lighten the call, and let the ability to not decide when dev don't know. All non-cygwin specific calls uses anyway OpamStd.Env.raw_env

Suggested change
let env = raw_env () in
let cyg_env ~env ~cygbin ~git_location =
let cyg_env ?env ~cygbin ~git_location =
let env = Option.Op.(env +! raw_env ()) in

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My rationale was that the dev should know 🙂 The exact environment in use at any given point can be quite tricky to determine, so I was figuring that being explicit was potentially better. I don't overly mind, though.

src/state/opamEnv.ml Outdated Show resolved Hide resolved
src/state/opamEnv.ml Outdated Show resolved Hide resolved
src/state/opamEnv.ml Show resolved Hide resolved
tests/reftests/opamroot-versions.test Show resolved Hide resolved
@rjbou
Copy link
Collaborator

rjbou commented Mar 21, 2024

I've pushed all my review changes, separated in tiny commits. Once no more changes are needed, I'll do a final commit history cleaning.

Copy link
Member Author

@dra27 dra27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests look good too, thanks!

Comment on lines 18 to 21
| Some cygbin ->
OpamStd.Env.cyg_env ~cygbin
~git_location:OpamCoreConfig.(!r.git_location) ()
| None -> OpamStd.Env.raw_env ()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind, but the thing I did like before is that it tried to make it clearer that we are always looking at OpamStd.Env.raw_env (), it's just whether it's had cygbin/git_location added to it.

Comment on lines -841 to +840
let env = raw_env () in
let cyg_env ~env ~cygbin ~git_location =
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My rationale was that the dev should know 🙂 The exact environment in use at any given point can be quite tricky to determine, so I was figuring that being explicit was potentially better. I don't overly mind, though.

dra27 and others added 8 commits March 27, 2024 18:27
This has the benefit of making them work without requiring Cygwin, but
the tiny caveat that they won't work for OCaml < 4.08 and it means that
Windows has a different default opamrc since eval-variables can't be
filtered.

Neither of these things feels like a problem.

Co-authored-by: Raja Boujbel <raja.boujbel@ocamlpro.com>
…efaults et state/OpamFormatUpgrade

Co-authored-by: Kate <kit-ty-kate@outlook.com>
eval-variables are in opamrc from the 2.x de-OCaml-ing of opam, but
they're conceptually in the wrong place (they should be declared in the
repository where they can be synchronised and upgraded).

This is a problem for the new variables added in 2.1, because they don't
get added to the config of existing roots.

This alters the format upgrade to update pre-existing variables and add
any new ones.

Obviously, if the user had initialised with a different opamrc then
we're overwriting that, but that is the conceptual weakness of the field
being in opamrc.
Forward planning for the rest of the base-system- packages
@rjbou
Copy link
Collaborator

rjbou commented Mar 28, 2024

My rationale was that the dev should know 🙂 The exact environment in use at any given point can be quite tricky to determine, so I was figuring that being explicit was potentially better. I don't overly mind, though.

Fair enough! I kept the env argument mandatory and added cyg_env documentation to advertise using OpamStd.Env.raw_env "by default".

I've updated the PR:

  • Reworked history to clean it for better readability
  • Added more information in master changes
  • Changed OpamEnv.sys_ocaml_eval_variables code to ensure that unix and win variables are defined.

@dra27
Copy link
Member Author

dra27 commented Mar 28, 2024

It's fine thing to have a single PR number with all three of us as authors against it 🙂

Approving the bits I didn't write, thank you!

@kit-ty-kate
Copy link
Member

Thanks!

@kit-ty-kate kit-ty-kate merged commit 239945f into ocaml:master Mar 29, 2024
29 checks passed
Opam 2.2.0 automation moved this from For beta2 to Done Mar 29, 2024
@dra27 dra27 deleted the eval-variables branch May 3, 2024 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Opam 2.2.0
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

3 participants