Fixes to reversal of environment updates #5935
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
OK, the fix I was proposing I didn't like because it didn't preserve unnecessarily escaped parts of existing variables. I then further found it regressed #4861 and consequently broke the msvs-detect package. I have a plan (which I'm currently implementing) - a failing test for the main issue (#5838) added and for the first bit of the fix, which I am certain is right. The change I think is required for #5838 without breaking #4861 I think will also address #5925 and #5926, so I've added tests showing the failure behaviour for them as well. |
CAML_LD_LIBRARY_PATH should appear in opam env --revert
MANPATH should appear in opam env --revert
As in this commit, there should be no output captured from the opam env --revert invocation.
NV_VARS5925_1 thru _4 should all be foo (i.e. without no colon)
- In `opam exec -- env` / `opam env`:
- NV_VARS_5926_L_2 should be `b::a` (the _L_1 case is correct)
- NV_VARS_5926_L_4 should be `:a:b` (the _L_3 case is correct)
- NV_VARS_5926_L_5 and _L_6 should be `b::a` (neither is correct)
- NV_VARS_5926_L_8 should be `:a:b` (the _L_7 case is correct)
- All four NV_VARS_5926_M_ all contain `a1:a2` instead of `a1::a2`
- NV_VARS_5926_T_2 should be `:b:a` (the _T_1 case is correct)
- NV_VARS_5926_T_4 should be `a::b` (the _T_3 case is correct)
- NV_VARS_5926_T_6 should be `:b:a` (the _T_5 case is correct)
- NV_VARS_5926_T_7 and _T_8 should be `a::b` (neither is correct)
- In `opam exec -- opam env --revert`:
- NV_VARS_5926_L_{2,4,6,8} revert to `a` instead of `:a`
- NV_VARS_5926_M_{1,2,3,4} should all revert to `a1::a2` where
- _M_1 and _M_3 only have a single colon (`a1:a2`)
- _M_2 and _M_4 have reversed the two components! (`a2:a1`)
- NV_VARS_5926_T_{2,4,6,8} revert to `a` instead of `a:`
- Note NV_VARS_5926_S_{1,2,3,4} revert to empty rather than `:`
(this is correct since both interpretations are equivalent and
opam can't know which it was)
Missing transformation of the argument, which affected reverting updates
with forward slashes (e.g. "%{lib}%/stublibs")
The cases of this match were the wrong way around. Note that this regresses ocaml#4861's quoted directory test.
Expose a generalised version of the Windows implementation of PATH-splitting as OpamStd.String.split_quoted.
Existing implementation was not correct w.r.t. separators which were themselves quoted. Use OpamStd.String.split_quoted instead. However, this causes superfluous quotes in variables to be removed. Note that this fixes the MANPATH issue of ocaml#5838 but the quoting-handling is still broken.
Ensure that if opam has internally synthesised an empty environment variable, then adding to it doesn't create a stray colon.
Change the internal representation of environment variables so that both the original quoted version and the parsed version are kept, along with the separator. This has the benefit that reconstituting the variable value does not require knowledge of the separator (which deals coherently with the parasitic case of two packages updating the same variable with a different separator: the value is almost certainly trashed, but it is at least trashed in a semantically coherent manner!)
When reverting [FOO += "value"], it is necessary to determine what to do if "value" is of the form "<value1><sep><value2>". Because FOO itself will have been split at <sep>, we will never match "<value1><sep><value2>" so it is necessary to split value at <sep> as well (this is the crux of ocaml#4861). However, the interaction of this and x-env-path-rewrite is incorrect. If x-env-path-rewrite has [FOO false] (the `norewrite case), then the value must be split according to the default interpretation of FOO (which is how it was added). If x-env-path-rewrite has a rewrite rule for FOO, then opam has already assumed that value refers to a single directory (the host/target and quoting transformation options all being this way). If x-env-path-rewrite does not have any rule (and does have have [FOO false]) then we are in essence in the backwards-compatible case, and must split value. In order to stop poor interaction between this and opam itself injecting Windows paths, opam's updates in OpamEnv (to MANPATH, etc.) are explicitly expanded so that in unzip_to they are treated as though they had an x-env-path-rewrite rule given (and are therefore not split). Combined with the previous fixes, this addresses ocaml#5838 without regressing ocaml#4861.
kit-ty-kate
reviewed
May 8, 2024
| let split_quoted path sep = | ||
| let length = String.length path in | ||
| let rec f acc index current last normal = | ||
| if index = length then |
There was a problem hiding this comment.
Suggested change
| if index = length then | |
| if Int.equal index length then |
| List.rev (if current <> "" then current::acc else acc) | ||
| else let c = path.[index] | ||
| and next = succ index in | ||
| if c = sep && normal || c = '"' then |
There was a problem hiding this comment.
Suggested change
| if c = sep && normal || c = '"' then | |
| if Char.equal c sep && normal || c = '"' then |
| let split_path_variable path sep = | ||
| let length = String.length path in | ||
| let rec f acc index current current_raw last normal = | ||
| if index = length then |
There was a problem hiding this comment.
Suggested change
| if index = length then | |
| if Int.equal index length then |
| else | ||
| let c = path.[index] | ||
| and next = succ index in | ||
| if c = sep && normal || c = '"' then |
There was a problem hiding this comment.
Suggested change
| if c = sep && normal || c = '"' then | |
| if Char.equal c sep && normal || c = '"' then |
| let current_raw = current_raw ^ segment in | ||
| if c = '"' then | ||
| f acc next current (current_raw ^ "\"") next (not normal) | ||
| else if next = length then (* path ends with a separator *) |
There was a problem hiding this comment.
Suggested change
| else if next = length then (* path ends with a separator *) | |
| else if Int.equal next length then (* path ends with a separator *) |
|
|
||
| let join_var ~(sepfmt:sep_path_format) var values = | ||
| let separator = | ||
| if value = String.make 1 sep then |
There was a problem hiding this comment.
Suggested change
| if value = String.make 1 sep then | |
| if String.equal value (String.make 1 sep) then |
| @@ -267,11 +328,11 @@ let apply_op_zip ~sepfmt op arg (rl1,l2 as zip) = | |||
| or empty lists is returned if the variable should be unset or has an unknown | |||
| previous value. *) | |||
| let reverse_env_update ~sepfmt var op arg cur_value = | |||
| let arg = transform_format ~sepfmt arg in | |||
| if String.equal arg "" && op <> Eq then None else | |||
| let (arg, _, _) = transform_format ~sepfmt var arg in | |||
There was a problem hiding this comment.
to make the code clearer, I feel like having a record instead of a triple sum type would make things clearer in the future and potentially better at avoiding any future bugs
| @@ -85,13 +137,14 @@ setenv: [ | |||
| [ RCTQ_ENVSET_ADD_WITH_COL += "a:/nother/gi;ven/path" ] | |||
| ] | |||
| build-env: [ | |||
| [ LC_ALL = "C" ] | |||
There was a problem hiding this comment.
I feel like we'd want this all the time, like globally. Couldn't we add this in run.ml instead?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes for #5838, #5925 and #5926. This PR has become quite an epic rabbithole, but the fixes and changes are very, very intertwined. It is possible to separate the final fix for #5926 to a subsequent PR, but the final fix itself is actually quite small.
The first commit adds a new
sortoperation to the reftest language. While working on this, the stability of the output ofopam envand the stability of the output when usingunorderedmade making the true effect of each commit tricky. I'm still not sure thatunorderedis behaving as well as it should when there's actually a change to the output (i.e. when one line changes, it's allowing some unchanged lines to be re-ordered noisily in the diff).I then add a whole range of tests:
MANPATH, I noticed thatCAML_LD_LIBRARY_PATHwasn't reverting correctly on Windows either, and it turns out that's because opam wasn't correctly handling updates with forward slashes (i.e.CAML_LD_LIBRARY_PATH += "%{lib}%/stublibsexpands to%OPAMROOT%\switch\lib/stublibsbutopam envnormalises the/stublibsto\stublibs. When reverting, this normalisation is missing.+=opetator inbuild-envto prepend an empty (but set) variable adds a trailing seperator #5925 tests+=,=+,:=and=:applied to a variable set to""in the samesetenvblock and shows+=and=+misbehaving.a:and:a, both set in the environment, and synthesised by using:=or=:on an unset variablea1::a2(a "middle" blank) entry:There have been a lot of issues and PRs in the past to do with the treatment of
:=and=:principally whereMANPATHis concerned. I'm relatively confident that the approach for this before has not been strictly correctly (and is the cause of the issue being seen in #5926). For bothPATH(as defined by Posix) andMANPATH(as defined by the tool itself), an unset value, an empty value and the value:are all equivalent. ForPATH, that means the current directory and all of these entries are literally equivalent toPATH=.. ForMANPATH, those three entries mean "use the default search path".Now, as far as I can see the previous discussions have been about ensuring that
MANPATHhas a leading or trailing:, to ensure that opam never hides the default path. Thus, ifMANPATHis not set, then we haveMANPATH =: "%{man}%"giving us:/home/dra/.opam/default/man. However, the focus should not be on having a leading or trailing:, the focus should be on ensuring that a "blank" entry is created if needed (this is the rule forFOO := "a"andFOO =: "a"area:or:arespectively ifFOOis unset or empty) and but then for all updates separately ensuring that the blank entry is preserved.i.e. if
FOOis:athen bothFOO += "b"andFOO := "b"should setFOOtob::aThe actual fixes then proceed as:
CAML_LD_LIBRARY_PATHissue arises from what appears to be an incorrect comment inOpamEnv.apply_op_zipindicating that the argument is already transformed, which may suggest whyOpamEnv.reverse_env_updatewas missing the transformation. Essentially, the transformation ensures thatopam env --revertreverts the same value as it applied inopam envOpamEnv.split_var, the cases for splitting using the quoted rules were the wrong way around. Fixing that regresses Fix reverting additions to PATH-like variables #4861, though[]. However, ifsetenvorbuild-envhasFOO = ""then we end up with[""]as opam must ensure that this variable is overwritten. The easiest solution for Using the+=opetator inbuild-envto prepend an empty (but set) variable adds a trailing seperator #5925 is simply to allow for that -[""]can only arise fromFOO = ""as an update, so+=and=+handle that case, and remove the""entry from the list.:=and=:at this point of course work correctly, because the""should be there.OpamEnv.split_varis not quite right - it doesn't cope correctly with a separator quoted inside the double-quote marks. My initial approach therefore was to replace it with the PATH-splitter inOpamStd.Env.split_path. This, however, reveals a more fundamental problem - the PATH-splitter correctly splits the PATH, but it removes all the quote marks. The user is at liberty to add quotes in a PATH even if they're not needed. For example, it's a common mistake on Windows to quote directories with spaces in them (having, for example,Pathset toC:\Windows\system32;"C:\Program Files\Something\bin"). With the corrected split function,PATH += "foo"causesPathto becomefoo;C:\Windows\system32;C:\Program Files\Something\binwith the superfluous quotes removed. This isn't correct, given that the effect ofPATH += "foo"should morally be much closer toexport PATH="foo;$PATH".x-env-path-rewrite, if two packages disagree over the separator, then we have the slightly odd situation where the variable is split according to the first package's separator and re-joined according to the second package's. i.e. ifPathisfoo;barand the first package to be evaluated hasPATH += "baz"thenPathinternally is["baz"; "foo"; "bar"]and will be exported asbaz;foo;bar. However, if another package is then added which incorrectly hasx-env-path-rewrite: PATH ":" "target"and executesPATH += "broken"then internally we have["broken"; "baz"; "foo"; "bar"]but the variable will be reconstituted asbroken:baz:foo:bar, so from an initial value offoo;bar,opam envchanged the existing separator.Opam.unzip_toto split the value inFOO += value. I've commented (hopefully clearly) the three cases in the commit. At this point, the Fix reverting additions to PATH-like variables #4861 test passes again and Environment revert not working correctly for Unix-like variables on Windows #5838 is fixed.::as required (see the changes in the reftests in that commit)OpamEnv.reverse_env_updatefor=+and=:where one list was reversed which shouldn't be. Again, the effect is visible in the reftests! The point is that a reversed environment variable was unzipped, and needs to reversed, but there was one reversal too many, given how environment variable updates are stored.TODO