Failure to install Oscar.jl on a clean macOS 12.3.1 install: can't install gap_packages artifact.

[fingolfin]

I am currently revising our install instructions and wanted to finally tackle oscar-system/oscar-website#217 -- so I installed a fresh macOS VM on my laptop. Installing developer tools and Julia 1.7.2 went smoothly. Good!

But then installing Oscar.jl itself failed:

┌ Error: Tree Hash Mismatch!
│   Expected git-tree-sha1:   12258958e233a45a336c1b92681cc2cafbd1fb90
│   Calculated git-tree-sha1: 9eca000fc45084e45b9dc0f4883f6406a27e18c2
  Downloaded artifact: gap_packages
ERROR: Unable to automatically install 'gap_packages' from '/Users/mhorn/.julia/packages/GAP/lK1LP/Artifacts.toml'
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)
    @ Pkg.Artifacts /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Artifacts.jl:441
  [3] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:617
  [4] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1182
  [5] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:268
  [6] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:149
  [7] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:144
  [8] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:407
  [9] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:385
 [10] do_cmd
    @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:376 [inlined]
 [11] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/REPLMode/REPLMode.jl:549
 [12] #invokelatest#2
    @ ./essentials.jl:716 [inlined]
 [13] invokelatest
    @ ./essentials.jl:714 [inlined]
 [14] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/REPL/src/LineEdit.jl:2493
 [15] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/REPL/src/REPL.jl:1232
 [16] (::REPL.var"#49#54"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:423

Sometimes this kind of issue happens due to the macOS file system not distinguishing case in file names; so if an artifact contains files foo and FOO in the same directory, then expanding the artifact on macOS causes one of the files to be lost/overwritten by the other, and hence the above checksum failure.

But what I don't understand is why I've not seen this anywhere else, nor has anyone reported this before, as far as I know?

Ping @benlorenz in case he has an idea.

[fingolfin]

Actually I just noticed that the "Calculated git-tree-sha1: keeps changing! I've now seen:

• 5060ad0cce9e7cbc0dbea3c9d585d97a90a61729
• 9eca000fc45084e45b9dc0f4883f6406a27e18c2
• 561249312b8bd1bcbed5a1187a7539f130d2980d

And after retrying several times, it worked... WTF? So perhaps an issue with my VM? Urgh.

[thofma]

Not sure if relevant, but I remember seeing mismatchs related to docker: https://discourse.julialang.org/t/adding-hdf5-tree-hash-mismatch-a-docker-on-windows-problem-solved/35817.

[benlorenz]

That the hash keeps changing would point to some problem with slow IO maybe.
I am not aware of any bugs with the tree hash in julia 1.7 (there were some in earlier versions). The tarball looks fine, I don't see any case-clashes and no hardlinks (which also did cause issues at some point).

What kind of filesystem is it using, is it a shared folder from outside of the VM?
Have you checked if there is a tree from such a failed extraction attempt that we could compare with the tarball?

[fingolfin]

I'll check the tree if this happens again. For now, I am going to close this as a fluke