IO bottleneck: repository decompression

[kit-ty-kate]

When calling opam install <pkg> one of the first processes opam calls are:

/usr/bin/tar xfz /home/opam/.opam/repo/default.tar.gz -C /tmp/opam-5953-44547
....
rm -rf /tmp/opam-5953-44547/default

These two commands seem to be the main IO bottleneck in CI where /tmp is not tmpfs for technical reason (cached containerization). I feel like we might see a good performance improvement by decompressing the repository in OCaml directly probably in some kind of stream mode if the thing that needs the repository information can be used this way.

Partially related to the later discussion in #3050

[talex5]

in CI where /tmp is not tmpfs for technical reason (cached containerization)

I guess OBuilder could add a (tmpfs /tmp) argument to (run ...), assuming we don't need to preserve anything there across build steps.

[dra27]

Dev meeting: @AltGr has a patch which can disable the tar operation, which at least stops it being slower than in 2.0. This can certainly go in 2.1.1, although we should also look at other ways of not needing to analyse so many thousand files on each update operation!

[AltGr]

The patch is here: OCamlPro@567cb24
In short, nothing needs to be changed for loading, it is enough to disable the generation of the tar.gz archives ; adding an option to selectively disable the code that the above commit comments out should be fairly straight-forward.

[AltGr]

ironic that these .tar.gz files were introduced as an optimisation, and are a huge benefit on standard Linux boxes 🤷

[AltGr]

Just one note:

When calling opam install one of the first processes opam calls are: [... tar xfz ...]

this is only true if you don't have a cache: normally the untarring is only needed on opam update or some repository operations ; otherwise the useful contents are found marshalled directly from .opam/repo/state.cache

[kit-ty-kate]

what would be the possible reasons for not having a cache?

[dra27]

Switching opam binary would invalidate it - aren't those opam roots made with opam 2 and you're then switching to 2.1?

[kit-ty-kate]

mmh, it seems that rm -rf ~/.opam got lost in translation in ocurrent/opam-health-check@7c83696 for some reason :/ (which is wrong on so many levels)

However I'm a bit surprised that opam switch create doesn't update the cache. What is the reason for that?

[dra27]

However I'm a bit surprised that opam switch create doesn't update the cache. What is the reason for that?

The cache isn't out of date at that point?

[kit-ty-kate]

How do I know if it's out of date?
My reasoning is that if opam install <pkg> does update the cache [*], opam switch create (called just before) should too.

The sequence of operation i had was roughly:

[some operation using opam 2.0]
[...]
[here I forgot to keep rm -rf ~/.opam]
opam init -ya --bare --disable-sandboxing ~/opam-repository
opam switch create 4.12.0
opam install <pkg>

[*]: does it update the cache or is it just untaring every time because it is in an "invalid" state?

[kit-ty-kate]

Just one note:

When calling opam install one of the first processes opam calls are: [... tar xfz ...]

this is only true if you don't have a cache: normally the untarring is only needed on opam update or some repository operations ; otherwise the useful contents are found marshalled directly from .opam/repo/state.cache

@AltGr after trying again with a clean opamroot (rm -rf .opam && opam init --bare ~/opam-repository) here is what I'm seeing in htop:

The second call to tar is expected, but the first one is not from what you're saying. This is using opam master (8977e43)

[dra27]

Notes from today: if we add a global option to allow this to be turned, Windows can benefit as it can be turned off by default (the caching mechanism is much slower than the already slow 2.0 mechanism!) and the option would of course have a corresponding environment variable allowing it to be set in CI systems.

[kit-ty-kate]

I was able to reproduce this behaviour quite easily with opam master (41f3684) in a docker container:

FROM ocaml/opam:debian-11-ocaml-4.14@sha256:e476a3531d4b276add76486f03cdc608028bff96b92995cf30fc3ee1c2fc60c4
RUN sudo ln -f /usr/bin/opam-2.1 /usr/bin/opam
RUN opam init --reinit -ni

From there run htop in one window and look at the subprocesses of the bash process created by docker run --rm -it <the-base-image-you-created> and run opam install core.
You will see /usr/bin/tar xfz /home/opam/.opam/repo/default.tar.gz -C /tmp/opam-5953-44547 being called every time around when The following actions will be performed: is outputed.

EDIT: Actually, the tar call appears a bit after the first archives retrivals if that helps.

[rjbou]

cool! in the good moment to test the PR :D

[kit-ty-kate]

opam install -vvv dune.2.9.1 returns:

The following actions will be performed:
  - install dune 2.9.1

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  1/3: [dune.2.9.1: http]
+ /usr/bin/curl "--write-out" "%{http_code}\\n" "--retry" "3" "--retry-delay" "2" "--user-agent" "opam/2.2.0~alpha~dev" "-L" "-o" "/home/opam/.opam/4.13/.opam-switch/sources/dune.2.9.1/dune-2.9.1.tbz.part" "--" "https://github.com/ocaml/dune/releases/download/2.9.1/dune-2.9.1.tbz"
+ /bin/mv "/home/opam/.opam/4.13/.opam-switch/sources/dune.2.9.1/dune-2.9.1.tbz.part" "/home/opam/.opam/4.13/.opam-switch/sources/dune.2.9.1/dune-2.9.1.tbz"
+ /bin/mv "/home/opam/.opam/4.13/.opam-switch/sources/dune.2.9.1/dune-2.9.1.tbz" "/tmp/opam-14-82d098/dune-2.9.1.tbz"
Processing  1/3:
+ /usr/bin/tar "xfj" "/tmp/opam-14-82d098/dune-2.9.1.tbz" "-C" "/tmp/opam-14-1e6357"
+ /bin/cp "-PRp" "/tmp/opam-14-1e6357/dune-2.9.1/CHANGES.md" "/tmp/opam-14-1e6357/dune-2.9.1/CONTRIBUTING.md" "/tmp/opam-14-1e6357/dune-2.9.1/HACKING.md" "/tmp/opam-14-1e6357/dune-2.9.1/LICENSE.md" "/tmp/opam-14-1e6357/dune-2.9.1/MIGRATION.md" "/tmp/opam-14-1e6357/dune-2.9.1/Makefile" "/tmp/opam-14-1e6357/dune-2.9.1/README.md" "/tmp/opam-14-1e6357/dune-2.9.1/bench" "/tmp/opam-14-1e6357/dune-2.9.1/bin" "/tmp/opam-14-1e6357/dune-2.9.1/boot" "/tmp/opam-14-1e6357/dune-2.9.1/bootstrap.ml" "/tmp/opam-14-1e6357/dune-2.9.1/configure" "/tmp/opam-14-1e6357/dune-2.9.1/configure.ml" "/tmp/opam-14-1e6357/dune-2.9.1/doc" "/tmp/opam-14-1e6357/dune-2.9.1/dune" "/tmp/opam-14-1e6357/dune-2.9.1/dune-action-plugin.opam" "/tmp/opam-14-1e6357/dune-2.9.1/dune-build-info.opam" "/tmp/opam-14-1e6357/dune-2.9.1/dune-configurator.opam" "/tmp/opam-14-1e6357/dune-2.9.1/dune-glob.opam" "/tmp/opam-14-1e6357/dune-2.9.1/dune-private-libs.opam" "/tmp/opam-14-1e6357/dune-2.9.1/dune-private-libs.opam.template" "/tmp/opam-14-1e6357/dune-2.9.1/dune-project" "/tmp/opam-14-1e6357/dune-2.9.1/dune-site.opam" "/tmp/opam-14-1e6357/dune-2.9.1/dune-workspace.dev" "/tmp/opam-14-1e6357/dune-2.9.1/dune.opam" "/tmp/opam-14-1e6357/dune-2.9.1/dune.opam.template" "/tmp/opam-14-1e6357/dune-2.9.1/editor-integration" "/tmp/opam-14-1e6357/dune-2.9.1/example" "/tmp/opam-14-1e6357/dune-2.9.1/otherlibs" "/tmp/opam-14-1e6357/dune-2.9.1/perf.sh" "/tmp/opam-14-1e6357/dune-2.9.1/plugin" "/tmp/opam-14-1e6357/dune-2.9.1/src" "/tmp/opam-14-1e6357/dune-2.9.1/test" "/tmp/opam-14-1e6357/dune-2.9.1/vendor" "/home/opam/.opam/4.13/.opam-switch/sources/dune.2.9.1"
-> retrieved dune.2.9.1  (https://github.com/ocaml/dune/releases/download/2.9.1/dune-2.9.1.tbz)
+ /bin/cp "-PRp" "/home/opam/.opam/4.13/.opam-switch/sources/dune.2.9.1" "/home/opam/.opam/4.13/.opam-switch/build/dune.2.9.1"
+ /usr/bin/tar "xfz" "/home/opam/.opam/repo/default.tar.gz" "-C" "/tmp/opam-14-832cb6"
Processing  2/3: [dune: ocaml bootstrap.ml]
+ /home/opam/.opam/4.13/bin/ocaml "bootstrap.ml" "-j" "31" (CWD=/home/opam/.opam/4.13/.opam-switch/build/dune.2.9.1)
Processing  2/3: [dune: ./dune.exe build]
+ /home/opam/.opam/4.13/.opam-switch/build/dune.2.9.1/./dune.exe "build" "-p" "dune" "--profile" "dune-bootstrap" "-j" "31" (CWD=/home/opam/.opam/4.13/.opam-switch/build/dune.2.9.1)
-> compiled  dune.2.9.1
-> installed dune.2.9.1
Done.

[kit-ty-kate]

Ok I got it. So the issue is that we’re not storing entirety of the repository inside of the state-cache, so everytime opam installs something it first needs to pull the extra-files out of the repository (even if it’s just to realize the package didn’t have any). And for that it needs to extract it.

https://github.com/ocaml/opam/blob/master/src/client/opamAction.ml#L441

To me there doesn’t seem to be any upsides in keeping that around and I would suggest reverting #3752 as opam-update now is slower than in opam 2.0:

• opam master (f70548b) init: 28.8s, update: 28.2s (with the default http repository)
• opam 2.0.10: init: 15.9s, update: 20.8s (with the same default http repository)

[rjbou]

For extra files, it doesn't need to extract it when it looks for extra files, it is extracted when the repository state is loaded. Then when looking for extra files, it checks files existence in the repo (in tmp new path for tarred repo, and locally in .opam/repo/xxx for non tarred repos). See this minimal test.

On the times, i got similar values on a debian: opam 2.0.10 < opam 2.2 with repo optim < opam 2.2 without repo optim. I didn't test yet the revert times. @AltGr do you remember or noted your times when you did the optim?