opam list --installed is slow (depexts + yum)

[eponier]

As already noted in another issue, the opam command written by opam itself in the config files of the editors is slow and thus slows down the start of instances of these editors. I noted that with vim and the comment linked above noted that for emacs.

The culprit is the command opam list --installed --short --safe --color=never [packages]. It is rather fast (typically 0.6 s on my system) when it is given installed packages and non-existing packages (although I would expect such command to be instantaneous, not just fast, but I don't know what exactly happens under the hood and it is not blocking in practice). But it is slow when it is given at least one uninstalled package (typically 6.5 s on my system).

I saw that a few issues are open related to performance regressions in latest versions. This may be related.

[eponier]

One easy solution is to remove from the config files the lines added by opam, which I did. So it not blocking, but with the current performance issue, these lines are too costly and thus sadly useless.

[rjbou]

for vim, there is some hints on ocaml-opam/opam-user-setup#53 to speed startup

[eponier]

Indeed, ocaml-opam/opam-user-setup#54 should solve the slow vim startup. But I tried instructions given in ocaml-opam/opam-user-setup#53 (comment), and it did not change anything. I still find it strange that opam list --installed is so slow, especially for non-installed packages.

[rjbou]

If you launch opam list --installed with --debug there is timestamps. Can you share the output?
Also, what is your opam version?

[eponier]

Here is the output of opam --version

2.1.0

Here is the debug output for opam list --installed --debug

00:00.013  CLI                    Parsing CLI version 2.1
00:00.013  GSTATE                 LOAD-GLOBAL-STATE @ /home/user/.opam
00:00.014  RSTATE                 LOAD-REPOSITORY-STATE @ /home/user/.opam
00:00.112  CACHE(repository)      Loaded /home/user/.opam/repo/state-392B1E43.cache in 0.097s
00:00.112  RSTATE                 Cache found
00:00.112  STATE                  LOAD-SWITCH-STATE @ coq.8.12
00:00.113  CACHE(installed)       Loaded /home/user/.opam/coq.8.12/.opam-switch/packages/cache in 0.000s
00:00.555  STATE                  Switch state loaded in 0.443s

Here is the debug output for opam list --installed coq (where coq is installed in the current switch)

00:00.014  CLI                    Parsing CLI version 2.1
00:00.014  GSTATE                 LOAD-GLOBAL-STATE @ /home/user/.opam
00:00.014  RSTATE                 LOAD-REPOSITORY-STATE @ /home/user/.opam
00:00.112  CACHE(repository)      Loaded /home/user/.opam/repo/state-392B1E43.cache in 0.098s
00:00.112  RSTATE                 Cache found
00:00.112  STATE                  LOAD-SWITCH-STATE @ coq.8.12
00:00.113  CACHE(installed)       Loaded /home/user/.opam/coq.8.12/.opam-switch/packages/cache in 0.000s
00:00.560  STATE                  Switch state loaded in 0.448s
# Packages matching: installed & name-match(coq)
# Name # Installed # Synopsis
coq    8.12.0      Formal proof management system

Here is the debug output for opam list --install ocp-build (where ocp-build is an existing package not installed in the current switch)

0:00.014  CLI                    Parsing CLI version 2.1
00:00.014  GSTATE                 LOAD-GLOBAL-STATE @ /home/user/.opam
00:00.014  RSTATE                 LOAD-REPOSITORY-STATE @ /home/user/.opam
00:00.111  CACHE(repository)      Loaded /home/user/.opam/repo/state-392B1E43.cache in 0.097s
00:00.111  RSTATE                 Cache found
00:00.111  STATE                  LOAD-SWITCH-STATE @ coq.8.12
00:00.113  CACHE(installed)       Loaded /home/user/.opam/coq.8.12/.opam-switch/packages/cache in 0.000s
00:00.556  STATE                  Switch state loaded in 0.445s
# Packages matching: installed & name-match(ocp-build)
00:00.571  GSTATE                 Failed to evaluate global variable sys-ocaml-version: "ocamlc": command not found.
00:00.686  XSYS                   Adding to env { LC_ALL=C }
00:06.707  STATE                  depexts loaded in 6.021s
# No matches found

[kit-ty-kate]

mmh, which distribution do you have?

[eponier]

Fedora 33

[eponier]

BTW I observe the same behaviour with opam show: fast on installed packages, and 6 s on non-installed ones.

[kit-ty-kate]

I believe #4791 should have fixed that. We're planning to start releasing opam 2.1.1 today (the release process is quite long so full announcement will probably be there on tuesday).

Could you try the 2.1 branch to confirm this fixes your issue?

[eponier]

Tested the commits of #4792, and indeed it is much much faster, but still slow in my opinion

00:00.013  CLI                    Parsing CLI version 2.1
00:00.013  GSTATE                 LOAD-GLOBAL-STATE @ /home/user/.opam
00:00.013  RSTATE                 LOAD-REPOSITORY-STATE @ /home/user/.opam
00:00.115  CACHE(repository)      Loaded /home/user/.opam/repo/state-3AF2812F.cache in 0.101s
00:00.115  RSTATE                 Cache found
00:00.115  STATE                  LOAD-SWITCH-STATE @ coq.8.14
00:00.116  CACHE(installed)       Loaded /home/user/.opam/coq.8.14/.opam-switch/packages/cache in 0.000s
00:00.555  STATE                  Switch state loaded in 0.440s
00:00.567  GSTATE                 Failed to evaluate global variable sys-ocaml-version: "ocamlc": command not found.
00:00.692  XSYS                   Adding to env { LC_ALL=C }
00:03.212  STATE                  depexts loaded in 2.520s

[eponier]

Actually dnf info also takes more or less 2.5s on my system, so maybe it can be considered acceptable. What was not acceptable was having this penalty each time a file is opened, but since #4792 this is no longer the case. I let you decide what to do.

[rjbou]

I'll close it, feel free to reopen it, especially if you have an idea of something faster than dnf