opam doesn't detect homebrew installed python3

[k4rtik]

❯ opam config report
# opam config report
# opam-version         2.1.2
# self-upgrade         no
# system               arch=arm64 os=macos os-distribution=homebrew os-version=12.1
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 9
# repositories         4 (http), 6 (local), 1 (version-controlled) (default repo at 3920a74b)
# pinned               0
# current-switch       latest
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /Users/kartik/.opam/latest/lib/ocaml/stublibs:/Users/kartik/.opam/latest/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       4.13.1

❯ brew list | grep python
python@3.10
python@3.9

❯ opam install conf-python-3
The following actions will be performed:
  ∗ install conf-python-3 1.0.0

The following system packages will first need to be installed:
    python@3

<><> Handling external dependencies <><><><><><><><><><><><><><><><><><><><>  🐫
Let opam run your package manager to install the required system packages?
(answer 'n' for other options) [Y/n] y
+ /opt/homebrew/bin/brew "install" "python@3"
- Warning: python@3.9 3.9.10 is already installed and up-to-date.
- To reinstall 3.9.10, run:
-   brew reinstall python@3.9

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
∗ installed conf-python-3.1.0.0
Done.

❯ opam config report
# opam config report
# opam-version         2.1.2
# self-upgrade         no
# system               arch=arm64 os=macos os-distribution=homebrew os-version=12.1
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 9
# repositories         4 (http), 6 (local), 1 (version-controlled) (default repo at 3920a74b)
# pinned               0
# current-switch       latest
[WARNING] Opam package conf-python-3.1.0.0 depends on the following system package that can no longer be found: python@3
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /Users/kartik/.opam/latest/lib/ocaml/stublibs:/Users/kartik/.opam/latest/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       4.13.1

Notice the warning in the second output of opam config report:

[WARNING] Opam package conf-python-3.1.0.0 depends on the following system package that can no longer be found: python@3

[rjbou]

The issue here is: when it is asked to brew to install python@3, it installs python@3.x and it is listed only as python@3.x. So opam, when pooling system install status, doesn't see python@3. Kind of virtual/aliases package no handling.
For the moment, we list installed packages with brew list --full-name. Do you know what would be the command to list also those alias packages?

[kit-ty-kate]

We could go through the output of brew info --json=v1 --installed

[k4rtik]

Is it not possible to use the return code of the brew list --full-name python3 command?

PlatiniumM1Pro in ~
❯ brew list --full-name python3
python@3.9

PlatiniumM1Pro in ~ took 570ms
❯ brew list --full-name python2
Error: No available formula with the name "python2". Did you mean ipython, bpython, jython or cython?

PlatiniumM1Pro in ~ took 651ms
✗ brew list --full-name python@3
python@3.9

PlatiniumM1Pro in ~ took 596ms
❯

Notice, for python 2 the return code is not 0 (success) marked by ✗ in my shell output.

[kit-ty-kate]

Sadly brew list --full-name <pkg> would have to be called for each homebrew depexts in opam, that would take too long (0.5s per depexts)

[kit-ty-kate]

xref #4759

[k4rtik]

That makes sense.

Another thought is how many virtual packages are there? If not too many, perhaps the OCaml code for pooling can make exceptions for those by either grepping the version from the package name after the @ symbol in the package name or call brew list --full-name <pkg> explicitly for just those packages.

[rjbou]

It's not conceivable, it will take time (depext mechanism must remain very short, it is called for half opam commands), and if we hard code virtual packages list, users need to wait for an release of opam for each new package added.

We'll try to dig with brew info, it seems the best solution.

In the meantime, you can fix your setup by telling to opam that python@3 system package is well installed, using assume-depext feature: opam option depext-bypass='["python@3"]' --global

[dra27]

We're considering a different way of tackling external dependencies in opam 3.x which might be able to fix this properly, but in the meantime it's at least possible to add the explicit bypass as noted in #5023 (comment). This isn't fixable in an adequate way with the 2.x depext system - closing this as wontfix, for now.