opam install . --deps-only --working-dir incorrectly reports "Nothing to do"

[jonsterling]

As far as I can tell, the --working-dir option of opam is not functional. I cannot tell if this is my own misunderstanding of what --working-dir or --deps-only are supposed to do, but this has happened often enough for me that I thought I would report it as a bug.

What happens

I have a program I am developing, and I have added a new dependency to my .opam file. In this case, I added a dependency on ptime. I now wish to bring my current switch into alignment with the dependencies specified by my project. Note that I have not committed this change in git. Therefore, I run the command opam install . --deps-only --working-dir, which the documentation has led me to believe will do the following: it should install the dependencies of the package specified in the current directory, and it should read the version of the .opam file that is currently in the working directory rather than the one that is committed to version control.

Here is what actually happens:

$ opam install . --deps-only --working-dir
Nothing to do.

I believe this behavior is either wrong or highly unintuitive; in the latter case, maybe we could come up with something to add to the documentation to clarify the intended function of these options.

What if I commit the change and don't use --working-dir?

When I commit the change to my .opam file and run opam install . --deps-only, then opam correctly offers to install ptime.

Diagnostic information

My .opam file

# This file is generated by dune, edit dune-project instead
opam-version: "2.0"
version: "2.1"
synopsis: "A tool for tending mathematical forests"
maintainer: ["Jonathan Sterling"]
authors: ["Jonathan Sterling"]
license: "GPL-3.0-or-later"
homepage: "https://github.com/jonsterling/ocaml-forester"
bug-reports: "https://github.com/jonsterling/ocaml-forester/issues"
depends: [
  "menhir"
  "ocamlgraph"
  "ocaml"
  "dune" {>= "3.7"}
  "xmlm"
  "ppx_deriving"
  "domainslib"
  "ptime"
  "yuujinchou" {>= "5.0.1"}
  "algaeff"
  "odoc" {with-doc}
]
build: [
  ["dune" "subst"] {dev}
  [
    "dune"
    "build"
    "-p"
    name
    "-j"
    jobs
    "@install"
    "@runtest" {with-test}
    "@doc" {with-doc}
  ]
]
dev-repo: "git+https://github.com/jonsterling/ocaml-forester.git"

You can also find the source repository here if you wish to reproduce on your own: https://github.com/jonsterling/ocaml-forester

opam config report:

# opam config report
# opam-version         2.1.4
# self-upgrade         no
# system               arch=arm64 os=macos os-distribution=homebrew os-version=13.4
# solver               builtin-mccs+glpk
# install-criteria     -removed,-count[avoid-version,changed],-count[version-lag,request],-count[version-lag,changed],-count[missing-depexts,changed],-changed
# upgrade-criteria     -removed,-count[avoid-version,changed],-count[version-lag,solution],-count[missing-depexts,changed],-new
# jobs                 7
# repositories         1 (http) (default repo at 1dce70f3)
# pinned               1 (git)
# current-switch       /Users/au598479/Source/ocaml-forester
# ocaml:native         true
# ocaml:native-tools   true
# ocaml:native-dynlink true
# ocaml:stubsdir       /Users/au598479/Source/ocaml-forester/_opam/lib/ocaml/stublibs:/Users/au598479/Source/ocaml-forester/_opam/lib/ocaml
# ocaml:preinstalled   false
# ocaml:compiler       5.0.0

opam list

# Packages matching: installed
# Name                # Installed # Synopsis
alcotest              1.7.0       Alcotest is a lightweight and colourful test framework
algaeff               0.2.1       Reusable Effects-Based Components
astring               0.8.5       Alternative String module for OCaml
base-bigarray         base
base-bytes            base        Bytes library distributed with the OCaml compiler
base-domains          base
base-nnp              base        Naked pointers prohibited in the OCaml heap
base-threads          base
base-unix             base
bwd                   2.1.0       Backward lists
chrome-trace          3.8.0       Chrome trace event generation library
cmdliner              1.2.0       Declarative definition of command line interfaces for OCaml
cppo                  1.6.9       Code preprocessor like cpp for OCaml
csexp                 1.5.2       Parsing and printing of S-expressions in Canonical form
domain_shims          0.1.0       A non-parallel implementation of Domains compatible with OCaml 4
domainslib            0.5.0       Nested-parallel programming library
dune                  3.8.0       Fast, portable, and opinionated build system
dune-build-info       3.8.0       Embed build information inside executable
dune-rpc              3.6.2       Communicate with dune using rpc
dyn                   3.6.2       Dynamic type
fiber                 3.6.2       Structured concurrency library
fmt                   0.9.0       OCaml Format pretty-printer combinators
lockfree              0.3.1       Lock-free data structures for multicore OCaml
menhir                20230415    An LR(1) parser generator
menhirLib             20230415    Runtime support library for parsers generated by Menhir
menhirSdk             20230415    Compile-time library for auxiliary tools related to Menhir
ocaml                 5.0.0       The OCaml compiler (virtual package)
ocaml-base-compiler   5.0.0       Official release 5.0.0
ocaml-compiler-libs   v0.12.4     OCaml compiler libraries repackaged
ocaml-config          3           OCaml Switch Configuration
ocaml-lsp-server      1.15.1-5.0  LSP Server for OCaml
ocaml-options-vanilla 1           Ensure that OCaml is compiled with no special options enabled
ocaml-syntax-shims    1.0.0       Backport new syntax to older OCaml versions
ocamlbuild            0.14.2      OCamlbuild is a build system with builtin rules to easily build most OCaml pro
ocamlc-loc            3.6.2       Parse ocaml compiler output into structured form
ocamlfind             1.9.6       A library manager for OCaml
ocamlformat-rpc-lib   0.25.1      Auto-formatter for OCaml code (RPC mode)
ocamlgraph            2.0.0       A generic graph library for OCaml
ocp-indent            1.8.1       A simple tool to indent OCaml programs
octavius              1.2.2       Ocamldoc comment syntax parser
omd                   1.3.2       A Markdown frontend in pure OCaml
ordering              3.6.2       Element ordering
pp                    1.1.2       Pretty-printing library
ppx_derivers          1.2.1       Shared [@@deriving] plugin registry
ppx_deriving          5.2.1       Type-driven code generation for OCaml
ppx_yojson_conv_lib   v0.15.0     Runtime lib for ppx_yojson_conv
ppxlib                0.29.1      Standard library for ppx rewriters
re                    1.10.4      RE is a regular expression library for OCaml
result                1.5         Compatibility Result module
seq                   base        Compatibility package for OCaml's standard iterator type starting from 4.07.
sexplib0              v0.16.0     Library containing the definition of S-expressions and some base converters
spawn                 v0.15.1     Spawning sub-processes
stdlib-shims          0.3.0       Backport some of the new stdlib features to older compiler
stdune                3.6.2       Dune's unstable standard library
topkg                 1.0.7       The transitory OCaml software packager
uutf                  1.0.3       Non-blocking streaming Unicode codec for OCaml
xdg                   3.8.0       XDG Base Directory Specification
xmlm                  1.4.0       Streaming XML codec for OCaml
yojson                2.1.0       Yojson is an optimized parsing and printing library for the JSON format
yuujinchou            5.0.1       A library for hierarchical names and lexical scoping

As you can see, the package ptime is not installed.

[AltGr]

Thanks for reporting! I had a similar issue with just opam install . --deps-only while the opam file with updated dependencies was properly committed.

It follows that this is unlikely to be related to --working-dir, and more probably due to the way the local opam file needs to be picked in advance when using a directory for package...

[eponier]

I had the same bug with Jasmin, without --working-dir. opam install . --deps-only at the root does not install coq-mathcomp-algebra (among others), and moreover does not report the package as unknown when it does not know it (I initially forgot to add https://coq.inria.fr/opam/released as another remote, so coq-mathcomp-algebra was unknown).

Version: 2.2.0~beta1

[jonsterling]

I would love to get an update on this ticket. This is a very serious problem that users face on day one of introducing themselves to the OCaml ecosystem, and it seems like it should be a high priority to get it fixed. I think that one of the difficulties about getting it fixed is that it seems like it may be caused by more than one thing, so it could be difficult to isolate exactly what is going on.

[kit-ty-kate]

@jonsterling sorry for the annoyance. Until early July 2024 we were focused on the release of opam 2.2 and left issues and PRs aside (except for regressions) to avoid delaying it any longer. After the release we had a backlog of PRs and issues as long as 2 legs and 3 arms and we only recently got to see the end.

Last month i tried to debug and fix this here issue (mentioned in #6101), but i switched to something else and that work was left in a git stash. Looking back at that stash the fix i looked into seems pretty involved, and doesn't fix your issue yet in its current state. I'm planning to continue working on it when i'm back from ICFP in September but as to when it'll be merged it'll depend on how involved the required fix is so i can't make any promises at that stage.

I also encounter this issue from time to time and i agree it's fairly annoying but there are usually workarounds as it seems to appear only when the package(s) you're trying to install the dependencies of is pinned. opam pin add . is my usual workaround. It's not ideal but it should work for you too

[jonsterling]

@kit-ty-kate Thanks very much for the quick response, I'm sorry if I sounded like a broken record! It's great to see the ongoing progress here, and I know that you have a lot on your plate.

Regarding opam pin add ., it is funny you mention this — in the past, that usually worked for me as a workaround, but yesterday it did not cause the offending package to be installed. But that may be caused by yet another issue that I haven't managed to suss out 😉

Anyway, thanks for your work on improving OCaml tools! We're all very endebted.

[kit-ty-kate]

#6209 should fix this issue and should lend in opam 2.4.0 planned for April next year. A backport in 2.3.0 sounds unlikely given the fix touches some intricate part of the code and we would like to have a much time to test it in the wild as possible during the next release cycle.