Updating dune to 3.0 could fail

[bobot]

It is hard to reproduce, but I successively did it in two different switches:

#=== ERROR while compiling xdg.3.0 ============================================#
# context     2.1.0 | linux/x86_64 | ocaml-base-compiler.4.12.0 | pinned(git+ssh://git@github.com/ocaml/dune.git#c46b89e1f#c46b89e1)
# path        ~/.opam/4.12.0/.opam-switch/build/xdg.3.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p xdg -j 4 --promote-install-files=false @install
# exit-code   1
# env-file    ~/.opam/log/xdg-2046324-72a86d.env
# output-file ~/.opam/log/xdg-2046324-72a86d.out
### output ###
# [...]
# Called from Fiber.Execution_context.run_jobs in file "src/fiber/fiber.ml",
#   line 204, characters 8-13
# -> required by ("<unnamed>", ())
# -> required by ("<unnamed>", ())
# -> required by ("load-dir", In_build_dir "default/bench")
# -> required by ("toplevel", ())
# 
# I must not crash.  Uncertainty is the mind-killer. Exceptions are the
# little-death that brings total obliteration.  I will fully express my cases. 
# Execution will pass over me and through me.  And when it has gone past, I
# will unwind the stack along its path.  Where the cases are handled there will
# be nothing.  Only I will remain.

I think one switch was starting with a master pinned at a previous version of 3.0 and the other before 3.0 but not sure. However once in this state I'm not able to install xdg. However an opam switch reinstall worked in the second switch.

[bobot]

CC @rgrinberg

[bobot]

The error hidden by opam is

Internal error, please report upstream including the contents of _build/log.
Description:
  ("Unexpected find result", { found = Not_found; lib.name = "pp" })

[bobot]

If in the opam build directory of xdg I remove vendor/pp the error is then about csexp.

[rgrinberg]

Okay, I think I found the issue. This bug was triggered when I introduced allow_overlapping_deps for the benchmarks.

[voodoos]

I just got the same error while pinning dune on my main 4.13 switch:

#=== ERROR while compiling xdg.3.0 ============================================#
# context     2.1.0~beta4 | macos/x86_64 | ocaml-base-compiler.4.13.1 | pinned(git+https://github.com/ocaml/dune.git#2a99ae5c)
# path        ~/.opam/4.13.1/.opam-switch/build/xdg.3.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p xdg -j 7 @install
# exit-code   1
# env-file    ~/.opam/log/xdg-50851-5b985a.env
# output-file ~/.opam/log/xdg-50851-5b985a.out
### output ###
# [...]
# Called from Fiber.Execution_context.run_jobs in file "src/fiber/fiber.ml",
#   line 218, characters 8-13
# -> required by ("<unnamed>", ())
# -> required by ("<unnamed>", ())
# -> required by ("load-dir", In_build_dir "default/bench")
# -> required by ("toplevel", ())
# 
# I must not crash.  Uncertainty is the mind-killer. Exceptions are the
# little-death that brings total obliteration.  I will fully express my cases. 
# Execution will pass over me and through me.  And when it has gone past, I
# will unwind the stack along its path.  Where the cases are handled there will
# be nothing.  Only I will remain.

Unfortunatley I don't have a lot more info to provide.
Here is my previous switch state but it's a bit to big:

opam-version: "2.0"
compiler: ["ocaml-base-compiler.4.13.1"]
roots: [
  "bisect_ppx.2.7.0"
  "cinaps.v0.15.1"
  "coq.8.14.1"
  "coq-native.1"
  "core_bench.v0.14.0"
  "ctypes.0.19.1"
  "dune.2.9.1"
  "dune-release.1.5.2"
  "js_of_ocaml.3.11.0"
  "js_of_ocaml-compiler.3.11.0"
  "lwt.5.5.0"
  "mdx.1.11.0"
  "merlin.4.4-413"
  "mtime.1.3.0"
  "ocaml-base-compiler.4.13.1"
  "ocaml-lsp-server.1.9.1"
  "ocamlformat.0.19.0"
  "odoc.2.0.0"
  "ppx_expect.v0.14.2"
  "ppx_inline_test.v0.14.1"
  "ppx_irmin.2.9.0"
  "ppxlib.0.23.0"
  "uri.4.2.0"
  "utop.2.8.0"
]
installed: [
  "alcotest.1.5.0"
  "angstrom.0.15.0"
  "asn1-combinators.0.2.6"
  "astring.0.8.5"
  "awa.0.0.4"
  "awa-mirage.0.0.4"
  "base.v0.14.2"
  "base-bigarray.base"
  "base-bytes.base"
  "base-threads.base"
  "base-unix.base"
  "base64.3.5.0"
  "base_bigstring.v0.14.0"
  "base_quickcheck.v0.14.1"
  "bheap.2.0.0"
  "bigarray-compat.1.0.0"
  "bigstringaf.0.8.0"
  "bin_prot.v0.14.0"
  "biniou.1.2.1"
  "bisect_ppx.2.7.0"
  "bos.0.2.0"
  "ca-certs.0.2.2"
  "ca-certs-nss.3.71.0.1"
  "camomile.1.0.2"
  "carton.0.4.3"
  "carton-git.0.4.3"
  "carton-lwt.0.4.3"
  "charInfo_width.1.1.0"
  "checkseum.0.3.2"
  "cinaps.v0.15.1"
  "cmdliner.1.0.4"
  "cohttp.4.0.0"
  "cohttp-lwt.4.0.0"
  "cohttp-lwt-unix.4.0.0"
  "conduit.4.0.2"
  "conduit-lwt.4.0.2"
  "conduit-lwt-unix.4.0.2"
  "conf-findutils.1"
  "conf-gmp.3"
  "conf-gmp-powm-sec.3"
  "conf-gnuplot.0.1"
  "conf-libffi.2.0.0"
  "conf-pkg-config.2"
  "coq.8.14.1"
  "coq-native.1"
  "core.v0.14.1"
  "core_bench.v0.14.0"
  "core_kernel.v0.14.2"
  "cppo.1.6.8"
  "crunch.3.2.0"
  "csexp.1.5.1"
  "cstruct.6.0.1"
  "cstruct-lwt.6.0.1"
  "cstruct-sexp.6.0.1"
  "cstruct-unix.6.0.1"
  "ctypes.0.19.1"
  "ctypes-foreign.0.18.0"
  "curly.0.2.0"
  "decompress.1.4.2"
  "digestif.1.1.0"
  "domain-name.0.3.1"
  "dot-merlin-reader.4.1"
  "duff.0.4"
  "dune.2.9.1"
  "dune-build-info.2.9.1"
  "dune-configurator.2.9.1"
  "dune-release.1.5.2"
  "duration.0.2.0"
  "easy-format.1.3.2"
  "either.1.0.0"
  "emile.1.1"
  "encore.0.8"
  "eqaf.0.8"
  "ethernet.2.2.1"
  "faraday.0.8.1"
  "fieldslib.v0.14.0"
  "fix.20211125"
  "fmt.0.9.0"
  "fpath.0.7.3"
  "git.3.6.0"
  "git-cohttp.3.6.0"
  "git-cohttp-unix.3.6.0"
  "git-paf.3.6.0"
  "git-unix.3.6.0"
  "gmap.0.3.0"
  "graphql.0.13.0"
  "graphql-cohttp.0.13.0"
  "graphql-lwt.0.13.0"
  "graphql_parser.0.13.0"
  "h2.0.8.0"
  "hkdf.1.0.4"
  "hpack.0.2.0"
  "httpaf.0.7.1"
  "hxd.0.3.1"
  "index.1.5.0"
  "integers.0.5.1"
  "ipaddr.5.2.0"
  "ipaddr-sexp.5.2.0"
  "jane-street-headers.v0.14.0"
  "js_of_ocaml.3.11.0"
  "js_of_ocaml-compiler.3.11.0"
  "jsonm.1.0.1"
  "jst-config.v0.14.1"
  "ke.0.4"
  "lambda-term.3.1.0"
  "logs.0.7.0"
  "lru.0.3.0"
  "lwt.5.5.0"
  "lwt-dllist.1.0.1"
  "lwt_log.1.1.1"
  "lwt_react.1.1.5"
  "macaddr.5.2.0"
  "macaddr-cstruct.5.2.0"
  "magic-mime.1.2.0"
  "mdx.1.11.0"
  "memtrace.0.2.1.2"
  "menhir.20211125"
  "menhirLib.20211125"
  "menhirSdk.20211125"
  "merlin.4.4-413"
  "metrics.0.3.0"
  "metrics-unix.0.3.0"
  "mew.0.1.0"
  "mew_vi.0.5.0"
  "mimic.0.0.3"
  "mirage-clock.4.0.0"
  "mirage-clock-unix.4.0.0"
  "mirage-crypto.0.10.5"
  "mirage-crypto-ec.0.10.5"
  "mirage-crypto-pk.0.10.5"
  "mirage-crypto-rng.0.10.5"
  "mirage-device.2.0.0"
  "mirage-flow.3.0.0"
  "mirage-kv.4.0.0"
  "mirage-net.4.0.0"
  "mirage-no-solo5.1"
  "mirage-no-xen.1"
  "mirage-profile.0.9.1"
  "mirage-protocols.7.0.0"
  "mirage-random.3.0.0"
  "mirage-stack.3.0.0"
  "mirage-time.3.0.0"
  "mmap.1.1.0"
  "mtime.1.3.0"
  "num.1.4"
  "ocaml.4.13.1"
  "ocaml-base-compiler.4.13.1"
  "ocaml-compiler-libs.v0.12.4"
  "ocaml-config.2"
  "ocaml-lsp-server.1.9.1"
  "ocaml-migrate-parsetree.2.2.0"
  "ocaml-options-vanilla.1"
  "ocaml-syntax-shims.1.0.0"
  "ocaml-version.3.4.0"
  "ocamlbuild.0.14.0"
  "ocamlfind.1.9.1"
  "ocamlformat.0.19.0"
  "ocamlformat-rpc-lib.0.19.0"
  "ocamlgraph.2.0.0"
  "ocp-indent.1.8.1"
  "ocplib-endian.1.2"
  "octavius.1.2.2"
  "odoc.2.0.0"
  "odoc-parser.0.9.0"
  "opam-core.2.1.1"
  "opam-file-format.2.1.3"
  "opam-format.2.1.1"
  "opam-repository.2.1.1"
  "opam-state.2.1.1"
  "optint.0.1.0"
  "paf.0.0.6"
  "parsexp.v0.14.1"
  "pbkdf.1.2.0"
  "pecu.0.6"
  "pp.1.1.2"
  "ppx_assert.v0.14.0"
  "ppx_base.v0.14.0"
  "ppx_bench.v0.14.1"
  "ppx_bin_prot.v0.14.0"
  "ppx_cold.v0.14.0"
  "ppx_compare.v0.14.0"
  "ppx_cstruct.6.0.1"
  "ppx_custom_printf.v0.14.1"
  "ppx_derivers.1.2.1"
  "ppx_deriving.5.2.1"
  "ppx_deriving_yojson.3.6.1"
  "ppx_enumerate.v0.14.0"
  "ppx_expect.v0.14.2"
  "ppx_fields_conv.v0.14.2"
  "ppx_fixed_literal.v0.14.0"
  "ppx_hash.v0.14.0"
  "ppx_here.v0.14.0"
  "ppx_inline_test.v0.14.1"
  "ppx_irmin.2.9.0"
  "ppx_jane.v0.14.0"
  "ppx_js_style.v0.14.1"
  "ppx_let.v0.14.0"
  "ppx_module_timer.v0.14.0"
  "ppx_optcomp.v0.14.3"
  "ppx_optional.v0.14.0"
  "ppx_pipebang.v0.14.0"
  "ppx_repr.0.5.0"
  "ppx_sexp_conv.v0.14.3"
  "ppx_sexp_message.v0.14.1"
  "ppx_sexp_value.v0.14.0"
  "ppx_stable.v0.14.1"
  "ppx_string.v0.14.1"
  "ppx_typerep_conv.v0.14.2"
  "ppx_variants_conv.v0.14.2"
  "ppx_yojson_conv_lib.v0.14.0"
  "ppxlib.0.23.0"
  "progress.0.2.1"
  "psq.0.2.0"
  "ptime.0.8.6"
  "randomconv.0.1.3"
  "re.1.10.3"
  "react.1.2.1"
  "repr.0.5.0"
  "result.1.5"
  "rresult.0.6.0"
  "rusage.1.0.0"
  "semaphore-compat.1.0.1"
  "seq.base"
  "sexplib.v0.14.0"
  "sexplib0.v0.14.0"
  "spawn.v0.14.0"
  "splittable_random.v0.14.0"
  "stdio.v0.14.0"
  "stdlib-shims.0.3.0"
  "stringext.1.6.0"
  "tcpip.6.4.0"
  "terminal.0.2.1"
  "textutils.v0.14.0"
  "tezos-base58.1.0.0"
  "time_now.v0.14.0"
  "timezone.v0.14.0"
  "tls.0.15.2"
  "tls-mirage.0.15.2"
  "topkg.1.0.4"
  "trie.1.0.0"
  "typerep.v0.14.0"
  "tyxml.4.5.0"
  "uchar.0.0.2"
  "uri.4.2.0"
  "uri-sexp.4.2.0"
  "utop.2.8.0"
  "uucp.14.0.0"
  "uuidm.0.9.7"
  "uuseg.14.0.0"
  "uutf.1.0.2"
  "variantslib.v0.14.0"
  "vector.1.0.0"
  "x509.0.15.2"
  "yaml.3.0.0"
  "yojson.1.7.0"
  "zarith.1.12"
  "zed.3.1.0"
]
pinned: [
  "dune.2.9.1"
  "dune-action-plugin.3.0"
  "dune-build-info.3.0"
  "dune-configurator.3.0"
  "dune-glob.3.0"
  "dune-private-libs.3.0"
  "dune-rpc.3.0"
  "dune-rpc-lwt.3.0"
  "dune-site.3.0"
  "dyn.3.0"
  "fiber.3.0"
  "ordering.3.0"
  "stdune.3.0"
  "xdg.~dev"
]

Or maybe it is linked to my pinned section:

pinned: [
  "dune.2.9.1"
  "dune-action-plugin.3.0"
  "dune-build-info.3.0"
  "dune-configurator.3.0"
  "dune-glob.3.0"
  "dune-private-libs.3.0"
  "dune-rpc.3.0"
  "dune-rpc-lwt.3.0"
  "dune-site.3.0"
  "dyn.3.0"
  "fiber.3.0"
  "ordering.3.0"
  "stdune.3.0"
  "xdg.~dev"
]

I apparently mixed 2.9.1 packages with 3.0 dev ones (probably the remnant of a dune pin that I did not completely removed). That might be the source of the problem.

[bobot]

The problem occured again, with the current master (nearly):

#=== ERROR while compiling dune.3.0 ===========================================#
# context     2.1.0 | linux/x86_64 | ocaml-base-compiler.4.12.0 | pinned(git+file:///home/bobot/Sources/dune#ctypes_dependencies#10520780)
# path        ~/.opam/4.12.0/.opam-switch/build/dune.3.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build ./dune.exe build -p dune --profile dune-bootstrap -j 4
# exit-code   1
# env-file    ~/.opam/log/dune-1941827-188dd2.env
# output-file ~/.opam/log/dune-1941827-188dd2.out
### output ###
Internal error, please report upstream including the contents of _build/log.
Description:
  ("Unexpected find result", { found = Not_found; lib.name = "pp" })
Raised at Stdune__code_error.raise in file "otherlibs/stdune/code_error.ml",
  line 11, characters 30-62
Called from Fiber.Execution_context.run_jobs in file "src/fiber/fiber.ml",
  line 218, characters 8-13
-> required by ("<unnamed>", ())
-> required by ("<unnamed>", ())
-> required by ("gen-rules", "default/test/unit-tests")
-> required by ("load-dir", In_build_dir "default/test/unit-tests")
-> required by ("toplevel", ())

In the opam build directory after removing the directory pp and csexp it compiles.

[bobot]

From a clean repo it can be triggered with:

"ocaml" "bootstrap.ml" "-j" 4
"./dune.exe" "build" "-p" "dune" "--profile" "dune-bootstrap" "-j" "4"

[bobot]

Strangely if the configure is run, bobot/jbuilder@025978d, there is no error.

  # opam 2 sets OPAM_SWITCH_PREFIX, so we don't need a hardcoded path
  ["ocaml" "configure.ml" "--libdir" lib] {opam-version < "2"}

[bobot]

@rgrinberg are you able to reproduce it?

[rgrinberg]

I was not able to reproduce it. I pinned the master branch and it installed just fine.

[bobot]

Do you tried also with a clean repo #5190 (comment) and the command run locally? I'm still able to reproduce it. I will look more.

[bobot]

I think the important part is to have dune 3.0 (main) installed in the switch and not just dune but all the dune*packages (I haven't minimized). It could be reproduced in a clean switch and clone with the following script:

#!/bin/sh

SWITCH_NAME=test_dune_5190

opam switch create --no-switch $SWITCH_NAME 4.13.1
opam pin --switch=$SWITCH_NAME "git@github.com:ocaml/dune.git" --with-version=3.0 --yes

git clone "git@github.com:ocaml/dune.git" /tmp/$SWITCH_NAME

cd /tmp/$SWITCH_NAME

opam exec --switch=$SWITCH_NAME -- "ocaml" "bootstrap.ml"
opam exec --switch=$SWITCH_NAME -- "./dune.exe" "build" "-p" "dune" "--profile" "dune-bootstrap"

[rgrinberg]

I'm able to reproduce the error with your script as well. But the build instructions in your script are wrong. To build dune, you must not filter out the packages in the repository with -p dune. Which is why the opam file states:

  ["./dune.exe" "build" "dune.install" "--release" "--profile" "dune-bootstrap" "-j" jobs]

[bobot]

So I'm closing it again . Perhaps a problem of spurious opam file