Properly specify dependencies of librairies in opam files

[Firobe]

Pins are currently performed via the scripts/pin-external-packages.sh script. opam supports pins in the switch description (and should support relative paths).

See if we can remove the script and specify everything in the switch description. (moved to #11120

There are already opam files for libraries, we should properly specify the deps of these libraries in opam files. Potentially using an automatic tool to ensure dune, dune-project and dune files are synchronized.

[mimoo]

pasting my comment from #11114 (comment)

random thought looking at this file. I was investigating adding pins here instead of our setup script, but modifying this file by hand seems hard (no documentation on how to do it that I could find) and error prone (as it is supposed to be a machine generated file).

It would be nice to be able to list all the dependencies, all the pins, and the OCaml version we use, in a single .toml or .json file (similar to a Cargo.toml in Rust)

[Firobe]

@mimoo I was wondering about this. Is your idea to create our own format in a .toml file (or whatever) and to have some machinery generating an opam.export file from that? I'm not sure how easy that is compared to modifying that file manually.

A nice compromise, I think, would be to do this (the first solution): https://opam.ocaml.org/doc/Tricks.html#Provide-a-set-of-packages-for-a-group-of-users-to-install

That is, make a package definition for the whole repository with all of our needed dependencies, which is how they're supposed to be specified in the first place.

[mimoo]

I think the ideal world is this:

• for every library, you have a configuration file that lists all of a library's direct dependencies and pins
• for the entire workspace, you have a config file that fixes the OCaml version
• these files have an extension and are backed by a specification language (toml, json, etc.)
• to build, run tests, setup your env, etc. you call a tool that will end up parsing these configuration files to make sure things are built with the correct ocaml version and dependencies

This allows us to always have these files that specify EVERYTHING from the OCaml version to where all the dependencies (and their versions) are located.

Without that, you never know what you're really doing. What switch are you using? Do you have the right dependencies installed on your switch or system? Do you have the right pins in place? Do you have the right OCaml versino? etc. I suspect that these are the source for the most time wasted with the build system.

I don't like the opam.export file for two reasons:

• it's not used by anything, except manually to do a one-time setup at the beginning. From there you can mess up your switch
• it's not meant to be manually modified or read by a human (it is extremely verbose and unorganized)

It looks like you can't do any of that with dune/opam. esy sounded like it would get us closer to a world like this, but it seems to have many bug and still relies heavily on you writing dune files. I don't think there's an out-of-the-box solution that exists in the OCaml ecosystem. I remember reading about https://ocamlpro.github.io/drom/ but it was quite messy as well (and not actively maintained)

[Firobe]

Tell me if I'm wrong but it seems to me that *.opam files match your requirements.

There is currently opam files for every library but no dependency is specified: we could incrementally get to a state where all dependencies are specified for all libraries in the appropriate place.

Then at the top level, we have a mina.opam depending on all local libraries and pinning necessary ones. To ensure you are in the correct state, just opam pin mina at any moment. And contrary to opam.export, the opam file format is actually meant to be read and written by humans.

it's not used by anything, except manually to do a one-time setup at the beginning. From there you can mess up your switch

This is not true anymore with the Nix setup thanks to @balsoft ! Currently if you setup nix on your system, you can just nix develop mina and develop while being certain all your dependencies are correct (and you cannot mess up the switch).

[mimoo]

Tell me if I'm wrong but it seems to me that *.opam files match your requirements.

I don't think so because:

• they are ignored by our dune commands to setup and build and run tests
• they don't care about the OCaml version I believe
• they don't allow you to specify pins

This is not true anymore with the Nix setup thanks to @balsoft ! Currently if you setup nix on your system, you can just nix develop mina and develop while being certain all your dependencies are correct (and you cannot mess up the switch).

this is cool! I still haven't setup nix on my mac, I'm wondering at which point it would make sense to move there to build and run tests :o I've heard it needs some time investment

[Firobe]

they are ignored by our dune commands to setup and build and run tests

I'm not sure what you mean or want here? Can you give an example of what you'd want in terms of interaction?

they don't care about the OCaml version I believe

You can specify the version of OCaml needed like any other package

they don't allow you to specify pins

You can! With the pin-depends clause

[Firobe]

Just to give an example of a what a complete opam file can look like, here's one where we specify the OCaml version and pin some packages (from an unrelated side project)

opam-version: "2.0"
depends: [
  "ocaml" {= "4.14.0"}
  "6502-ml"
  "tsdl" {> "0.9.0"}
  "stdint"
  "stdint-literals"
  "fpath"
  "digestif"
  "dune" {>= "3.0" & build}
  "odoc" {with-doc}
]
pin-depends: [
   ["6502-ml.dev"         "git+https://github.com/Firobe/6502-ml.git"]
   ["stdint-literals.dev" "git+https://github.com/Firobe/ocaml-stdint-literals.git"]
]
build: [
  ["dune" "subst"] {dev}
  [
    "dune"
    "build"
    "-p"
    name
    "-j"
    jobs
    "@install"
    "@runtest" {with-test}
    "@doc" {with-doc}
  ]
]
dev-repo: "git+https://github.com/Firobe/NES-ml.git"

[mimoo]

they are ignored by our dune commands to setup and build and run tests
I'm not sure what you mean or want here? Can you give an example of what you'd want in terms of interaction?

Basically all of our errors (someone posted another one on slack today) come from: "I ran dune build or dune runtest and I'm getting some cryptic compile error" which usually is solved by nuking your switch and re-importing the file. The way it works in Golang/Rust is that your build command will always use dependencies as specified by some dependency file (and hopefully a lock file). So these kind of errors never happens.

At the very least dune should tell you "your switch dependencies are at the wrong version" or "your ocaml version is wrong"

But really, ideally, I should never have to worry about having messed up my switch. Dependencies and pins listed in some file should be the source of truth whenever I run build or test or other commands that will need to use dependencies.

Just to give an example of a what a complete opam file can look like

isn't all of that completely ignored by dune?

[Firobe]

The discussion continued a bit on Slack, the gist is the dune and opam are not enough in sync.
#11212 is the issue of implementing the initial solution of putting submodules as opam exports in the switch, and I'm repurposing this issue as creating proper opam files for the code base with dependencies, and making a thin wrapper around dune to ensure the state is correct.

[ylecornec]

Regarding specifying dependencies in opam files, it seems that the opam-dune-lint tool should be able to do this automatically. However it does not work with dune 3 at the moment (there is an issue about it). And there is also this related dune issue about checking that the library used are the one declared in opam files (or the dune-project one).

It seems that we can wait until this is sorted out to specify the dependencies in the opam files, so until then, we could use a wrapper around dune that works with the current opam.export file (see this PR).

[mimoo]

yeah I tried using opam-dune-lint in the past and it doesn't work well. I agree that a dune wrapper is the best option. I really don't want to call dune myself considering all the pitfalls.