pull overwrites everything

[hannesm]

I use mirage, which uses opam-monorepo. Now I sometimes want to improve one package and test changes within a unikernel setup. How I achieve this is rm -rf duniverse/my-package ; cd duniverse ; ln -s ../../my-package . ; cd .. -- so removing the stuff duniverse pulled, and creating a link to my local development version.

Now, when I have 5 packages symlinked, eventually I want to revert some of them to the upstream version. What I attempt to do is rm duniverse/my-package - removing the symlink. Now, a opam monorepo pull removes all symlinks.

The question I have now: is there a better supported workflow to achieve the same? Is there a way to tell "opam monorepo" to only pull non-existing things, rather than overwriting everything?

[Leonidas-from-XIV]

opam-monorepo operates with the assumption that it has the duniverse folder for itself, e.g. the dune file in there mentions that it will likely get overwritten (but so will other folders, there is no logic in place to deal with symlinks or preserving anything in there). This has the advantage that it always starts from a clean slate, no potentially hard to debug issues with stale files from e.g. previous pulls etc.

A possibility for the workflow that you're attempting is this: move the projects that you want to work on into a different folder, like vendor/ and then modify them there. This has the downside that on the next pull they will be pulled in again and dune will complain about duplicate definitions, so you'll have to remove them from the duniverse/ again. A potential workaround could be marking the packages you're working on as opam-provided but that might have unintended consequences for the dependencies of those packages, being also defined as opam-provided, so a different folder and occasional deleting seems to me like the most practical way to achieve your workflow.

[hannesm]

Thanks for taking your time to answer this question. The outlined solutions look pretty strange to me. So is "opam-monorepo" not really meant for a workflow where dependencies are worked on? I'm still struggling with the more detailed semantics and workflows "opam-monorepo" attempts to solve.

[Leonidas-from-XIV]

Personally having "pull creates a clean slate" is something I find extremely useful (and in fact being able to just nuke the installed packages and start from scratch instead if debugging the local switch or having to do a costly opam switch remove && opam switch create).

How would an option like opam-monorepo pull --preserve-symlinks sound to you? That would still require the repos you want to develop on to be symlinked (since there's no simple way to distinguish locally modified folders) but maybe that's an acceptable compromise? It's probably better to only have it symlinked anyway, since if a project is in duniverse/ then it has (vendored_dirs) enabled thus building like in release-mode.

[hannesm]

Ok, that may work. But the question in the room still is whether a workflow like this is supported? Or how do people use "opam monorepo" with local changes?

The other question is what the semantics of "opam monorepo" is? Will it be able to figure out that my symlink from "my-dns-directory" is the same library/packages as the tarball from a "https://github.com/mirage/ocaml-dns.git" release?

As mentioned in #364 I still don't understand how "opam monorepo" is 'designed' to be used with local changes in different repositories (which I understood is the primary goal of such a monorepo approach) and how such local changes are to be upstreamed? Surprisingly, in my workflow I even sometiimes add and remove dependencies (thus my need to execute "opam monorepo" a second time)?