Improved documentation for, and questions about, the new npm-like workflow

[ELLIOTTCABLE]

So, I'm trying to figure out a workflow that's similar to what I'm used to with under-development Node packages. (Happy to see OCaml taking some of the good things the JavaScript community has to offer! Super-cool.)

Some questions:

1. Is opam lock the officially-sanctioned / suggested way to go about achieving stable dependencies? It looks like it never got merged in to the parent project before the release of 2.0, so …

2. Is there an equivalent to npm install --save <PACKAGE>, or is one on the roadmap? Currently, I find myself basically using opam install <PACKAGE>, then manually adding the package-name and the version that I saw the previous command install to the opam file manually.

3. I'm really missing semver's caret ^ and tilde ~ operators for comparing versions. Currently, I'm having to write my version-constraints out very manually (adding insult to the injury of №2 above):

   "uchar" {>= "0.0.2" & < "0.1.0"}
   "gen" {>= "0.4" & < "0.5"}
   

4. How do I update the packages in a project to the latest versions, while retaining compliance with the opam-file declarations?

   i.e., with the above constraints, if I run opam upgrade, I get this:

   The following actions will be performed:
     ∗ install   base-bytes base           [required by gen]
     ↗ upgrade   dune       1.1.1 to 1.4.0
     ↻ recompile jbuilder   transition     [uses dune]
     ↗ upgrade   gen        0.4 to 0.5.1
   ===== ∗ 1   ↻ 1   ↗ 2 =====
   

   (note, of course, that gen is being upgraded past the limitation declared in the opam-file)

5. Is there an equivalent of npm's devDependencies? Software not required for building, nor at runtime, but necessary to contribute to the library or change the source-code? (Think Merlin, etc.)

Basically, I'd really like to see a page of the documentation dedicated to this newly-supported workflow, with information both from the package-creator's point-of-view, and the package-contributor's point-of-view. (There's the page on the GitHub wiki; but it's really out-of-date.)

Thanks for reading! <3

# opam config report
# opam-version      2.0.0 
# self-upgrade      no
# system            arch=x86_64 os=macos os-distribution=homebrew os-version=10.14
# solver            builtin-mccs+glpk
# install-criteria  -removed,-count[version-lag,request],-count[version-lag,changed],-changed
# upgrade-criteria  -removed,-count[version-lag,solution],-new
# jobs              7
# repositories      1 (http) (default repo at e8a1a5a5)
# pinned            0
# current-switch    /Users/ec/Sync/Code/excmd-parser

[Khady]

1. Yes. 2. No, packages must be added to the opam file by hand. 4. Mystère et boule de gomme. Opam upgrade && opam install pkg.opam --deps-only && opam lock pkg.opam With pin-depends I think it would be different. Some packages will be upgraded and downgraded tho :( 5. You can use {dev} annotation on a dependency. It will be only installed if the package is installed using a pin or opam install ./pkg.opam or the source is not stable (a git repo instead of a tarball for example) I tried to document the closest I got to the npm workflow in those blog post: - https://khady.info/opam-sandbox.html - https://khady.info/opam-npm.html The npm workflow is also mentioned on the opam wiki. If opam is willing to support this workflow, it would be great to store the commit hash of packages that are installed from a dvcs. To store the pins created by pin-depends in the lockfile too (it might be done already I don't have a computer to check right now). And to improve the story regarding pin-depends. It's pretty troublesome to have to run opam upgrade after opam install pkg.opam --deps-only to get the pin packages installed or upgraded. At work I'm trying to have a group of developers using exactly the same packages and it's a nightmare. Honestly if this kind of workflow isn't supported by opam soon, I have no doubt that we will end up using esy instead. It misses some importants bits too. But it is still easier to get a consistent state for everyone in the team.

[rjbou]

Thanks for reporting!

3. You can see this issue support for semver version constraint operator #2976

4. You can also pin the local package, to have its constraints taking into account in the upgrade operation of the switch.

GitHub wiki; but it's really out-of-date

yes, completely! The documentation is now on the opam webite opam.ocmal.org and in the manpages for each command.

At work I'm trying to have a group of developers using exactly the same packages and it's a nightmare.

@Khady why? locking the opam file, isn't enough to have everyone the same packages installed? If you need to have upgrade its dependencies, you can have a job that upgrade the locked file and pushes it in a vcs. As all your team have the locked package pinned, at next update, they will have their dependencies updated also.

[Khady]

No, lock files are not enough. They don't store commit hash for packages where the source is a git branch. It doesn't include pins either.

[ELLIOTTCABLE]

Hm. I don't quite understand pinning, apparently. Why would I pin the package being developed? As it's under development, it's obviously not actually at the version stated in the metadata — I suppose it's always at version + 0.0.1, implicitly? I need to play with this a little more, I think.

So it sounds like the only unanswered question here is №2 above; I'll open that as a new, separate issue, and you can consider this Issue focused on the lack of documentation around these intentions? (=

[hcarty]

@Khady I've been doing some experiments with opam-lock and a project.opam.locked file. It seems to capture pins as long as they are installed + pinned in the current switch. If a pin targets a specific commit that seems to carry over too.

[Khady]

I have sqlgg pinned using pin-depends. It is installed. But it doesn't appear in the lock file.

$ opam pin
sqlgg.git        git  git+https://github.com/ygrek/sqlgg#3bde70b5a2442c6eb8f2b451394fe71f9839ea59
$ tail -3 example.opam 
pin-depends: [
  [ "sqlgg.git" "git+https://github.com/ygrek/sqlgg#3bde70b5a2442c6eb8f2b451394fe71f9839ea59" ]
]
$ opam lock example.opam
Wrote /home/louis/Code/demo/example.opam.locked
$ cat example.opam.locked  | grep sqlgg
$ cat example.opam.locked  | grep git | tail -3
  "devkit" {= "git"}
  "ragel" {= "6-git"}
  "tld" {= "git"}
$ opam show sqlgg

<><> sqlgg: information on all versions <><><><><><><><><><><><><><><><><><><><>
name                   sqlgg
all-installed-versions git [ahrefs]
all-versions           0.2.4  0.2.5  0.3.0  0.4.3  0.4.4  git

<><> Version-specific details <><><><><><><><><><><><><><><><><><><><><><><><><>
version      git
pin          git+https://github.com/ygrek/sqlgg#3bde70b5a2442c6eb8f2b451394fe71f9839ea59
source-hash  3bde70b5
url.src:     "git+https://github.com/ygrek/sqlgg#3bde70b5a2442c6eb8f2b451394fe71f9839ea59"

[Khady]

4. You can also pin the local package, to have its constraints taking into account in the upgrade operation of the switch.

Sometimes is it not possible or not convenient to pin the local packages, unfortunately. And managing pins in a team is troublesome.

[rjbou]

@Khady You need to have the pin-depend package also on your depends field to have opam handle it as a dependency (doc updated).

[rjbou]

What is the status of this issue ?