RFC: chef, stable package manager based on nimble+homebrew

[timotheecour]

chef

chef is a nimble package (hosted in https://github.com/nim-lang/chef) that maintains a (growing) set of third-party nimble packages that are tested to work well together and with supported versions of nim compiler (1.0, 1.2, and latest patch release in nim devel), across supported platforms (linux, osx, windows and more). It does the following:

• enable reproducible builds by specifying version-controlled specific revisions (url+tag/hash) for each supported package and their recursive dependencies
• runs its own CI that installs (via nimble install foo@revision) each package, and a specified test command
• it may eventually subsume the role of important_packages.nim, by calling chef with appropriate parameters to achieve same results

relationship with important_packages.nim

• important_packages doesn't keep track of specific revisions that are known to work together, and in particular will install different versions of the same package (eg if foo1 depends on bar@1.0 and foo2 depends on bar@1.1); chef specifies a fixed revision for each package
• important_packages can break whenever upstream updates their latest release tag (or even their HEAD, due to a fixable design bug in how important_packages works)

relationship with nimble

• chef is implemented on top of nimble so is not a replacement; it calls it (via cmdline) to install packages at specific revisions. unlike nimble, it maintains a consistent set of revisions that work well together, and manages all that by running its own CI, and provides cmdline+library utilities for users to upgrade their chef-installed packages.

relationship with homebrew

• chef takes some inspiration from homebrew, but is narrower in scope, being focused on maintaining an ecosystem of nim packages that work well together

non-nim dependencies

• by default, chef takes care of installing non-nim dependencies specified in a nimble package (when a foreignDep field is mentioned) by leveraging std/distros.foreignDep (which uses brew on OSX, etc)

chef usage for end users

this is heavily inspired homebrew and calls nimble commands behind the hood

• nimble install chef # installs chef nimble package, available as both cmdline + library

• chef update [stable|head] # updates chef

• chef install foo # installs foo at the revision specified by chef, which is version controlled and can change after a chef update; skip install if foo is already installed even if it was an older version, unless chef determines it needs to

• chef upgrade # upgrade each package to the

chef usage for chef contributors

• chef bump-formula-pr foo@#e1cc021 bar@1.2.3 baz bumps foo and bar to specified hash/tag, and baz to latest release tag, and creates a PR, that runs CI to make sure tests would pass. Once merged, packages foo,bar,baz have been effectively updated in chef HEAD. This is the exact analog of brew bump-formula-pr in homebrew

scratch below

for each supported nim compiler version (guarantees via its own CI that each supported package can be installed

It is usable as a library and command line application,

introduce a new module, for example in tools/chef.nim (usable as library or cmdline), installed by koch tools, to manage, update and test a set of vetted nimble packages (with large overlap with important_packages). It does the following:

• maintain specific revisions (hash or tag) for each package, eg foo@0.9.2 including their (recursive) dependencies

• ensures (via CI) that each package can be installed at their specific revision via nimble install foo@0.9.2, along with a test command (that defaults to nimble test)

• provides cmdline helpers to make it easy to update packages to their latest upstream release tag:

• chef upgrade: upgrade all packages
  Forks can be used if package maintainer goes awol or if a fix is needed and waiting for maintainer is undesirable, or if there's concern maintainer might force push or delete his repository. In this case, the package's url is simply updated to point to the fork (perhaps temporarily until upstream fixes the code).

local override

users can easily override the revisions maintained by tools/chef.nim to fit their needs (eg during development etc), for example using ~/.config/nim/chef.nims or maybe nimble develop

that are tested to work

• introduce a nimble package called contrib hosted on https://github.com/nim-lang/contrib
• contrib is a nimble package that installs (via nimble) third party nimble packages at fixed revisions (a git hash or release tag) that are known to work together in combination
• nimble install contrib installs a binary contrib (analog to koch) that manages those depdencies, runs CI, automatically updates revisions to keep them as up to date as possible while maintaining a green CI. It notfies upstream packages when contrib could not

packages at their specified revisions, and

• each package foo not hosted under https://github.com/nim-lang/ is mirrored under https://github.com/nim-lang/mirror-foo (or if name clashes, https://github.com/nim-lang/mirror-$username/foo for example)
• contrib binary helps automate the process of updating the mirrors,

links

• D20200421T170935
• TITLE: fusion: design discussion · Issue #241 · timotheecour/Nim
• questions · Issue #9 · disruptek/dist
• [RFC] fusion packages · Issue #115 · timotheecour/Nim
• fusion: design discussion · Issue #241 · timotheecour/Nim