When I started worch I had heard about guix. My somewhat brief evaluation left me concluding that it was not suitable for exploitation. I’ve recent reacquainted myself with it and am very impressed with its progress. On the whole, I think guix could make a suitable replacement for worch but at a cost of radically changing how many are used to managing packages built from source. After accepting these changes there are also some perceived deficiencies that would need to be addressed in guix before it would be suitable. As different as the two are, they also share a lot of commonalities. The rest of this document goes into some thoughts on this.
Despite being developed independently I was amazed at how much commonality worch and guix have (maybe we are spiraling in on an Hylaean ideal form?).
- they both have goofy names
- they both are “meta build systems” operating on packages’ native build systems
- they abstract the installation into steps (or “phases” in guix terms)
- they parameterize the build to keep distinct description and action (guix allows some blurring)
- guix’s functional approach is similar to worch’s idempotency
- both allow reuse of previously builds
- both allow expression and satisfaction of dependencies (with some differences)
- both are user systems (guix has some root-level requirements)
Some of the features in guix are good and even preferable given different metrics but are considered unwanted in worch. The list below doesn’t imply that guix is not suitable (even as a worch replacement) but that these features will not be considered for adoption into worch, per se.
- configuration language
- Guix uses Scheme for its configuration language. It’s debatable if a full-fledged language is better or worse than a declarative description. As a programmer I always want to trend toward the former but it’s also gotten me into trouble as configuration becomes more and more complex. Also, as much as I like Scheme and LISP in principle and have been trying to learn it for years, I find it very difficult to grok. The ConfigParse language used for configuring worch is, I think, unarguably easier for the majority of people to work with and one may still write waf tools in full Python if/when needed.
- layout
- The installation layout is predefined and not under user control. This is somewhat mitigated by the fact that it’s a rather smart layout. It does not seem suitable for users that want to maintain multiple installations at the same time. It’s unsuitable for emulating some existing installation infrastructure (eg UPS products area) although one could imagine emulating this with yet more symlink redirection.
- comprehensiveness
- With guix you can bootstrap an entire user-space OS. This is pretty amazing but not something worch needs to do (now). It would result in building/installing many packages that people “feel like” the OS should supply. It does look possible to simply leave unmentioned whatever packages one wants to take for granted in the “inputs” field. This will let one create a suite of guix configuration files at a level that worch targets so it’s not a fatal strike against using guix as a worch replacement. I think building full(er) software distributions even to the point of including most of the OS is something that is very attractive as a means to capture all that went into any given Experimental result. I feel this feature of guix is ahead of its time. To shun this approach already comes at a cost of an inflation in the number of “platforms”. With guix the kernel+machine is all that really needs to be distinguished. Current systems “save” time by not building OS-level packages but this means they are subject to whatever differences there are in those packages from distribution-to-distribution. As more platforms are supported it mean more time must be spent rebuilding higher-level software to accommodate these changes. Consider the differences between the same machine running Debian or Sci. Linux. There are significant differences due to Debian’s multi-arch support and SL’s biarch for example. There are also the myriad of detailed differences in package versions. If both installed down to
libcwith guix (which is what is done) then there is no practical difference in the binaries needed to support these two native distributions. This personal control of the OS-level packages also allows VMs or other vitalization to be significantly smaller. Given the work on booting to guix (or ratherdmd) the VM container can be very tiny indeed. Again, this is all ahead of its time. I won’t be surprised if this type of thing becomes the norm in a few years. - hashes
- The “real” installation area of a package is named after a hash of its contents and the “inputs” (source). This is an incredibly powerful tool to assert provenance but not something that worch will consider at this time. This is another feature that I think is a bit ahead of its time and will become something of importance in the near future.
- daemon
- This provides a gatekeeper on populating the package binary storage area so that multiple users on the same computer can initiate package builds and share the results. It also helps for building each package in an isolated
chroot. For better or worse, worch assumes a site has a centrally managed package area with a few individuals responsible for its population and all other use is via read-only access. - offload
- Guix can send build jobs to other computers. This is great for building a full distribution where the number of CPUs needed is larger than typical computers provide. However, in the case of the high-level suites that worch targets, the size and dependencies in even the largest one are such this limit is not reached. A 4-core machine can be maxed out but the 32-core nodes which are commonly available on institutional machines are not.
Guix is targeting a somewhat different problem than worch but they are close. If guix were to replace worch some additional features are probably needed:
- variants
- On a given platform, a system may used a shared installation (guix’s
/gnu/store) with the same packages having some variation. Debug/opt is one, which guix does address at lease for GDB’s use of alib/debug/directory. Other variants include different build options for ROOT (include xrootd, or don’t) but these can maybe be handled by finer-grained packaging (“multiple outputs” in guix terms). Packages based on different C++ compilers need accommodation while, ideally, all being able to share the same C-based packages. - versions
- versions can be given when using guix to install a package but there is no mechanism to define a suite of versioned packages. This could be handled by a layer on top of guix. There was also (that I found) no way for a package to declare a versioned dependency.
- multiple platforms
- I didn’t see how one could (or should) use a common guix store to support multiple platforms. This is often done for sites with a heterogeneous collection of hardware. I think it is somewhat suspect to combine the binaries for these diverse platforms into one area, but it’s done. This is probably not an important deficiency.
- multiple installations
- I don’t see how I can maintain multiple, independent and concurrent installations. Everything is all together under
~/.guix-profile/. I suspect I’m missing a way to have multiple instances of this profile directory. This is very important as everyone multitasks on different experiments and even w/in one experiment, multiple versions are needed.
Some things are neither here nor there.
- locality of package description
- By construction, worch has a central suite description. Multiple suites can interleave but this will work only “accidentally” if the authors of the suites take care. On the other had, guix describes each package in isolation. Dependencies are expressed by referencing other packages by name following some universally agreed convention. This assures a wide variety of possible final package distribution at the loss of any a’priori assertion (again, which could be a layer on top of guix).
Some of the ideas of guix are ripe for incorporation into worch, or at least for contemplation.
- layout
- The symlink-based flat-hierarchy layout is nice as it allows trivial user environment setup and avoids wholesale environment variable pollution. If such a feature is developed for worch installs I see it as a separate layer and not part of worch itself.
- garbage collection
- The ability to prune unwanted installations is missing from worch and is a strongly needed feature. (AFS space holding central installations is expensive). This has obvious connections with the symlink-based layout feature.