This document is the final submission report for my Google Summer of Code project. My project is about adding Parameterized Packages to GNU Guix. The original project proposal can be viewed here.
GNU Guix is a transactional package manager and a GNU/Linux distribution. This means that all package management operations in Guix are pure functions of the package’s dependencies, which provides a lot of benefits such as:
- preventing dependency hell
- seamless rollbacks
- transactional determinacy
- reproducible actions
This makes Guix uniquely suitable for tasks that value reliability and reproducibility, such as High-Performance Computing. One unique trait of Guix that makes it stand out from other transactional package management systems is its commitment to ensuring that all packages in its repositories can be built from source. My project takes advantage of this trait to introduce a generic interface for compile-time options.
Package Parameterization refers to adding compile-time parameters to packages. To give a concrete example, imagine that you are running a web server that hosts a personal website. As of such, you do not require graphical display capabilities, however because you can only get programs as binaries in the project’s repositories, you cannot remove the graphical display capabilities from programs despite the developer providing that option when compiling the package. A huge number of applications come with such options, and in areas such as Scientific Computing and Embedded Systems being able to harness them matters a lot. For example, if you wish to compile software for RISC-V, it’s necessary to disable tests for some applications. This project makes it possible to define both generic and package-specific compile-time options, and also makes it easy for users to enable or disable them through a convenient command-line interface.
This project has achieved almost all of the goals in the initial proposal and has exceeded some of them.
The code works and users can parameterize packages and specify specific parameter combinations with the help of the --with-parameter package transformation option.
While it is easy to prototype code with S-expressions, it is important to have data structures in place as they provide much more robust debugging capabilities. I first began writing the parsers and resolvers with just S-expressions, but there are now proper SRFI-9 Record Data Structures for Parameters, Parameter Types and Parameter Specifications.
The initial code only accepted boolean (on or off) parameters, but it is now possible to specify parameters that can take any number of finite values and also optional negative and default values (in the case of booleans, they are off and on respectively).
The syntax for specifying parameter lists and the accompanying parsers and resolvers went through multiple changes as the design slowly matured. Based on the feedback from Guix Maintainers and my mentors, I moved away from the more DSL-esque syntax of my original proposal to a more uniform Scheme-like syntax which is a bit more verbose but a lot easier to understand and read.
It is now possible to specify global parameters that can be applied to packages without being in the package’s parameter specification. I have also implemented a feature known as parameter predicates, which are functions that take in a package and return true or false. If the value they return is true, the given global parameter is applied to the package and otherwise it is not. These are useful when a parameter is build-system specific for example. By default, the predicate function for all global parameters is (const #f), which always returns false.
Parameters can now depend on other parameters (local or global). This is a very important feature in packages with a lot of parameters like GNU Emacs; for example Emacs’ tree-sitter functionality relies on the emacs-next.
It is useful to be able to throw errors and halt compilation if these dependencies are not satisfied.
In the original proposal, it was only possible to have transforms for specific parameters. It is now possible to not only have package transformations, but also general functions that return packages that are activated when a parameter is at a particular value. Users can also add build system requirements to these options. They can also use keywords such as #:package-name or #:parameter-value to substitute-in the current package’s name or the parameter’s value inside these transforms and functions; this is quite useful as a lot of transforms require the package’s name to work.
These all-purpose parametric package transformers are called parametric variants, as they help create package variants based on parameters and they exceed the scope of regular package transformations.
I have not only added the parametric if statement from the original proposal but also a whole host of other parameter-based conditionals like parameter-match and parameter-modify-inputs. These are very useful for more complicated packages, where it’s not possible to parameterize them with functions or transforms alone. They are much more powerful than simple conditionals as users can specify the specific values of any parameters they’d like them to activate at and it’s possible to create complex combinations to mirror real-world needs.
Package Transformations are robust interfaces for transforming Guix packages. The --with-parameter transform I have implemented lets users set the value for any global or local parameter for a given package. For example, guix install parameterized-emacs --with-parameter=parameterized-emacs=next=on will install the package parameterized-emacs with the parameter next set to on. It is possible to chain multiple such options together, and also use them in files to define new packages that are transformed versions of other packages.
Finally, I have implemented a simple search UI for parameters. When searching for a particular package, users will be able to see the parameters in the parameter-spec of that package along with the type of that parameter. This will be useful for users wanting to see what all parameterization options a package supports.
Currently, I’ve given three examples for global parameters. I would like to write more example parameters and parameterized packages to help people understand how to write their own parameterized packages.
Currently, parameters can only depend on other parameters. In the future, I’d like to make it possible to also add package dependencies for parameters, as you often need to pull a new package to add a new feature.
Currently #:transform, #:build-system and #:lambda are the only parameter variant options. I’d like to add a modify-inputs option too as currently users need to use parameter-modify-inputs inside the package definition instead.
There will be some combinations of parameters that users might need often, and it could be useful to have substitutes (binaries) available for them. It could also be useful to make these ‘canonical’ combinations discoverable.
I initially wanted to add a function that measures the combinatorial complexity of a parameter-spec, however I later found out that this problem is #P-complete. Carelessly writing a function that does this could lead to some problems down the line, so I decided to implement a proper function for it later.
Currently, you can only see the parameters present in the parameter-spec of packages. This is to avoid querying the predicate of every single global parameter for all search results. I’d like to add a UI that lets you see all possible parameters, even ones not in the spec, for a single package.
I’d like to write a convenience DSL for writing parameters. I had written a few functions for the same, but after some feedback I realized that having the underlying code be extremely macro-heavy is not a good idea.
I need to write proper documentation and unit tests for parameters and all associated functions. I plan on doing so when the project is polished, extensively tested and is in a state where it’s ready to be merged to main.
The style of Scheme used in big, professional projects like GNU Guix is very different from the type that you’d use in small projects, and the majority of my learning experiences were understanding what kind of coding practices are used in professional Scheme projects. I learned that it’s important to use concrete data structures instead of lists because it’s easier to handle errors that way, and that using match instead of ca*r or cd*r can greatly increase the readability and maintainability of code.
I faced quite a few challenges with writing the resolver for enumerated types, as I was initially just trying to adapt the boolean resolver to fit enumerated types and that did not work out. In the process of writing it, I also learned to leverage filter, fold and map more. The macros were quite challenging to write too; I was used to writing defmacro-style macros, so it took me some time to pick up hygienic macros. However, once I learned how to use them properly they were extremely rewarding to use and in my opinion also much easier to debug compared to defmacro-style macros.
I also faced a few challenges when it came to the design of the functions. Ultimately, this project is akin to a library that will be used by a wide variety of users and it was hard to find the right balance between power, user-friendliness and flexibility.
One such example is the predicate field of the package-parameter record; the technically correct way to specify it would be to only accept functions, and have users pass (const #t) or (const #f) when they always want it to return #t (true) or #f (false). But it was much more user-friendly to just accept #t and #f and convert them to (const #t) and (const #f) under the hood, as users might not be aware of const’s existence.
Another small difficulty was that Guile Scheme’s documentation is a bit lacking in examples, and the examples presented can sometimes be hard to understand. This is especially true for macros and other more advanced language features.
Apart from challenges with code, I found incorporating community feedback to be a bit difficult. People often have opposing preferences and it can get a bit hard to choose one thing over the other. Luckily, I had people willing to guide me and help me with making these decisions and I’m satisfied with the final outcome.
The project got a bit difficult right after the midterms because I had to rewrite quite a bit of the code I had written to fit the new final design spec. It was also a bit painful to let go of some of the features I had already coded in, but I have plans of reusing a lot of them when I write a convenience DSL for parameters.
Communicating the frequent changes in the syntax for parameterization was also a bit difficult, as people would often get confused between the old and new syntax. I plan on writing a completion blog post to clarify all of these doubts, and updating the previous posts to remove deprecated syntax. I also did not do a great job of communicating new features, and people ended up asking me if I was planning on implementing things that I had already implemented.
- Post on the GNU Guix Blog
- The First Post about Parameterized Packages
- The First Update
- The Second Update
- Parameterization Patch - this contains the final code that will be submitted to main after a stricter quality review.
- Github Mirror of the Patch
- Main Work Repository
- Github Mirror of the Repository
I’m extremely grateful to my mentors Pjotr Prins and Gabor Boskovitz for guiding me throughout this journey. It’s very difficult to maintain a consistent work output and have proper direction when working on a project with such a huge scope, but I was able to do that thanks to their help and guidance. I’d also like to thank Ludovic Courtes, Arun Isaac and Efraim Flashner for their constant feedback, suggestions and encouragement. A lot of the improvements in the code quality are thanks to their help. The Guix community also deserves praise for being an excellent, welcoming community with many helpful and creative individuals. I’m very happy that my first big experience with Open Source software development was with GNU Guix.