Overlord is an experimental build system for Common Lisp, inspired by Redo.
Overlord addresses two problems which might seem unrelated, but which, on closer examination, turn out to the same problem:
It lets you reproducibly specify the desired state of a Lisp system which is to be saved as an image.
Overlord expects to be used alongside ASDF, with ASDF responsible for compiling and loading Lisp code, and Overlord doing everything else.
For more discussion of the thinking behind Overlord, how it relates to Redo and other build systems, consult the wiki.
Advice for users
Overlord is experimental. For the most part, trying to document the API at this stage would be futile. Instead, this README discusses the concepts behind Overlord. If you’re looking for the current syntax, consult the test suite and the files it uses.
Before loading Overlord, it would be a good idea to make sure you are running the latest version of ASDF.
Note that, to run the test suite, you will need to
download Core Lisp, and, if not on Windows, you must have the
touch program in your search path. (On Windows, Powershell is
Overlord stores its persistent data structures in a cache directory.
On Linux, this is
$XDG_CACHE_HOME/overlord. The data structures
stored there are versioned. Since this version number is increasing
rapidly, it might worth checking the cache directory from time to time
to delete obsolete files.
Overlord is developed and tested on Clozure and SBCL. In the future it may support other Lisp implementations, but that is not a priority. Lisp implementations that do not support image-based persistence (e.g. ECL) are unlikely to receive support.
When I say “experimental”, I mean it. Anything may change at any time.
Here are some examples of how to make direct use of Overlord:
Proctor. Proctor treats tests as build targets, allowing you to precisely specify their dependencies and re-run tests only when necessary.
Vernacular. Provides a module system for embedding languages, with arbitrary syntaxes, into Common Lisp systems.
One thing that might not be obvious about Redo-style build systems is that they afford unusually good opportunities for parallelism.
Overlord supports building in parallel using threads. Targets can
request that their dependencies be built in parallel by using
pdepends-on instead of
depends-on. However, threads are not
enabled by default; this feature is still an experiment within an experiment.
If you want to try building in parallel, execute:
(setf (overlord:use-threads-p) t)
It is not recommended that you try parallelizing targets that call the Lisp compiler. (This includes importing modules.) On SBCL the Lisp compiler is protected by a global lock, and trying to use it from multiple threads can result in deadlock.
Overlord only uses parallelism when it is explicitly requested, but even without parallelism, it tries to discourage reliance on side effects by, whenever possible, randomizing the order in which targets are built.
Freezing the Lisp image
During development, as targets are defined and re-defined, and rebuilt or not rebuilt, the actual state of the Lisp world will drift away from the one specified by Overlord’s dependency graph. Before dumping an image such discrepancies must be resolved. It is obviously undesirable, for example, for an image built on one machine to try to lazy-load a module on another machine where the source of that module is unavailable. (Actually it would be a disaster, since that source file might be provided maliciously.)
Thus, before an image is saved, Overlord needs to do two things:
Finalize the state of the image by making sure that all defined targets have been built.
If you use
uiop:dump-image to save the image, you don’t need to do
anything; Overlord will finalize the state of the image, and disable
If you are using implementation-specific means to save an image,
however, you will need to arrange to call
overlord:freeze before the
image is saved.
The default policy is to allow the restored image to be unfrozen, and
development to be resumed, by calling
overlord:unfreeze. This is
probably what you want when, say, saving an image on a server. In
other scenarios, however — like delivering a binary – you may want to
strip the build system from the image entirely. This is possible by
changing Overlord’s “freeze policy”, using the
;;; The default: can be reversed after ;;; loading the image by calling ;;; `overlord:unfreeze`. (setf (overlord:freeze-policy) t) ;;; Irreversible: before saving the ;;; image, Overlord should destroy its ;;; internal state. (setf (overlord:freeze-policy) :hard)