Common Lisp in Practice
Introduction
One of the things which has kept Common Lisp out of my day-to-day toolbox is a lack of clear instructions how to get up and running with it — not in the REPL, but building programs that can be called from the shell. I tend to reach for Python or Emacs Lisp for a lot of these cases, since they’re readily available and I understand them, but I’ve always felt that Common Lisp could be a potent tool for these situations.
After reading my friend Steve’s Road to Common Lisp, I was inspired to figure this out. With some patient help from him, I believe I’ve finally got a handle on it.
Building a project in Lisp can be confusing, because Lisp itself works so differently to other languages, and this affects how builds work. While Lisp is compiled to machine code like many languages, the mechanisms are somewhat different.
Goals
This isn’t a tutorial on Lisp programming, because there are many great resources for that already. It doesn’t rathole on every possible approach or permutation you could possibly use, but tries to tread and illuminate the beaten path. It’s an attempt to explain the non-obvious nuts-and-bolts of building Common Lisp programs: Where to put your source code, how to make a binary, and how to use libraries.
If you’d like to run the example code, you’ll need to install Steel Bank Common Lisp (SBCL) and Quicklisp. The Quicklisp installation is somewhat strange if you’re coming from other languages, so it’s also fine to read along and see what things are like before investing in a Lisp environment.
I wrote this in a literate programming style using org-babel. All program output should be very, very close to what you’d see if you ran those programs. The original Org document and source code extracted from it are available in my Clip GitHub repo.
Background: Lisp environments & images
A typical compiler forks a new OS process for each file, producing a binary artifact corresponding to its input, then combining those into a final binary which you can run.
Common Lisp is much more comprehensive and tightly integrated than other languages. Rather than separate binaries for compiling, linking, and debugging, these features are built into the language itself and can be used by any Lisp programs, including yours.
When you start Common Lisp, it initializes a Lisp environment in your computer’s memory, then evaluates a toplevel function. The environment contains the Lisp language and tools; the standard toplevel is the REPL. If you type code into the REPL, or load code from a file, it’s added to the environment and can be used by anything else inside it.
This is an important point to understand. Nearly every other language is either unusable without multiple binaries which do different things, or ships with a significant amount of functionality locked up in programs which have to be run as independent processes.
For example, Python 3 ships with five binaries:
dpkg -L python3-minimal | grep -c /bin/5
OpenJDK has 21:
dpkg -L openjdk-8-jre-headless | grep -c /bin/21
GCC has 16:
dpkg -L gcc | grep -c /bin/16
And in order to actually use GCC, you need binutils, which has nearly 40 more:
dpkg -L binutils | grep -c /bin/37
Contrast this with Steel Bank Common Lisp, one of the more popular implementations of the language:
dpkg -L sbcl | grep -c /bin/1
Just one, /usr/bin/sbcl. Nearly everything you can do happens
inside its environment. Anything else is an option given to
sbcl(1).
Another thing that’s different is that the environment can be saved to disk in a Lisp image (or “core”), then restored from it at a later date[fn:1]. When you save the image, you can specify a toplevel function other than the REPL which should be evaluated.
To make an executable program which you can run from a UNIX shell, you load your code into the Lisp environment, then create an image with the toplevel set to your entry point.
Version 1: Quick & dirty
The goal is to make a traditional “Hello, World” program which will:
- Run from a shell.
- Use the first argument given to it as the name of the person or thing to greet.
Starting from the ground up, a function to create the greeting:
(defun greet (whom)
"Create a greeting message for WHOM."
(format nil "Hello, ~A." whom))Trying this in the REPL shows that it works:
(greet "World")"Hello, World."
The toplevel function
Satisfying the first requirement, running from the shell, means a toplevel function is needed — this will be evaluated when the image is restored.
I named the toplevel function MAIN, but it can be called anything,
because the toplevel function is explicitly specified when the image
is dumped. Any function which accepts zero arguments can be used as
a toplevel.
(defun main ()
"Greet someone, or something."
(write-line (greet (car (uiop:command-line-arguments))))
(uiop:quit))There are two functions in here that may be new to you,
UIOP:COMMAND-LINE-ARGUMENTS and UIOP:QUIT. These are part of
ASDF, which I’ll cover in a bit, and provide a portable interface to
Lisp-implementation- and OS-specific behavior. They do what they
say on the tin: COMMAND-LINE-ARGUMENTS evaluates to a list of
arguments given to the Lisp image, with each list element containing
a single argument; and QUIT terminates the process.
Packages
The next piece to get a handle on is packages. Packages are
containers for symbols — programmer-defined functions like MAIN
and GREET, library functions, and the language itself.
When the REPL starts, it plops you into the COMMON-LISP-USER
package, a scratch area you can tinker in without wrecking the whole
environment[fn:2].
For the Hello World program, it should be in its own package[fn:3],
which I’ve creatively named HELLO.
(defpackage :hello ; Define a package and name it HELLO
(:use :common-lisp) ; The package needs Common Lisp
(:export :greet :main)) ; This package has two public
; symbols, GREET and MAIN.This can seem weird, because the declaration is a forward reference.
The package has to be defined with DEFPACKAGE before it can be
made active with IN-PACKAGE, and the package has to be active
before anything can be defined in it. So it has to be like this.
The :USE form tells Common Lisp that symbols from the
COMMON-LISP package should be made visible inside your package.
The form expects a list, so if you need multiple things, you’d do:
(:use :common-lisp :foo :bar)This has nothing to do with loading those packages — they have to be loaded already, or you’ll get an error. This can be surprising for those used to other languages, since many treat loading and making visible in the current file or namespace as a single operation.
The entirety of the Common Lisp API exists inside the COMMON-LISP
package, and none of those symbols are visible unless you say you
want them, so you’ll want this in every DEFPACKAGE. This isn’t
needed in the REPL, because the COMMON-LISP-USER package it starts
you in uses the COMMON-LISP package.
The :EXPORT argument enumerates the symbols of the package which
should be visible to other packages. It can also contain
non-exported symbols for internal use, but the exported symbols make
up its API, similar to public / private in C++ or Java.
You may note that I’ve written the name of the package as HELLO,
which it is, but it’s in the code as :hello. For a deeper
explanation on why this is the case, I recommend the chapter on
Packages and Symbols from Programming in the Large. In the mean
time, you’ll just have to trust that it’s right and I know what I’m
doing[fn:4].
Tying it all together
The complete source for Hello World now looks like:
<<packages>>
(in-package :hello) ; DEFPACKAGE only defines the
; package, it doesn't make it
; active.
<<greet>>
<<main>>Building an image
Because the Common Lisp toolchain exists inside the Lisp environment, build scripts for Common Lisp project are written in, you guessed it, Lisp.
(load "hello.lisp") ; Load the code into the Lisp
; environment
(sb-ext:save-lisp-and-die "hello" ; Save a Lisp image
:toplevel 'hello:main ; The toplevel function is
; MAIN, inside the HELLO
; package.
:executable t) ; Make an executable.The =LOAD= function does what you’d expect, it loads the contents of
hello.lisp into the Lisp environment. The second call,
=SB-EXT:SAVE-LISP-AND-DIE=[fn:5] is what dumps the image[fn:6].
For this toy example, this could be put at the end of
hello.lisp, but in a larger project, this is a poor separation of
concerns[fn:7]. It should go into build.lisp instead[fn:8].
Executing the build script with sbcl(1) will produce the binary:
sbcl --non-interactive --load build.lispThis is SBCL 1.3.14.debian, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. [undoing binding stack and other enclosing state... done] [defragmenting immobile space... done] [saving current Lisp image into hello: writing 4800 bytes from the read-only space at 0x20000000 writing 3216 bytes from the static space at 0x20100000 writing 1179648 bytes from the immobile space at 0x20300000 writing 13720752 bytes from the immobile space at 0x21b00000 writing 37027840 bytes from the dynamic space at 0x1000000000 done]
Running it shows the message:
./hello WorldHello, World.
Passing in the name of the current user also works:
./hello $(whoami)"Hello, ieure."
Now that the program works, and you hopefully understand why and how, it’s time to tear it down and rebuild it. Several times.
Version 2: Structure
Having all the code in one file is fine for a toy, but larger programs benefit from more organization. If the core functionality is split from the CLI, other Lisp projects can reuse the greeting function without the CLI code. Having the packages definition out of the way is a good idea, since as a project grows, it can get unwieldy. Since all this work will produce multiple source files, the code making up the main functionality ought to be separated from that used to build the system.
What this should look like is:
build.lisppackages.lispsrc/greet.lispmain.lisp
Even though the organization is different, the contents of the files are almost exactly the same.
build.lisp
(load "packages.lisp") ; Load package definition
(load "src/greet.lisp") ; Load the core
(load "src/main.lisp") ; Load the toplevel
;; Unchanged from v1
(sb-ext:save-lisp-and-die "hello"
:toplevel 'hello:main
:executable t)src/greet.lisp
(in-package :hello) ; We have to tell Lisp what
; package this is in now.
;; Unchanged from v1
<<greet>>src/main.lisp
(in-package :hello)
;; Unchanged from v1
<<main>>The rest of the files are unchanged from v1.
Building and running works the same way:
sbcl --non-interactive --load build.lisp
./hello WorldThis is SBCL 1.3.14.debian, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. [undoing binding stack and other enclosing state... done] [defragmenting immobile space... done] [saving current Lisp image into hello: writing 4800 bytes from the read-only space at 0x20000000 writing 3216 bytes from the static space at 0x20100000 writing 1179648 bytes from the immobile space at 0x20300000 writing 13720752 bytes from the immobile space at 0x21b00000 writing 37027840 bytes from the dynamic space at 0x1000000000 done] Hello, World.
Version 3: Systems
The next yak in this recursive shave is systems. Packages are
part of the Lisp language specification, but systems are provided by
a library. There have been several approaches to defining systems,
but the dominant one at the time of writing is ASDF, which means
“Another System Definition Facility.” ASDF is included in the
contrib/ directory of SBCL, and well-behaved SBCL installations
should include it for you. If not, Quicklisp bundles a version, so
between the two you ought to have a usable ASDF.
Systems and packages are orthogonal, but it can be confusing,� because they both deal with some of the same parts of the project.
A package is a way of organizing the symbols of a project inside the Lisp environment. The contents of one package can be split between multiple files, or a single file can contain multiple packages. From the Lisp environment’s perspective, the only important thing is that certain things live in certain packages.
A system is a description of how to load your project into the
environment. Because of Lisp’s flexibility organizing packages,
you need a system to load the pieces in the right order. This is
visible in the previous build script: The package definition is
loaded first, then greet.lisp, then main.lisp. Any other order
will break. Systems solve this problem.
Throwing in a further complication, one project can have multiple systems. If you write unit tests, you’ll want a system for that, because you need to load different things like your test code and the test framework. Putting this in a different system means that anyone using your library doesn’t drag all that along with it.
Defining the system
Starting from the ground up again, this is the system which defines
the main HELLO, which contains the package definition and GREET.
(defsystem :hello ; The system will be named
; HELLO, same as the project
:serial t ; Load components in the same
; order they're defined. This
; is *per component*, so if
; you have multiple files in
; src/ in the same module,
; you'll want it in there,
; too.
:components ((:file "packages")
(:module "src" ; A module is a collection of pieces of
; your program
:components ((:file "greet"))))) ; Load the greet
; function from
; greet.lisp. The
; file extension is
; implied, and must
; not appear here.And then a secondary system for the binary:
(defsystem :hello/bin ; The name HELLO/BIN indicates that this
; is a secondary system of system HELLO.
:depends-on (:hello) ; This system needs the core HELLO system.
:components ((:module :src
:components ((:file "main"))))) ; ...and includes one
; additional file.In the build script, the manual loading gets replaced with an ASDF load:
(asdf:load-system :hello/bin)
(sb-ext:save-lisp-and-die "hello"
:toplevel 'hello:main
:executable t)ASDF has to be told where to find any system it loads, including this one — it doesn’t look in the current directory. This is a complex topic, but the simplest approach is:
- Use Quicklisp.
- Make a symlink from Quicklisp’s =local-projects= directory, named after your project, which points to your source tree.
This is easily the grossest thing about this entire setup.
rm -f ~/quicklisp/local-projects/{hello,system-index.txt}
ln -sf $PWD/v3 ~/quicklisp/local-projects/helloThe rest of the source is unchanged from v2.
<<v2-main>>Running works the same way:
sbcl --non-interactive --load build.lisp
./hello WorldThis is SBCL 1.3.14.debian, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. ; compiling file "/home/ieure/Dropbox/Projects/cl/lh/v3/packages.lisp" (written 03 SEP 2018 03:56:31 PM): ; compiling (DEFPACKAGE :HELLO ...) ; /home/ieure/.cache/common-lisp/sbcl-1.3.14.debian-linux-x64/home/ieure/Dropbox/Projects/cl/lh/v3/packages-TMP.fasl written ; compilation finished in 0:00:00.001 ; compiling file "/home/ieure/Dropbox/Projects/cl/lh/v3/src/greet.lisp" (written 03 SEP 2018 03:56:31 PM): ; compiling (IN-PACKAGE :HELLO) ; compiling (DEFUN GREET ...) ; /home/ieure/.cache/common-lisp/sbcl-1.3.14.debian-linux-x64/home/ieure/Dropbox/Projects/cl/lh/v3/src/greet-TMP.fasl written ; compilation finished in 0:00:00.002 ; compiling file "/home/ieure/Dropbox/Projects/cl/lh/v3/src/main.lisp" (written 03 SEP 2018 03:56:31 PM): ; compiling (IN-PACKAGE :HELLO) ; compiling (DEFUN MAIN ...) ; /home/ieure/.cache/common-lisp/sbcl-1.3.14.debian-linux-x64/home/ieure/Dropbox/Projects/cl/lh/v3/src/main-TMP.fasl written ; compilation finished in 0:00:00.001 [undoing binding stack and other enclosing state... done] [defragmenting immobile space... done] [saving current Lisp image into hello: writing 4800 bytes from the read-only space at 0x20000000 writing 3216 bytes from the static space at 0x20100000 writing 1187840 bytes from the immobile space at 0x20300000 writing 13721392 bytes from the immobile space at 0x21b00000 writing 37093376 bytes from the dynamic space at 0x1000000000 done] Hello, World.
V4: Using libraries
The final step is to replace UIOP’s basic program arguments with a more full-featured library, unix-opts.
Common Lisp libraries are installed via Quicklisp, and loaded with ASDF. As with other Common Lisp tasks, actually installing the library is done from the REPL.
Quicklisp
Quicklisp is not a package manager like pip or cargo. There’s no
project-specific setup, like with virtualenv or rbenv. There’s
definitely no node_modules.
Quicklisp is more of a caching mechanism, similar to Maven’s
~/.m2 mechanism. A single copy of the code is stored in
~/.quicklisp, and can be loaded into a Common Lisp environment
with ASDF.
As with other Common Lisp tooling, the primary interface for Quicklisp is the Lisp environment.
Installing unix-opts
The Quicklisp documentation discusses this, but I’m going to cover the essentials.
Searching for available libraries can be done with
ql:system-apropos:
(ql:system-apropos "unix")Installing is done with ql:quickload:
(ql:quickload "unix-opts")("unix-opts")
And the library can be loaded with asdf:load-system:
(asdf:load-system :unix-opts)The new MAIN
The new system definition looks the same as before, except
:UNIX-OPTS has been added as a dependency. This makes ASDF load
it when the :HELLO/BIN system is loaded. This does not install
it, this is something you need to do by hand.
Then it’s a matter of using the library. This is mostly copied from the unix-opts example.
(in-package :hello)
(unix-opts:define-opts
(:name :help
:description "Print this help text"
:short #\h
:long "help"))
(defun main ()
"Greet someone, or something."
(multiple-value-bind (options free-args)
(unix-opts:get-opts)
(if (or (getf options :help) (/= (length free-args) 1))
(unix-opts:describe
:prefix "A Hello World program."
:args "WHOM")
(write-line (greet (car free-args)))))
(uiop:quit))Before this works, the Quicklisp local-projects symlink needs to
be updated:
rm -f ~/quicklisp/local-projects/{hello,system-index.txt}
ln -sf $PWD/v4 ~/quicklisp/local-projects/hello… And the ASDF registry cleared:
(asdf:clear-source-registry)After building, the new options parser is working:
sbcl --non-interactive --load build.lisp./helloA Hello World program.
:
Available options: -h, --help Print this help text
:
./hello $(whoami)Hello, ieure.
Conclusion
At over four thousand words, this has been a lot more than I set out to write. The process of learning, organizing, and refining my own understanding has been wonderful. I hope you’ve been able to take away some of that, and will go forth with useful new tools.
Further reading
- A Road to Common Lisp
- CL-Launch is a wrapper to ease running CL from the shell. It can produce binaries, but is more suited to simple one-file programs.
Footnotes
[fn:1] It’s worth pointing out that this mechanism is mind-bogglingly powerful in a way unheard of in most languages. Saving and loading images is an undo mechansim for your entire programming environment. The whole state is saved — any data you loaded, any functions you wrote, it’s all there. Screw it up? Restore from the image you saved when you started. Switching, upgrading, or rebooting your machine? Save an image and you’ll be right back where you left off in seconds. Need to produce repeatable research? Dump an image with your dataset and programs for others to examine and run.
[fn:2] It is absolutely possible to wreck the Lisp environment if your’re not careful, so this is a good thing. For example, if you eval:
(in-package :common-lisp)
(fmakunbound 'defun)It will remove the function binding from the DEFUN symbol, with the
upshot that you can’t define new functions. Oops.
[fn:3] It doesn’t have to be in its own package, but if you’re working on a real program, you’ll want it to be.
[fn:4] I have absolutely no idea what I’m doing.
[fn:5] The SB-EXT prefix indicates that this is a SBCL extension,
rather than part of the Lisp language specification.
[fn:6] The SB-EXT: prefix specifies the package the function lives
in. SB-EXT is a package which contains SBCL-specific extensions
which aren’t part of the Common Lisp language specification.
[fn:7] If SAVE-LISP-AND-DIE was in hello.lisp, and that file was
loaded into any Lisp environment, it would immediately terminate,
which is unacceptably antisocial behavior.
[fn:8] There are other approaches to this problem, but this is the one I’m sticking with.