ASDF 3 TUTORIAL
Historic Overview of ASDF
transform source (for humans) into binary (for machine)
a bit like
make for C
enable division of labor
divide the source into separate components
multiple people can collaborate, each making changes to a few components
people in different teams, in same team, in same cranium.
system: CL name for top-level unit of software management
In other languages they are called: library, package, module, bean, egg, class, archive…
Configuration: find where is each file needed
Dependencies: build things in correct order
Incrementality: re-build iff changed
No build system
What a manual load file might look like,
(load #p"/path/to/library1.lisp") (defparameter *library2-directory* #p"/path/to/library2/") (load (merge-pathnames #p"source/loader.lisp" *library2-directory*)) (setf (logical-pathname-translations "LIBRARY3") `(("**;*.*.*" #p"/path/to/library3/*.*"))) (load #p"LIBRARY3:load-library3.lisp") (load (compile-file (merge-pathnames "file1.lisp" *this-software-directory*))) (load (compile-file (merge-pathnames "file2.lisp" *this-software-directory*))) (load (compile-file (merge-pathnames "file3.lisp" *this-software-directory*)))
Previous example with
(defsystem this-software :depends-on (library1 library2 library3) :components ((:file "file1") (:file "file2" :depends-on "file1") (:file "file3" :depends-on "file1")))
Can find libraries w/o specific configuration
Can find files inside library w/o extra configuration
Configuration is done separately and uniformly
dependencies: finer information is captured
incrementality: only build what’s needed
more: portability, extensibility, etc.
ASDF descends from
build system: compile source files
specialized: oriented toward CL software
not geared for arbitrary tasks with dependencies
in image: also load software
totally unlike either
maintain long-lived system state
declarative: describe system dependencies
not imperative instructions on how to build
got more declarative as
DEFSYSTEM grew older
Lisp build system history
197x: Lisp Machine
Chine Nual: components and manual rules
198x: kmp’s MIT AI Memo 801, rer’s MIT AI TR 874.
very elaborate, proprietary
MK-DEFSYSTEM. 3.6i: 218kB.
free, portable, but complex, feature poor, not extensible
defsystem of Allegro, LispWorks
proprietary, quality between
2002: ASDF, by Dan Barlow et al. 1.85: 38kB. 1.369: 77kB.
configurable, extensible, semi-portable.
2010: ASDF2, by Faré et al. 2.000: 138kB. 2.26: 198kB.
robust, portable, usable, upgradable
See “Evolving ASDF: More Cooperation, Less Coordination”
2013: ASDF 3, by Faré. 2.27: 409kB. 3.0.1: 459kB.
Fix 30-year old bug by making design coherent, new features
Future: ASDF 4?
A simpler, better replacement for
Use CLOS, don’t support obsolete platforms
focus on SBCL and Unix
ported to a handful other implementations
Inter-system configuration: find systems though
No need to edit a file for every system any more!
Typically, “symlink farms” – but Unix specific
Intra-system configuration: none needed, use
Brilliant key idea establishes ASDF dominance
Extensibility: use of CLOS to model dependencies
Example in SB-GROVEL
Its configuration mechanism was a brilliant innovation
Before you laugh, compare to autotools, pkgconfig, etc.
Extensible CLOS model also innovative, but not fully understood
Not by me until I rewrote it, not by Dan Barlow himself.
In many ways, a discovery, not an intentional design.
Became de facto standard
quicklisp: over 700 libraries
Now a key piece of community infrastructure
Therefore cursed with backward-compatibility
if it’s not backward…
ASDF 1 issues
Not very portable
A lot of bugs outside the common case
No standard way to load it
Yet development stalled:
Users wait for new version before to rely on features / bug fixes
Implementers wait for users to demand new version before to change and break compatibility
Some distributions pre-package CL with ASDF pre-loaded, others don’t
If an old one is pre-loaded, it’s too late to upgrade with a version with bugs fixed
ASDF 2 Features
Hot-upgradable: reverse incentive so development can happen
Portable: 15 implementations, 4 OSes
Robust: Massive bug fixes
Massive cleanup of internals. Pathname hell. Corner cases.
Faster: Don’t use lists when inappropriate
Can now scale to thousands of files
Configurable: by end-users, not just developers
Domain-Specific Language for better configuration
Modular update of configuration
Usable: a whole lot of small missing features
(asdf:load-system :foo) instead of
(asdf:operate 'asdf:load-op 'foo)
load-system test-system require-system
:defsystem-depends-on :force-not :encodings :around-compile :compile-check
ASDF 3 Features
Complete refactoring, fixed deep conceptual bugs.
Deliver your system(s)
as single fasl (
as single lisp source file (
as an executable program (
program-op), with runtime hooks
Portability: new library
Condition Control: muffle warnings, keep deferred warnings
naming: multiple systems in
How to use ASDF
What ASDF does
Compile and load Lisp code in current image
Locates software based on configuration
Provide extensible object model to developers
What ASDF does not
Download code (but
Solve version hell (only checks as specified)
Build non-Lisp stuff (awkward)
Example minimal ASDF session
(require :asdf) (asdf:load-system :inferior-shell) (in-package :inferior-shell) (run `(pipe (echo ,(* 90 137)) (tr "1032" "HOLE"))) ;; More: (run `(grep "Mem" "/proc/meminfo") :output :lines) (asdf:test-system :inferior-shell)
Using ASDF, the safe way
;; CLISP alone won't accept :asdf (require "asdf") ;; active implementations provide ASDF2 or later #-asdf2 (error "You lose") ;; force ASDF2 to upgrade to your installed ASDF3 (asdf:load-system :asdf)
Using ASDF, the hard way
tries hard when the implementation doesn’t provide ASDF.
Even harder: see
quux (to be published)
configure asdf, twice, to work around cases of unsmooth upgrade.
Using CL-Launch from command-line
cl-launch -s this-software -i '(this-software:main)' \ -- arg1 arg2
Using CL-Launch from script
#!/bin/sh ":" ; DIR="$(cd $(basename "$0");pwd)" #| exec cl-launch -l ccl -S "$DIR//:" -i "$0" -- "$@" exit |# (some lisp code)
How to configure ASDF
How to configure ASDF
Optimization, verbosity, etc.
Default Installation Paths
No need to configure if you use defaults
Source Registry, via config file
(:source-registry (:directory "/myapp/src") (:tree "/home/tunes/cl") :inherit-configuration)
Unlike ASDF 1, forgiving of no final
Source Registry, via modular config file
Source Registry, via environment
Source Registry, via Lisp evaluation
(asdf:initialize-source-registry `(:source-registry (:directory ,appdir) (:tree ,librootdir) :inherit-configuration))
Old Style central registry
(pushnew #p"/myapp/src/" asdf:*central-registry* :test 'equal)
Catch: ASDF 1 was unforgiving if you forgot the trailing
Magic: argument actually evaluated.
ASDF 2 has
No portable place to do it with ASDF 1.
~/.sbclrc on SBCL.
source-registry can be configured in a decentralized way
Each can specify what he knows,
none need specify what he doesn’t
Where is the fasl for
Multiple implementations and variants may use the same name
Allegro 9.0 SMP vs Allegro 9.0 normal
SBCL 1.1.0 vs SBCL 1.1.8
SBCL 1.1.0 x86 vs SBCL 1.1.8 x86_64
Many ASDF1 extensions to move FASLs away, but hard to configure
No consensus solution on where to put things
Output Translations, via config file
(:output-translations (t (,cache-root :implementation)) :ignore-inherited-configuration)
Output Translations, via modular config file
("/myapp/src/" ("/var/clcache" :implementation "myapp/src"))
(asdf:initialize-output-translations `(:output-translations (t (,cache-root :implementation)) :ignore-inherited-configuration))
Output Translations, $PWD/sbcl-1.2-x86/foo.fasl
(asdf:initialize-output-translations `(:output-translations (t (:root :**/ :implementation :*.*.*)) :ignore-inherited-configuration))
(load "quicklisp/setup.lisp") does it all
I’m not sure about
clbuild — use the source-registry
How do I find a library?
Google it, search
Ask the community, e.g.
Where do I download it?
To some place in your source-registry
(declaim (optimize ...)
(setf *compile-verbose* nil)
easy build script:
sbcl --load build.lisp
For portability, use
cl-launch as above
How to define a simple ASDF system
Creating Basic ASDF Systems
(asdf:defsystem foo :components ((:file "foo")))
Depending on other systems
(defsystem foo :depends-on (:alexandria :cl-ppcre) :components ((:file "foo")))
(defsystem foo ... :components ((:file "pkgdcl") (:file "foo" :depends-on ("pkgdcl")) (:file "bar" :depends-on ("pkgdcl"))))
Typical small system
(defsystem foo ... :components ((:file "pkgdcl") (:file "specials" :depends-on ("pkgdcl")) (:file "macros" :depends-on ("pkgdcl")) (:file "utils" :depends-on ("macros")) (:file "runtime" :depends-on ("specials" "macros")) (:file "main" :depends-on ("specials" "macros"))))
Bigger system: divided in modules
(defsystem foo ... :components ((:module "base" :components ...) (:module "runtime" :depends-on ("base") :components ...) ...))
Logical Modules, same directory
(defsystem foo ... :components ((:module "base" :pathname "" :components ...) ...))
(:file "foo/bar") (:file "foo" :pathname "../sibling-dir/foo") (:file "foo" :pathname #p"../sibling-dir/foo.LiSP")
(:file "../sibling-dir/foo") (:module "../sibling-dir/foo") (:file "foo" :pathname "../sibling-dir/foo") (:file "foo" :pathname #p"../sibling-dir/foo.LiSP")
Punting on fine-grained dependencies
(defsystem foo :serial t :components ((:file "pkgdcl") ... (:file "main")))
:serial t is the current module or system
not its submodules or systems.
You can easily nest serial / parallel dependencies
:depends-on ("foo" "bar/baz" "quux")
defsystem forms for
Any classes, methods from
No other methods, no side-effect, no pushing features
Other files in a project
Automatically create the skeleton
How (not) to map packages and systems
A system may or may not define a package of same name
Strategy 1: one package per system
The traditional way
Strategy 1b: one package per subsystem
Whether you subsystem is a second system or a module
Strategy 2: interface vs implementation package
Strategy 3: one package per file
More discipline, reduces mess
dependencies implicit from defpackage
See source code of
ASDF 3 itself
quick-build use it for dependencies!
if you :use or :import-from a package, load it first
Part of UIOP, new in ASDF 3
Works well with hot-upgrade
Automation common patterns:
(:mix "foo" "bar")
(:reexport "foo" "bar")
.asd file syntax
ASDF 3: now read in UTF-8 encoding, not
ASDF 3: Now read in package
ASDF-USER, not a temporary package
Compatibility: NOT binding
Deprecated: arbitrary code in
Recommended: only calls to
Issue: avoid name conflict issues between
Old ASDF 1 & 2 read each file in its own temporary package
ASDF 3 now all reads them in a common package
UIOP due to conflict with
ASDF is not the right place for this “innovation”
If you’re CL programmer, you know your package discipline
If you don’t know your package discipline, you’re screwed anyway
Best package practice
No need for
(in-package :asdf) in your
Read in shared namespace
ASDF-USER — usual discipline applies
If you bind new symbols, use
On ASDF 3, it
UIOP/PACKAGE for its
How to use advanced ASDF features
Using Extensions: CFFI Grovel
(defsystem foo :defsystem-depends-on (:cffi-grovel) :depends-on (:cffi) :components ((:cffi-grovel-file "c-prototypes") (:file "lisp-code" :depends-on ("c-prototypes"))))
Character encoding, since 2.21
(defsystem foo :encoding :latin1 :components ((:file "pkgdcl" :encoding :utf-8) (:module "russian" :encoding :iso-8859-5 :components ((:file "bar" :encoding :koi8-r) ...))))
*default-encoding* is now
:utf-8 since 2.31
a boon for most programs, work predictably
breaks a handful on unmaintained packages in quicklisp
Finalizers, since 2.23
(defsystem :asdf-finalizers-test :defsystem-depends-on (:asdf-finalizers) :around-compile "asdf-finalizers:check-finalizers-around-compile" :depends-on (:list-of :fare-utils :hu.dwim.stefil) :components ((:file "asdf-finalizers-test")))
(defun foo (l) (check-type l (list-of string))) (asdf-finalizers:final-forms)
Compile in a fork, load in current image.
Replay compilation errors in current image
antifuchs 2007-2008: build ASDF systems in parallel
fare 2009-2013: robust, portable, integrated to ASDF
Deterministic by default given initial state
Faster option: more parallelism
Can fork on
Graceful fallback if no forking.
Handle deferred warnings
How the ASDF object model works
Components, Operations, Actions
COMPONENT’s describe your source code
OPERATION’s are stages of processing to perform on components
ACTION is a pair of an
OPERATION and a
(cons (find-operation () 'load-op) (find-component "this-software" "file1"))
The dependency graph is a direct acyclic graph of
It is not a graph of components that depend on each other.
Plan first, then perform
PERFORM-PLAN was a recent change before ASDF 3.
TRAVERSE walks the dependency graph and returns a plan
Traditionally, a LIST of actions to perform in order
Can be overridden. POIU returns a representation of the complete graph.
PERFORM-PLAN walks the plan calling
PERFORM-WITH-RESTARTS on each
PERFORM-WITH-RESTARTS sets up proper restarts and calls
The graph is computed by
Misnamed: actions, not components, have dependencies.
Arguments: an operation designator, component designator
(COMPONENT-DEPENDS-ON 'LOAD-OP '("this-software" "file2"))
CLOS: OO multi-dispatch on two arguments!
Return a list of lists of operation designator and component designators
((#<LOAD-OP> #<CL-SOURCE-FILE "this-software" "file1">))
CLOS: don’t forget to append the
we could have used the
APPEND method combinator, but are not,
for historical backward compatibility reasons
CLOS: inherit from mixins to achieve desired effects
CLOS makes things very modular. Big win!
component module system source-file cl-source-file cl-source-file.cl cl-source-file.lsp static-file cffi-grovel-file
Typical component tree
system cl-source-file-1 cl-source-file-2 module1 cl-source-file-3 cl-source-file-4 cl-source-file-5
new in ASDF 3:
Also new in ASDF3,
bundle-op and friends:
Typical operations mixins (ASDF 3):
OUTPUT-FILES: output-translations in an
INPUT-FILES: automation in
An action is
NEEDED-IN-IMAGE-P iff its
OUTPUT-FILES is nil
Otherwise, it need not be
PERFORM‘ed again in current image if files up to date
Important notion implicit in ASDF 1&2, introduced by
TRAVERSE may visit an action twice
NIL and oncep with it
The bug that launched ASDF 3
ASDF 2.26 was stable
ASDF had been completely rewritten since ASDF 1
Now made portable, robust, usable, etc.
Everything had been touched except trivial things
But core dependency traversal algorithm unchanged
To fix bugs, refactored out of spaghetti code, but
functionally equivalent, modulo bug fixes
TRAVERSE was the holy relic passed by Dan Barlow
I didn’t grok the design, it felt slightly wrong.
Couldn’t change anything by fear of backward compatibility
Remained only one bug to procrastinate on
All other bugs were wishlist items made difficult by current design
Failure to propagate dependency changes
lp#479522 changes fail to trigger a rebuild across systems
explicitly disabled in
In olden days, some have argued for the former bug as a “feature”
It was only a crock to work around lack of
When you enable the obvious fix, it only works in current session
system2 depends-on system1
in one session, change system1, recompile it
in another session, compile system2 that didn’t change
ASDF 1 and 2 fail to recompile system2
Not just between systems!
More common failure mode:
Use a stateful macro, such as
file1 define the macro,
file2 use it
file2 is not recompiled
Other common failure mode:
have file1, file2, file3 with serial dependencies
file1 has changed, file3 hasn’t
file2 completely breaks the build
you fix file2, and restart the build
ASDF 2 fails to recompile file3
Decades Old Dependency Bugs
Cause: ASDF only checked timestamp for files of action
Doesn’t even try to propagate timestamp from dependencies! lp#1087609
Need-to-recompile may be propagated only from current session
Bug present in 1991 MK-DEFSYSTEM and the original 197X DEFSYSTEM
Optional fix in Symbolics, Allegro, LispWorks defsystem
offer a different kind of dependencies than the default
broken by default (backward compatibility?)
not a complete fix in LispWorks
Fixing the bug requires a complete rewrite of ASDF’s
Twice. Because then you find you need a correct dependency model
along which to correctly propagate timestamps.
Why never reported before?
Usually not THAT big an issue
Most Lispers hack on one small system at once.
Usually you interactively use the
CONTINUE restart after fixing bug.
When you change
file1, you often need to change
file3, too, anyway.
In doubt, you
:force a build from clean or erase all the fasls.
Now given in large systems built in batch with stateful macros… Ouch.
false positives and negatives waste time in building and testing
uncontroled non-determinism in testing is bad
Not your typical Lisp development style!
Live Programming vs Dead Programs
Live Programming: code is mutable
Short feedback “OODA” loop. Low overhead (meta)computing.
Dead Programs: code is immutable
Easier to analyze before it’s run. Too late to debug afterwards.
Both matter for the same reason:
programmer interaction is a scarce resource
On-line, adj.: The idea that a human being should always be accessible to a computer.
Computing systems of the future should support both in synergy.
Live style to metaprogram dead style programs.
Zombie programs that resurrect on-demand.
ASDF 3: traversing dependencies correctly
Solution: road to ASDF3
This in turn necessitates a complete graph representation
This means refactoring downward propagation away from
traverse and the
This means reorganizing the source code
Split the code into files so it makes sense
Merge in and fix the
Recursively use new
traverse to walk the partial plan for an action
It now makes sense to have a separate portability layer
UIOP, spend time making it a quality library
Many cleanups and new features are now unlocked
Spend a lot of time implementing them robustly
Some new features are oh so slightly backward incompatible
Spend a lot of time fighting the community, and losing
introduced to fix a conceptual bug in the ASDF object model.
“load the dependencies of a component and its parents”
Propagates upward in the component hierarchy, not downward
TRAVERSE special cases such dependencies no more
TRAVERSE was gutted out
Not only bug fixes, but much simpler, sensible semantics
Now propagating timestamps along a graph and that only
Refactored into reusable higher-order functions and objects
The object model now actually makes sense, and can be extended
No more implicit descending into children components
downward-operation for such propagation
methods take a
plan object, NIL for actual action
Informed by interface-passing-style and experience with
Was necessary to get
BUNDLE-OP right portably
Many many thanks to antifuch’s
COMPONENT-DEPENDS-ON is now more powerful
can express dependencies on arbitrary operation objects
Supported: depend not just on siblings
Supported: express arbitrary build graphs
Deprecated: operations with different options
Deprecated: depending on component in other system
COMPONENT-DO-FIRST is no more
It used to specify some dependencies that were skipped
if no re-build was triggered based on local timestamps;
ASDF 1 didn’t let the users control it,
ASDF 2 only let you control it since 2.017 or so.
In ASDF 3,
NEEDED-IN-IMAGE-P mechanism supersedes
COMPONENT-DEPENDS-ON is used for all dependencies.
:in-order-to everywhere you used to use
:do-first, if ever.
new attribute of
accepts an arbitrary feature expression
:if-feature (:and :sbcl (:or :x86 :x86-64))
Beware: no magic reading in keyword package — use : syntax
Replaces the misguided
:if-component-dep-fails attribute of
could not be salvaged when refactoring
Dropped that attribute and the accompanying
Limited backward compatibility just for
SB-GROVEL and co.
ASDF 3’s new DEFSYSTEM features
ASDF3 ~70% slower than ASDF2
Slightly faster when
*RESOLVE-SYMLINKS* is false (default true)
ASDF2 much faster than ASDF1: don’t (ab)use LIST data structures
Underneath, ASDF3 does much more work, correctly
Cache expensive computations in hash-table in dynamic variable
One package per file
ASDF 3 was rewritten in the style of
Each file has its own
UIOP/PACKAGE:DEFINE-PACKAGE for hot-upgrade and reexport
Future: actually support
build a single Lisp file from all the source in a system
MONOLITHIC-CONCATENATE-SOURCE-OP to transclude dependencies
Used by ASDF itself to split it in multiple files
ASDF has more than doubled in size between ASDF 2.26 and ASDF 3.0.1
Had already increased manifold since ASDF 1.
It just does that much more work.
The ASDF 1 bits have actually been much simplified.
ASDF-BUNDLE was merged into ASDF.
Fewer headaches for users of ECL
More features for users of other implementations
Can create a single fasl per system with
Makes software delivery easier.
Support for pre-compiled systems.
SBCL patch to use that for contribs.
create standalone executables on supported implementations
clisp ccl cmucl ecl lispworks sbcl scl
See example in
Uses image hooks above.
A generic operation that will do the “right thing” for each system
Not super supported yet, but the future(?)
:force to actually work as advertised by ASDF 1.
t, or a list of system names
:force-not and based on it
Can’t force builtin systems (e.g.
WARNING: rpg may revert that
FORCE has precedence over
name be recognized by defsystem as located in
Somewhat backward compatible
in ASDF2, you had to manually ensure
foo.asd was loaded beforehand
in ASDF3, works automatically
Allows sensible way to define multiple systems in an
Internals: grep for function
Don’t drop info on yet undefined functions
allegro ccl cmucl sbcl scl
Disabled by default.
#+asdf3 (setf asdf::*warnings-file-type* (asdf::warnings-file-type))
Dump info for
Checked at the end of the build on each system
In a method to
PERFORM (COMPILE-OP SYSTEM)
As if a
WITH-COMPILATION-UNIT around each system
Now can be reliably turned off:
(setf asdf:*resolve-symlinks* nil)
TRUENAME is slow or bogus on your OS
Necessary if using symlinks to content-addressed storage
e.g. the Google build system
Warnings if you don’t follow the convention of
version-satisfies now uses uiop:version<= for comparison
No more checking for a same major version number
Was undocumented behavior since ASDF 1, still in version-compatible-p
:VERSION spec in
Now also accept
(:read-file-form <path> :at <formpath>)
Now also accept
(:read-file-line <path> :at <linenum>)
:at optional, defaults to 0, 0-based
<formpath> as per
(:read-file-form "specials.lisp" :at (2 2))
(:read-file-form "specials.lisp" :at (third third))
Easier to manage versioning from master location
ASDF 3 will always start by automatically upgrade itself
Proviso against downgrade, with warning
Just have the
asdf/ tree somewhere in your
Only sane way to deal with potential upgrade
Otherwise, if any recursive dependency loads ASDF, kaboom
not algorithmically detectable:
.asd files not declarative
:PROPERTIES initarg of
Still works for now
To be retired before a hypothetical future ASDF 4.
Used by few, never with any name convention.
Recommended instead: use
DEFCLASS a subclass of
to add new slots and/or initargs.