MAGICL

Matrix Algebra proGrams In Common Lisp by Rigetti Computing. (née FLAIL: Finally, Linear Algebra In Lisp!)

(Note: The high-level interface is experimental and subject to change.)

Requirements

MAGICL has two main systems:

• MAGICL/CORE: This is pure Lisp code with no foreign dependencies. This system establishes MAGICL's API (for the most part).

• MAGICL: This is MAGICL with all extensions loaded.

The system MAGICL/CORE requires:

• SBCL (> 1.3.19), CCL (>= 1.11) or ECL (>= 20.4.24) on AMD64
• quicklisp

The system MAGICL, on the other hand, requires several foreign dependencies not shipped with MAGICL, like:

• libffi
• BLAS and LAPACK

Detailed instructions on how to install libffi and BLAS/LAPACK can be found here.

Currently this library is SBCL-, CCL- and ECL-only. The non-portable code is in with-array-pointers.lisp and magicl.lisp.

Installation

First ensure you have the necessary requirements installed, as described in the previous section.

To install MAGICL, clone this repository into your Quicklisp's local-projects folder. You can quickly check where this is by running sbcl and evaluating ql:*local-project-directories*. Once installed, confirm that MAGICL is working properly by running the tests, as described in the next section.

Lisp-Only vs Accelerated MAGICL

Extensions

MAGICL/CORE only uses pure ANSI Common Lisp code. If you wish to accelerate it or extend the functionality, you may load MAGICL extensions. These extensions typically install new backends to MAGICL functions. The available extensions are:

• MAGICL/EXT-BLAS: for BLAS functions
• MAGICL/EXT-LAPACK: for LAPACK functions
• MAGICL/EXT-EXPOKIT: for expokit (matrix exp()) functions

For backwards compatibility, MAGICL loads every extension under the kitchen sink. This may change in future versions of MAGICL! If you depend on an extension, depend on it explicitly!

If you use extensions, you'll need the requisite C/Fortran libraries. Expokit will automatically build for you, as its source is included in the distribution of MAGICL.

Backends

Accelerated functionality is installed with a notion called "backends". A backend is a name of a group of functionality, typically denoted by a symbol or keyword. The :lisp backend is the default one, and several backends can be active all at once. Each extension above adds a new backend. The current backends are:

• :lisp: Pure Lisp code
• :blas: BLAS-backed code
• :lapack: LAPACK-backed code
• :expokit: expokit-backed code

In most cases, one does not need to concern themselves with backends; MAGICL functionality should "just work" and dispatch to the appropriate backend. However, the programmer always has control, even dynamically in the program, of which backends should be used at a given time with the magicl.backends:with-backends macro. For instance,

(magicl.backends:with-backends (:blas :lisp)
  ;; ... code ...
  )

says that the code should be executed, always preferring :blas-accelerated functions, and using :lisp-implemented functions as a fall-back.

(magicl.backends:with-backends (:lisp)
  ;; ... code ...
  )

says to only use :lisp-implemented functions, even if other backends are loaded.

The active backends can be found with the function magicl.backends:active-backends, which lists the backends to use in priority order.

One can be even finer-grained than with-backends. Given a function f which has many backend implementations, one can get a specific implementation by using the function:

(magicl.backends:backend-implementation 'f :backend-name)

For instance

(magicl.backends:backend-implementation 'magicl:csd :lapack)

will give the implementation of the cosine-sine decomposition function in LAPACK. This can be called in exactly the same way magicl:csd can be called.

In backend-implementation, if both the function name and the backend name are (quoted) constants, this will be looked up at compile-time, which is useful for writing efficient code that does not dispatch. But note that by doing this, with-backends will not be respected.

Testing MAGICL

You can run the MAGICL tests from your Lisp REPL with:

(asdf:test-system :magicl)

You currently need all of the extensions working for the tests to run.

High-level Interface

See the high-level doc for an extensive discussion and comparison of MAGICL functions with those of MATLAB and NumPy.

Developer's Guide: How to Add New Functions

See the developer how-to to understand how to add new functionality to MAGICL.

Fortran Bindings

See the Fortran Functions on how to re-generate the Fortran bindings from the original BLAS, LAPACK, and Expokit reference code.

See the same document for how to query for available Fortran functions in the currently loaded dynamic libraries.

History and Credits

MAGICL development started at Rigetti Computing by Robert Smith and Joe Lin in 2017.

CL-BLAPACK is a library developed by Ryan Rifkin and Evan Monroig. Rigetti Computing created a fork of this library and renamed it MAGICL, and made significant changes that departed from the original design, including:

• Fixing several bugs in the Fortran parsing to make it work with the latest reference BLAS and LAPACK, leading to significant refactoring.

• Adding support for matrix exponentiation with Expokit.

• Adding support for loading various BLAS and LAPACK implementations.

• Removing the use of the FNV library in favor of native Lisp arrays.

• Adding a high-level interface to various functions.

• Adding function availability reporting.

The most important common design decision between CL-BLAPACK and MAGICL is allowing direct access to the Fortran library functions by way of automatically generated Lisp bindings from the reference sources.