Graven Image

Thou shalt not make unto thee any graven image, or any likeness of any thing that is in heaven above, or that is in the earth beneath, or that is in the water under the earth.

Graven Image is a Common Lisp portability library (a less fancier name might’ve been trivial-debugging) for better interaction and debugging of a running Lisp image. One can inspect and debug all the things under heaven and above it—all in their own REPL-resident Lisp image!

Graven Image reuses compiler internals to improve/redefine the existing standard functions. This “improvement” often comes at a cost of changing the API of a function (for better customizability) or making it slightly less reliable (due to unstable, compiler-internal, or otherwise hacky implementation.)

The library is purposefully limited in scope:

• Improving the standard functions has a priority over introducing new ones.
  • repl-utilities as a contrasting approach: it defines lots of “DWIM” functions, significantly altering the REPL interaction.
  • There are helpers like function-lambda-list* in Graven Image, but these are merely aliases/one-liners over standard/improved functions.
    • I get carried away sometimes, though. benchmark* is one of such nerd snipes. But still, it’s quite close to the underlying with-time* 😉
• The interaction paradigm of the improved functions should stay standard (i.e. use *query-io* where standard requires *query-io*.)
  • This is to ensure that libraries like Ndebug can rely on standard facilities safely when used with Graven Image.
• Function arglist can be modified for convenience, but it should preferably stay as close to standard/implementation-specific arglist as possible.
• Graven Image strives to not modify the REPL/image/shell in any way, relying on implementation defaults instead.
  • CIEL is taking a different direction: redefining the REPL for increased utility.
  • flight-recorder uses a separate shell script to add a new functionality.
• Portably reusing implementation-specific functionality is better than re-implementing it. But if some functionality is e.g. unique to SBCL, it might be dropped as non-portable.
  • tracer, supertrace, cl-framegraph, and other non-portable SBCL-specific improvements.

Getting started

Clone the Git repository:

git clone --recursive https://github.com/aartaka/graven-image ~/common-lisp/

And load :graven-image in the REPL:

(asdf:load-system :graven-image)
;; or, if you use Quicklisp
(ql:quickload :graven-image)

You can also install Graven Image via Guix, using the bundled guix.scm file:

guix package -f guix.scm

Someday (after at least minor version bump) I’ll send a patch to Guix so that you can also install it with guix install. Until then—stay tuned (or send the patch yourself—I don’t mind.)

Convenient use

If you want your REPL to start with Graven Image constructs accessible without package prefix, then simply use the package directly.

(asdf:load-system :graven-image)
;; Imports external symbols of Graven Image into the current package
;; (CL-USER?) Safe, because Graven Image shadows no CL symbols.
(use-package :graven-image)

Or, if you’re more orderly and disciplined about packages than I am, you can always use trivial-package-local-nicknames and define a shorter Graven Image nickname:

(trivial-package-local-nicknames:add-package-local-nickname
 :g :graven-image :cl-user)

If you want to replace the standard utilities with Graven Image ones, you can safely do so with the familiar:

;; For functions
(fmakunbound 'apropos)
(setf (fdefinition 'apropos) (fdefinition 'gimage:apropos*))
;; For macros
(setf (macro-function 'time) (macro-function 'gimage:time*))

Enhanced functions (mostly from ANSI CL Debugging Utilities chapter)

The functions that Graven Image exposes are safely :use-able star-appended functions/macros (i.e. describe* instead of describe). Currently improved ones are:

• y-or-n-p* / yes-or-no-p*
• apropos* / apropos-list*
• function-lambda-expression*
• time*
• describe* / inspect*
• dribble*
• documentation*

All of the functions exposed by Graven Image are generics, so one can easily define :around and other qualified methods for them.

y-or-n-p*, yes-or-no-p* (generic functions)

Signature:

y-or-n-p* &optional control &rest arguments => generalized-boolean
yes-or-no-p* &optional control &rest arguments => generalized-boolean

Improvements are:

• Both functions accept options from graven-image:*yes-or-no-options*, thus allowing for “nope” or “ay” to be valid responses too.
• Both functions mean the same now, because it makes no sense in differentiating them (and because most Emacs users use a magical (fset 'yes-or-no-p 'y-or-n-p) in their config, setting the precedent for shorter yes/no queries).
• No beeps (just define a yes-or-no-p* :before method to add beeps if you like ‘em; see the “Customization” section below).

apropos-list*, apropos* (generic functions)

Signature:

apropos-list* string &optional (package nil) exported-only docs-too => list of symbols
apropos* string &optional (package nil) exported-only docs-too => no values

apropos-list* now allows listing exported symbols only (with exported-only), which was a non-portable privilege of SBCL/Allegro until now. Search over docs (more intuitive for apropos(-list)* than mere name search) is possible with docs-too.

Based on this foundation, apropos* lists symbols with their types, values, and documentation, so that implementation-specific formats are gone for a better and more unified listing:

(apropos* :max)
;; MAX                                            [FUNCTION (NUMBER &REST
;;                                                           MORE-NUMBERS) : Return the greatest of its arguments; among EQUALP greatest, return...]
;; :MAX                                           [SELF-EVALUATING]
;; CFFI::MAX-ALIGN
;; SB-ASSEM::MAX-ALIGNMENT                        [CONSTANT = 5]
;; ...
;; SB-C::MAXES
;; ALEXANDRIA:MAXF                                [MACRO (#:PLACE &REST NUMBERS) : Modify-macro for MAX. Sets place designated by the first argument to the...]
;; SB-KERNEL::MAXIMAL-BITMAP
;; ...
;; SB-LOOP::LOOP-ACCUMULATE-MINIMAX-VALUE         [MACRO (LM OPERATION FORM)]
;; SB-LOOP::LOOP-MAXMIN-COLLECTION                [FUNCTION (SPECIFICALLY)]
;; SB-LOOP::LOOP-MINIMAX                          [CLASS (STRUCTURE-OBJECT)]
;; ...

function-lambda-expression* (generic function)

Signature:

function-lambda-expression* function/macro/method/symbol &optional force => list, list, symbol, list
;; Alias:
lambda-expression* function/macro/method/symbol &optional force => list, list, symbol, list

This function tries to read source files, process the definitions of functions, and build at least a barebones lambda from the arglist and documentation of the function. So that CL function-lambda-expression returns:

(function-lambda-expression #'identity)
;; => NIL, T, IDENTITY
(function-lambda-expression #'print-object)
;; => NIL, T, PRINT-OBJECT

While the new Graven Image function-lambda-expression now returns:

(function-lambda-expression* #'idenitity)
;; => (LAMBDA (THING) "This function simply returns what was passed to it." THING),
;;    NIL, IDENTITY, (FUNCTION (T) (VALUES T &OPTIONAL))
(function-lambda-expression* #'print-object t) ; Notice the T for FORCE, to build a dummy lambda.
;; => (LAMBDA (SB-PCL::OBJECT STREAM)), NIL, PRINT-OBJECT, (FUNCTION (T T) *)

Which means:

• identity is actually not a closure, and has a reliable source!
• print-object is a generic and thus is not really inspectable, so we build a dummy lambda for it when force argument is provided.
  • This might be a questionable choice, but it at least allows us to get function arglists from function-lambda-expression in a portable-ish way. The standard doesn’t provide us with much ways to know an arglist of a function beside this.

Return values

Things that function-lambda-expression* now returns are:

• Lambda expression.
  • For lambda functions, their source.
  • For regular functions, their defun turned into a lambda.
  • For anything else, a constructed empty (lambda (arglist...) documentation nil) (only when force is T).
  • Or, in case all the rest fails, NIL.
• Whether the thing is a closure
  • If it is, might return an alist of the actual closed-over values, whenever accessible (not for all implementations).
  • If closed-over values are not accessible, returns T.
  • If it’s not a closure, returns NIL.
• Function name. Mostly falls back to the standard function-lambda-expression, but also inspects implementation-specific function objects if necessary.
• Function type, whenever accessible (SBCL and ECL).

Helpers

Based on these new features of function-lambda-expression*, here are some Graven Image-specific helpers:

function-lambda-list*

Get the lambda list of a function.

function-arglist*

Alias.

lambda-list*

Alias for function-lambda-list*.

arglist*

Alias.

function-name*

Get the name of a function.

function-type*

Get its ftype.

function-lambda-list* function => list
function-arglist* function => list
lambda-list* function => list
arglist* function => list
function-name* function => symbol
function-type* function => list

time* (macro)

Signature:

time* &rest forms => return-values

The improved time* from Graven Image reuses as much implementation-specific APIs as possible, with the predictable output format.

And it also allows providing several forms, yay!

benchmark* (macro)

Signature:

benchmark* (&optional (repeat 1000)) &body forms => return-values

While time* is the standard benchmarking/profiling solution, it’s almost always too simple for proper benchmarking. Most systems getting complex enough end up with some form of custom benchmarking. Shinmera’s trivial-benchmark is one such example. Graven Image benchmark* is heavily inspired by trivial-benchmark, but has a more portable foundation in the form of with-time*.

As many other benchmarking macros, benchmark* repeats its body a certain number of times, collecting timing stats for every run, and then prints aggregate statistics for the total runs.

(gimage::benchmark* (20) ;; Repeat count.
  (loop for i below 1000 collect (make-list i) finally (return 1)))
;; Benchmark for 20 runs of
;; (LOOP FOR I BELOW 1000
;;       COLLECT (MAKE-LIST I)
;;       FINALLY (RETURN 1))
;; -                   MINIMUM        AVERAGE        MAXIMUM        TOTAL
;; REAL-TIME           0.0            0.00175        0.019          0.035
;; USER-RUN-TIME       0.000668       0.0016634      0.016315       0.033268
;; SYSTEM-RUN-TIME     0.0            0.00021195     0.003794       0.004239
;; GC-RUN-TIME         0.0            0.00085        0.017          0.017
;; BYTES-ALLOCATED     7997952.0      8008154.5      8030464.0      160163090.0

with-time* (macro)

Signature:

with-time* (&rest time-keywords) (&rest multiple-value-args) form &body body

As the implementation detail of time* and benchmark*, with-time* allows to get the timing data for interactive querying. time-keywords allow &key-matching the timing data (like :gc time or bytes :allocated) for processing in the body. While multiple-value-args allow matching against the return values of the form. So we get best of the both worlds: timing data and return values. This flexibility enables time*, with its requirements of printing the data and returning the original values at the same time.

For example, here’s how one would track the allocated bytes and garbage collection times when running a cons-heavy code:

(gimage:with-time* (&key aborted gc-count gc allocated)
    (lists lists-p)
    (loop for i below 1000
          collect (make-list i :initial-element :hello)
            into lists
          finally (return (values lists t)))
  (unless aborted
    (format t "Bytes allocated: ~a, GC ran ~d times for ~a seconds"
            allocated gc-count gc)))
;; Bytes allocated: 7997952, GC ran NIL times for 0 seconds

describe* (generic function)

Signature:

describe* object &optional (stream t) respect-methods

Describes the object to the stream, but this time with portable format of description (determined by graven-image:description* and specified for many standard classes) and with predictable set of properties (graven-image:fields*). In Graven Image, both describe and inspect have the same format and the same set of fields.

As a note of respect to the original describe, Graven Image one allows to reuse the describe-object methods defined for user classes. To enable this, pass T to respect-methods.

graven-image:fields* (generic function)

Signature:

fields* object &key strip-null &allow-other-keys

Returns an undotted alist of properties for the object. Custom fields provided by Graven Image are named with keywords, while the implementation-specific ones use whatever the implementation uses. Arrays and hash-tables are inlined into fields to allow indexing these right from the inspector.

See fields* documentation for more details.

graven-image:description* (generic function)

Signature:

description* object &optional stream

Concise and informative description of object to the stream. Useful information from most of the implementations tested—united into one description header.

inspect* (generic function)

Signature:

inspect* object &optional strip-null

New’n’shiny inspect* has:

• Most commands found in other implementation, with familiar names.
• Abbreviations like H -> HELP (inspired by SBCL).
• Ability to set object field values with (:set key value) command (inspired by CCL).
• Built-in pagination with ways to scroll it (:next-page, :previous-page, :home) and change it (:length).
• Property indexing by both integer indices and property names (with abbreviations for them too!).
• Ability to ignore nil properties with strip-null argument (inspired by SBCL). On by default!
• And the ability to evaluate arbitrary expressions (with :evaluate command or simply by inputting something that doesn’t match any command).

And here’s a help menu of the new inspect* (in this case, inspecting *readtable*), just to get you teased:

This is an interactive interface for 5
Available commands are:
:?                            Show the instructions for using this interface.
:HELP                         Show the instructions for using this interface.
:QUIT                         Exit the interface.
:EXIT                         Exit the interface.
(:LENGTH NEW)                 Change the page size.
(:WIDTH NEW)                  Change the page size.
(:WIDEN NEW)                  Change the page size.
:NEXT                         Show the next page of fields (if any).
:PREVIOUS                     Show the previous page of fields (if any).
:PRINT                        Print the current page of fields.
:PAGE                         Print the current page of fields.
:HOME                         Scroll back to the first page of fields.
:RESET                        Scroll back to the first page of fields.
:TOP                          Scroll back to the first page of fields.
:THIS                         Show the currently inspected object.
:SELF                         Show the currently inspected object.
:REDISPLAY                    Show the currently inspected object.
:SHOW                         Show the currently inspected object.
:CURRENT                      Show the currently inspected object.
:AGAIN                        Show the currently inspected object.
(:EVAL EXPRESSION)            Evaluate the EXPRESSION.
:UP                           Go up to the previous level of the interface.
:POP                          Go up to the previous level of the interface.
:BACK                         Go up to the previous level of the interface.
(:SET KEY VALUE)              Set the KEY-ed field to VALUE.
(:MODIFY KEY VALUE)           Set the KEY-ed field to VALUE.
(:ISTEP KEY)                  Inspect the object under KEY.
(:INSPECT KEY)                Inspect the object under KEY.
:STANDARD                     Print the inspected object readably.
:AESTHETIC                    Print the inspected object aesthetically.

Possible inputs are:
- Mere symbols: run one of the commands above, matching the symbol.
  - If there's no matching command, then match against fields.
    - If nothing matches, evaluate the symbol.
- Integer: act on the field indexed by this integer.
  - If there are none, evaluate the integer.
- Any other atom: find the field with this atom as a key.
  - Evaluate it otherwise.
- S-expression: match the list head against commands and fields,
  as above.
  - If the list head does not match anything, evaluate the
    s-expression.
  - Inside this s-expression, you can use the `$' function to fetch
    the list of values under provided keys.

dribble* (generic function)

Signature:

dribble* &optional pathname (if-exists :append)

Dribble the REPL session to pathname. Unlike the implementation-specific dribble, this one formats all of the session as load-able Lisp file fully reproducing the session. So all the input forms are printed verbatim, and all the outputs are commented out.

Beware: using any interactive function (like inspect etc.) breaks the dribble REPL. But then, it’s unlikely one’d want to record interactive session into a dribble file.

documentation* (generic function)

Signature:

documentation* object &optional (doc-type t)
doc* object &optional (doc-type t)

Improved version of documentation. Two main improvements are: doc-type is now optional, and doc* alias is available for convenience.

documentation.lisp also defines more documentation methods (and respective setf method) to simplify documentation fetching and setting. In particular, method on (symbol (eql t)) to simplify symbol documentation search; and (t (eql 'package)) with a new doc-type for package documentation convenience.

break* (macro)

Signature:

break* &rest arguments

A more useful wrapper for break, listing the function it’s called from and the provided symbol values. See examples in the docstring.

Customization

Graven Image is made to be extensible. That’s why most of the improved functions are generic: one can define special methods for their data and patch the behavior with :before, :after, and :around methods. Most of Graven Image functions mention the variables/things influencing them in the docstring. Here’s a set of useful customizations:

Beeping before yes-or-no-p*

Restoring the standard-ish (beeping with bell (ASCII 7) character) behavior:

(defmethod gimage:yes-or-no-p* :before (&optional control &rest arguments)
  (declare (ignore control arguments))
  (write-char (code-char 7) *query-io*)
  (finish-output *query-io*))

Changing the accepted yes/no options for yes-or-no-p* and y-or-n-p*

;; Make it strict yes/no as per standard.
(defmethod gimage:yes-or-no-p* :around (&optional control &rest arguments)
  (declare (ignore control arguments))
  (let ((gimage:*yes-or-no-options*
          '(("yes" . t)
            ("no" . nil))))
    (call-next-method)))

;; Add more yes/no options (Russian, for example).
(defmethod gimage:y-or-n-p* :around (&optional control &rest arguments)
  (declare (ignore control arguments))
  (let ((gimage:*yes-or-no-options*
          (append
           gimage:*yes-or-no-options*
           '(("да" . t)
             ("ага" . t)
             ("нет" . nil)
             ("не" . nil)
             ("неа" . nil)))))
    (call-next-method)))

Sorting apropos-list* lists

Implementations are not good at sorting things, and their results are not often useful. Sorting things the way one needs is a useful extension. Here’s a simple yet effective :around method that sorts things by string occurence:

(defmethod gimage:apropos-list* :around (string &optional packages external-only docs-too)
  "Sort symbols by the relation of subSTRING count to the length of symbol."
  (declare (ignorable packages external-only docs-too))
  (let ((result (call-next-method)))
    (sort
     (remove-duplicates result)
     ;; For more comprehensive matching, see
     ;; a1b4ebd649e0268b1566e80709e7cea41363d006 and other commits
     ;; before c090d6dc14e05c561cf5c39cf5f6cc02e8cd04c5.
     #'> :key (lambda (sym)
                (let ((match-count 0))
                  (uiop:frob-substrings
                   (string sym) (list (string string))
                   (lambda (sub frob)
                     (incf match-count)
                     (funcall frob sub)))
                  (/ match-count (length (string sym))))))))

Changing printer settings for Graven Image output

Graven Image inspect* function uses *interface-lines* for the number of properties to list. If your screen is more than 20 lines high, you might want to add more lines:

(defmethod gimage:inspect* :around (object)
  (declare (ignore object))
  (let ((gimage:*interface-lines* 45))
    (call-next-method)))

Most of Graven Image functions also rely on implementation/REPL-specific printer variables, which might be un-intuitive, overly verbose, or too short. Binding printer variables around Graven Image functions helps that too:

(defmethod gimage:apropos* :around (string &optional package external-only docs-too)
  (declare (ignore string package external-only docs-too))
  ;; Note that you can also use
  ;; `sb-ext:*compiler-print-variable-alist*' and
  ;; `sb-ext:*debug-print-variable-alist*' on SBCL.
  (let ((*print-case* :downcase)
        (*print-level* 2)
        (*print-lines* 2)
        (*print-length* 10))
    (call-next-method)))

A noisy apropos function listing like

X86::*X86-OPERAND-TYPE-NAMES* [VARIABLE = ((:REG8 . 1) (:REG16 . 2) (:REG32 . 4) (:REG64 . 8) (:IMM8 . 16) (:IMM8S . 32) (:IMM16 . 64) (:IMM32 . 128) (:IMM32S . 256) (:IMM64 . 512) (:IMM1 . 1024) (:BASEINDEX . 2048) (:DISP8 . 4096) (:DISP16 . 8192) (:DISP32 . 16384) (:DISP32S . 32768) (:DISP64 . 65536) (:INOUTPORTREG . 131072) (:SHIFTCOUNT . 262144) (:CONTROL . 524288) (:DEBUG . 1048576) (:TEST . 2097152) (:FLOATREG . 4194304) (:FLOATACC . 8388608) (:SREG2 . 16777216) (:SREG3 . 33554432) (:ACC . 67108864) (:JUMPABSOLUTE . 134217728) (:REGMMX . 268435456) (:REGXMM . 536870912) (:ESSEG . 1073741824) (:INVMEM . 2147483648) (:REG . 15) (:WORDREG . 14) (:IMPLICITREGISTER . 75890688) (:IMM . 1008) (:ENCIMM . 464) (:DISP . 126976) (:ANYMEM . 2147547136) (:LLONGMEM . 2147547136) (:LONGMEM . 2147547136) (:SHORTMEM . 2147547136) (:WORDMEM . 2147547136) (:BYTEMEM . 2147547136) (:LABEL . 4294967296) (:SELF . 8589934592))]

turns into a much more readable

x86::*x86-operand-type-names* [variable = ((:reg8 . 1) (:reg16 . 2) (:reg32 . 4) (:reg64 . 8) (:imm8 . 16) (:imm8s . 32) (:imm16 . 64) (:imm32 . 128) (:imm32s . 256) (:imm64 . 512) ...)]

Suppressing documentation errors in documentation*

Several implementations throw errors when trying to get documentation for non-existent method combinations, classes, etc. It’s convenient to suppress these:

(defmethod gimage:documentation* :around (object &optional doc-type)
  (ignore-errors (call-next-method)))

Actually, one can try to write an :around method for regular documentation, but this modification is not guaranteed to work on all implementations.

Contributing

You can help with any of the open issues most are well-described and split into bite-sized tasks. See .github/CONTIBUTING.md for the contributing guidelines.