A LISP programmer knows the value of everything, but the cost of nothing.
— Alan Perlis
This library attempts to measure the execution time of Lisp code using only portable mechanisms, i.e. get-internal-run-time. Using some macrology and sampling, the measurements can be accurate to the nanosecond.
Apart from the benchmark utilities, the library features a test suite that reports the runtime of various Common Lisp features like CLOS, function calls, hashtables, garbage collection and many others. This may give valuable performance insights, e.g. for which size hashtables outperform linked lists on a given implementation.
I hope you find it useful. If you have new ideas for benchmarks, or suggestions on how to improve existing ones, feel free to contact me.
To run all benchmarks, simply execute
To obtain the execution time of an expression in seconds, execute
To have said execution time immediately and human-readably written to trace-output (much like CL:TIME), use
To measure only some parts of an expression, type
(nested-benchmark (foo) (bar) (benchmark SUBEXPRESSION1) (let ((a (baz))) (benchmark SUBEXPRESSION2)))
Furthermore the library exports the following functions
And some functions to register and query benchmarks:
Note: The set of exported symbols might change over time, especially the facilities for defining and running benchmarks. Please do not :USE this package and :IMPORT-FROM it instead.
A statement like
(nested-benchmark (loop for key across keys do (benchmark (gethash key table))))
is a macro that expands into something like
(measure-execution-time (lambda (#:iterations723) (loop :repeat #:iterations723 :do (loop for key across keys do (touch (gethash key table))))) :overhead (lambda (#:iterations723) (loop :repeat #:iterations723 :do (loop for key across keys do (touch nil)))))
The function TOUCH is explicitly declared notinline, to prevent compiler optimization. The function MEASURE-EXECUTION-TIME invokes its argument functions with higher and higher numbers of iterations, until the difference between them is significant.
There are lies, damned lies, and benchmarks.