Skip to content

scymtym/sb-benchmarks

Repository files navigation

Introduction

Dependencies

  • A recent version of SBCL
  • For plotting: gnuplot (tested with version 4.6)

Running Benchmarks

  1. Set SBCL_WC to a directory containing a SBCL working copy.
  2. Run benchmarks with
    • sh run-benchmarks.sh to run all benchmarks with default result reporting style
    • sh run-benchmarks.sh FILES to run only the benchmarks in FILES.
    • sh run-benchmarks.sh --report-style plot to produce one or more plots of the results (requires gnuplot)
    • sh run-benchmarks.sh --report-style plot --compare-to SBCL-VERSION to produce plots comparing the SBCL version in ${SBCL_WC} to SBCL version SBCL-VERSION. This only works if benchmarks have previously been run for SBCL-VERSION.

Writing Benchmarks

New benchmarks should be added in the appropriate file in the project root directory.

All benchmarks have to use the with-benchmark macro to associate a unique name with each test and record benchmark results with that name. The macro also prevents a whole file of benchmarks from going up in smoke when a single benchmark signals an error. The basic syntax is

(with-benchmark (:name NAME)
  (measuring BODY))

where NAME is either a symbol or a list of symbols. All of these symbols have to be in one of the packages keyword, cl or sb-* to make test names READ able in a different SBCL image.

It is often desirable to analyze how the runtime (or memory allocation) of a piece of code scales in relation to some parameter. For this purpose, the with-benchmark macro accepts the :parameters keyword parameter::

(with-benchmark (:name name
                 :parameters ((input-size (:expt 10 (:iota 4)))))
  (measuring BODY-USING-INPUT-SIZE))

The above example causes BODY-USING-INPUT-SIZE to run four times with input-size successively bound to 1, 10, 100 and 1000 respectively.

If possible, benchmarks should not signal warnings during compilation or runtime and should not produce output in order to not clutter the output of the benchmark runner and make problems easier to identify.

About

Benchmarking suite for SBCL

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published