sxpyr: a flexible Lisp reader for Python

Parse s-expressions, edn, and a variety of lisp dialects.

Introduction

Currently supports something approximating the union of Common Lisp, Scheme, Racket, Clojure, Hy, Fennel, and Emacs Lisp.

Since sxpyr is designed to support a variety of different dialects, a second pass is almost always needed in order to match the output of native readers. This second pass is a good place to raise additional syntax errors for the dialect you are working with. Some dialects have already been implemented.

Due to deeper differences between dialects, in some cases it is necessary to configure the reader in advance. This happens in cases where a particular form is valid in two different dialects and one or both parses are sufficiently divergent that it is not possible to recover the correct parse for the other dialect.

At the moment the reader cannot be modified at runtime. This means that it cannot be used to parse named readtables or things like Racket’s #lang at-exp. In the future it might be possible to pass the generator through to the functions that handle dispatch macros and make it possible to register new reader macros, but that is well out of scope at the moment.

Originally this library was intended to be vendored, however since then it is significantly increased in complexity. A stripped down version that is suitable for parsing a minimal sane dialect that can be used as a configuration language may be added in the future.

Usage

git clone https://github.com/tgbugs/sxpyr.git
pushd sxpyr
pip install --user -e .[cli]
python -m sxpyr.cli parse test/data/nested-splicing-unsyntax.rkt test/data/plist-test.sxpr
popd

Testing

In order to run the tests you will need the following repos.

• sbcl
• ccl
• emacs
• org-mode
• clojure
• spec.alpha
• code.specs.alpha
• tawny-owl
• hy
• Fennel
• txr

For now you will also need the following installed on your system.

• xemacs
• app-xemacs/xemacs-packages-all
• guile
• gambit

Fuzzing

You will need a copy of afl.

pip install --user -e .[fuzz]
pushd test
mkdir initial-inputs
cp data/plist-test.sxpr initial-inputs/  # other seed files can be added as well
py-afl-fuzz -m 400 -i initial-inputs/ -o fuzzing-results/ -- python -m sxpyr.cli parse --fuzz

python -m sxpyr.cli parse test/fuzzing-results/crashes/*

Note that initial inputs cannot be symlinks becuase afl creates hardlinks to the input files.

Also note that afl seems to die after 175k execs?

Sometimes it also seems that some errors are escaping the except block and making it be classified as crashes?