Module system for languages embedded in Common Lisp
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
demo Move language tests into Vernacular repo. Aug 20, 2018
elisp Rework elisp. Sep 5, 2018
lang Only resolve the source when it has no extension. Sep 23, 2018
tests Another test for file locals. Oct 2, 2018
LICENSE.txt Include explicit license. Aug 20, 2018
README.md Update readme. Oct 10, 2018
all.lisp Includes with dependencies. Sep 21, 2018
compile-to-file.lisp Minor: docstrings for packages. Oct 9, 2018
file-local.lisp Bug fix: check substring is bound. Oct 10, 2018
file-package.lisp Minor: docstrings for packages. Oct 9, 2018
hash-lang-syntax.lisp Merge branch 'master' of github.com:ruricolist/vernacular Oct 9, 2018
hash-table-module.lisp Minor: docstrings for packages. Oct 9, 2018
import-set.lisp Minor: docstrings for packages. Oct 9, 2018
importing.lisp Merge branch 'master' of github.com:ruricolist/vernacular Oct 9, 2018
include.lisp Includes with dependencies. Sep 21, 2018
lang.lisp Merge branch 'master' of github.com:ruricolist/vernacular Oct 10, 2018
module.lisp Minor: docstrings for packages. Oct 9, 2018
parsers.lisp Reimplement language support. Aug 30, 2018
shadows.lisp Reimplement language support. Aug 30, 2018
simple-module.lisp Merge branch 'master' of github.com:ruricolist/vernacular Oct 9, 2018
specials.lisp Merge branch 'master' of github.com:ruricolist/vernacular Oct 9, 2018
tests.lisp Merge branch 'master' of github.com:ruricolist/vernacular Oct 10, 2018
types.lisp Merge branch 'master' of github.com:ruricolist/vernacular Oct 9, 2018
universal.lisp Reimplement language support. Aug 30, 2018
vernacular.asd Fasls should depend on the version of Vernacular. Aug 31, 2018

README.md

Vernacular

Vernacular is a module system for languages embedded in Common Lisp. It is inspired by Racket.

Vernacular enables embedding languages where modules are lexical and are scoped to files. This is a scenario for which ASDF is useless, but which applies to almost all languages one might actually want to embed. Vernacular fills this gap.

Vernacular builds on Overlord.

Examples

Here are some example language embeddings:

  1. vernacular/demo/js. A simple demo language built on CL-JavaScript. Shows how to convert a pre-existing CL language implementation to work with Vernacular.

  2. Bosom Serpent. Shows how to wrap a foreign runtime (Python, using burgled-batteries) as an Vernacular module.

  3. cl-yesql. Lisp port of Clojure’s yesql. Includes a parser, and shows how (and why) to load the same file in different languages.

  4. Core Lisp. A hygiene-compatible implementation of the Lisp dialect ISLISP (itself a conceptual subset of Common Lisp). Shows how to use Vernacular to build “language towers.”

Embedding languages

Vernacular enables languages as libraries. Vernacular languages have several important properties:

  1. Languages are first-class. Modules live in their own files, just like Lisp code, and are compiled into FASLs, just like Lisp code.

  2. Languages can use any syntax. Unlike embedded DSLs, which are limited by what can be done with reader macros, full languages can use any parser they like.

  3. Languages are interoperable. Lisp code can import modules written in embedded languages, and modules written in embedded languages can import other modules in the same language – or even modules written in other languages.

  4. Languages are reusable. Support for meta-languages allows different languages to share the same parser or for the same language to be written in more than one syntax.

Specifying languages

A Vernacular module is a file in a language. The language of a module file can be specified in three ways.

The language can be specified as part of the file itself, with a special first line. The special first line looks like this:

#lang my-lang
....

This is called (following Racket) a hash lang.

The hash lang is the preferred way to specify a language. If you are creating a new language with a new syntax, this is what you should use. It takes precedence over all other ways of inferring the language.

Much of the time, however, you will be re-using an existing syntax. Sometimes this is the point, because it lets you employ existing tooling like editors, linters, etc. In this case we piggyback on Emacs’s syntax for specifying modes, with a special string in the first line:

# -*- mode: my-lang -*-

You can consult the Emacs manual for the details of the syntax.

The advantage of this approach is that the sequence between -*- markers does not have to appear at the beginning of the file; it can be commented out using the appropriate comment syntax. (If the file starts with a #! shebang, the mode can also be specified in the second line; this is also true of hash langs.)

Lastly, the language of a module can be specified as part of the import syntax. This lets you use files as modules without having to edit them at all, which may be useful for shared files you cannot edit.

Languages

In Vernacular, a language is just a package. The package exports a reader and an expander. The function bound to the symbol named read-module is the package reader. The macro bound to the symbol named module-progn is the package expander.

The important thing: when the package’s reader is called, that same package is also bound as the current package. It is then the responsibility of the reader to make sure any symbols it reads in, or inserts into the expansion, are interned in the correct package. (There is a shortcut for this, vernacular:reintern.)

(There is one exception to the rule of language=package. If another package exists, having the same name, but ending in -user, and this other package inherits from the original package, then this user package is the package that is made current while reading (and expanding). E.g. a file beginning with #lang cl would actually be read in using the cl-user package, not the cl package itself.)

Note that the reader is responsible for returning a single form, which is the module. That is, the form returned by the package reader should already be wrapped in the appropriate module-progn. The exported binding for module-progn is only looked up when the language is being used as the expander for a meta-language.

(Meta-languages are for language authors who want to reuse an existing syntax.)

Defining languages

Any package can be used as a hash lang, as long as its name is limited to certain characters ([a-zA-Z0-9/_+-]). Of course this name can also be a nickname.

(Note that resolution of package names is absolute, even in a Lisp implementation that supports package-local nicknames.)

It is recommended, although not required, that your language package inherit from vernacular/cl rather than from cl. The result is the same, except that vernacular/cl globally shadows Common Lisp’s binding and definition forms so they can, in turn, be shadowed locally by language implementations.

The package must at least export a binding for one of read-module, for direct use, or module-progn, for use with a meta-language. Preferably, it would export both.

If the syntax of your language makes it possible to determine exports statically, you should also define and export static-exports. If your language defines static-exports, then Vernacular can statically check the validity of import forms.

(This also has implications for phasing. If your language doesn’t provide a static-exports binding, then the only way Vernacular can expand a request to import all bindings from a module is by loading that module at compile time to get a list of its exports.)

Imports and exports

What Vernacular imports and exports are not values, but bindings. Bindings are indirect (and immutable): they refer to the module, rather than to the value of the export. This allows for modules to be reloaded at any time. It is even possible to unload modules.

Note that exports in Vernacular, with one exception, form a single namespace. This is in order to keep the notation for imports simple. Importing from a language with multiple namespaces into a language with multiple namespaces would create a Cartesian product problem.

The one exception is macros. A single namespace for run-time bindings and macros would not make sense in Vernacular where modules can be dynamically reloaded.

Finally, Vernacular allows local imports: imports that only take effect within the body of a with-imports form.

Simple modules

Most of the time, your language’s package expander will return a simple-module form.

(vernacular:simple-module (#'moo)
  (defun make-moo (o)
    (concat "M" (make-string o :initial-element #\o)))

  (defun moo (&optional (o 2))
    (print (make-moo o))))

This exports a single name, moo, bound to a function that says “Moo” with a varying amount of “oo”.

What makes simple modules simple is that they cannot export macros. If you do want to export macros, you need something more complex (see below).

The simple-module form builds on the support for internal definitions in Serapeum (the local macro), and shares its limitations with regard to the precedence of macro definitions. Macro definitions must precede all function or variable definitions, and all expressions.

To be clear: you can define macros in a simple module (with defmacro), you just can’t export them.

Macro exports

Vernacular’s syntax for import and export supports macros.

The ability to export macros from modules is not useful in itself. It only becomes useful in the presence of certain forms of macro hygiene. After experimenting with different ways to do this, I have concluded that the correct thing to do, if you want your language to be able to export macros, is to embed a hygiene-compatible language in Lisp, and then compile your language to that.

I’m not being flippant. Embedding a hygiene-compatible language in CL is not just doable; it’s already been done. As a proof of concept, I have converted Pascal’s Costanza’s hygiene-compatible implementation of ISLISP in Common Lisp (“Core Lisp”) to work with Vernacular’s module system. This version of Core Lisp lives in its own repository.

How macro exports are supported is one aspect of the Vernacular module system that is very likely to change.