HLisp

HLisp Starter

HLisp is a lisp implementation built atop ClojureScript that attempts to mitigate several significant problems with interactive web development. A prototype implementation demo can be viewed here.

Motivation, Project Goals

This project aims to address some of the causes of incidental complexity in frontend (user interface) web development.

• Too many disparate domains.

  The programmer must manage too many tightly coupled, fundamentally incompatible subsystems: HTML markup, CSS rules, JavaScript behaviors/events, HTML templating tools, etc. The HTML itself may be generated by server side code that the programmer must write in yet another programming language.

  These domains do not have a common set of primitives, means of abstraction, or means of composition. This results in having no coherent meaning for entities in these systems: entities do not have a single identity or way to manage references to them, they can't be composed with each other in a uniform way.

  The primary technique for managing complexity is the use of modular code and the separation of concerns. The tight coupling of disparate domains torpedoes this approach. Concerns which may be orthogonal in one domain may not be orthogonal in one of the other domains. The ability to create more specific abstractions by composition of more general ones is greatly restricted.

  Goal: Single, unified set of primitives, means of abstraction, and means of composition.

• Pervasive use of global references and shared, mutable state.

  The lack of a common set of primitives or means of abstraction/composition dictates the use of global identifiers to pass information between domains. This could be setting the id attribute of an element, or it could be a more complicated selector-based mechanism.

  Goal: Unified evaluation environment and namespace with lexical scope for all entities.

• Callback based event handling.

  The browser event handling system is designed around a callback mechanism. Callbacks generally have the void type (aside from those that return boolean false to cause side effects). Thus, there is no way to compose them, and they must rely entirely on side effects and shared state to perform their tasks.

  Event streams are not first-class entities, so there is no way to form abstractions around them. They can't be composed with each other. They can't be locally referenced. This forces even more shared state upton the system.

  Goal: Push-based event system, first-class event streams, and composable event handlers.

Additional Requirements

• Don't get rid of HTML.

  Designers will need to work with HTML markup. The browser renders it. The elimination of HTML markup would necessarily introduce complexity in the form of an interface between the designer and the programmer, at the very least. So if possible, keep HTML markup as the primary means of building the front end DOM structure.

• Simple to learn, minimal tooling to get started.

  The system should be learnable by the motivated designer. They shouldn't have to be skilled programmers to use it. The system should expose useful functionality to the user without requiring a lot of boilerplate setup code.

• Robust set of libraries.

  It must be practical to build extensive libraries of modular code. This means some mechanism for managing references, namespaces, etc. Some way to distribute libraries and include them in projects is also key.

HTML is Semantically Compatible with Lisp

To achieve the above goals, it is necessary to perform all front-end processing in a single, unified evaluation environment. The semantic structure of HTML suggests a Lisp type language, for the following reasons:

• HTML structure is fundamentally lists.

  A DOM element is semantically a list structure. Tools like hiccup use this characteristic to advantage.

• DOM appendChild is semantically equivalent to Lisp function application.

  With one caveat (elaborated below), appending children to a DOM element is semantically equivalent to Lisp function application. This is also the means of composition in both systems.

A Possible Solution

Is Lisp an acceptable environment for the full user interface stack? Does Lisp provide a solution to the above problems?

• The HTML document can be evaluated as a Lisp program.

  Given that HTML and Lisp are semantically compatible, it's possible to actually evaluate an HTML document as a Lisp program. The browser could be made to evaluate the body of the document when the page loads and replace the body contents with the result.

• Lisp provides rich means of abstraction.

  The abstraction-building capability of Lisp is well known.

• Lisp is flexible enough to encompass all of the disparate domains.

  The power and flexibility of a full-featured Lisp is more than capable of handling the demands of user interface programming. Templating tools, event handling, etc. can all be implemented in Lisp.

• Evaluating HTML as Lisp unifies the evaluation environment.

  Since the markup is code that is evaluated in some environment, lexical scope, namespaced references, and all the other Lisp mechanisms for managing references is available in both the library code and the markup itself. The markup elements become first-class entities.

Choice of Lisp Implementation

If Lisp is to be the environment, the next question is, "Which one?" After some experimentation ClojureScript was chosen. Possible choices were:

ClojureScript

ClojureScript is a Lisp that compiles to JavaScript.

• Most of the features of Clojure.
• Persistent, immutable data structures, and lazy sequences.
• Excellent type system with records, "deftype" types, protocols, multimethods, etc.
• Macros. (Currently macros are somewhat limited, but that will probably improve over time.)
• Good JavaScript interoperability.
• Uses Google's Closure library and compiler.
• Supports aggressive optimization, dead code removal, etc.
• Hlisp compiler can use the Clojure reader to parse forms—it's the same as the cljs reader.
• There are excellent tools for managing Clojure projects that can be used immediately to manage hlisp projects.
• A natural choice if the compiler is to be written in Clojure.

LispyScript

LispyScript is JavaScript with Lispy syntax and macros.

• Excellent JavaScript interop, since it is JavaScript.
• Has macros, monads, templates

Custom Interpreter

This approach was tried initially but there were issues with JavaScript interoperability and performance was not stellar.

Issues

What are the important issues that must be solved?

• Semantic differences between Lisp and HTML.

  HTML is sort of a Lisp-1.5 The car of the list representation of an HTML element is the tag name. This must be a symbol and can't be another list. This is a departure from Lisp's uniform treatment of list elements, where the evaluator evals the car as well as the cdr. HTML semantics can't express ((f x) y z), for example.

  This is a result of the fact that there are no HTML primitives other than the Element node, which means that all elements are semantically lists. This makes it impossible to express (f) (meaning function application with no arguments), without giving up the capability to express f (function as a value).

  Solution: Give up ability to call functions without arguments from HTML markup. This can be accounted for by simply requiring that all functions that are exposed to the user take an argument, even if they discard it. This preserves the ability to reference functions as values, which is key for the formation of higher-order functions, and is central to Lisp. In a functional system, zero-argument functions are rare, as they can only produce side effects. Additionally, some kind of funcall or call mechanism can be used to accomodate the former case.

• Syntactic differences between Lisp and HTML.

  Clearly, HTML is vastly more verbose than Lisp is. It would be exceedingly tedious to write a lot of functional code in HTML. A mechanism must be available by which the programmer can program in Lisp or HTML, as required. HTML tags also have different syntactic requirements from Lisp symbols. There are some symbols which are not valid HTML tags.

  Solution: The document <head> must contain a <script type="text/hlisp"> element. Lisp source can be contained in this script element, to be evaluated in the environment of the page. Furthermore, it's possible to simply refrain from naming user-level functions and entities with names that are not compatible with both.

• Restricted set of primitives in HTML.

  HTML has only one primitive type: the list.

  Solution: Numbers, strings, etc. can perhaps be read as special lists. For example, the number 42 could be expressed as <val_number>42</val_number> and the string "hello" as <val_string>hello</val_string>. This is cumbersome, but it's offset by the fact that this type of coding can be done in a library or in the lisp <script> tag in the document <head>.

• Code libraries and namespaces.

  How will library code be managed?

  Solution: ClojureScript has excellent support for libraries, namespaces, import and export of references, etc.

• Event handlers and dynamic interface.

  How will the goals of first-class event streams and composable callbacks be satisfied? Is there existing library code that can be used in situ or with reasonable modification?

  Solution: The flapjax library is an excellent implementation of FRP in the browser. It provides all the features required.

• Cross-browser DOM API library.

  Which cross-browser DOM API library should be used?

  Solution: Hlisp core will be built on cljs, which includes the Google Closure DOM libraries. However, projects built on top of the core could use other more user-friendly libraries.

• HTML templating tools.

  How will templating and HTML processing work in HLisp?

  Solution: None, yet. However, access to macros, powerful list processing libraries, zippers, and so forth in cljs suggests that tools could be developed that far exceed the capabilities of current templating systems without giving up the unified evaluation environment.

• Application data, server-client interaction.

  How does the front end get the data it needs from the server?

  Solution: ClojureScript provides access to the XmlHttpRequest JavaScript subsystem, which can be integrated with Flapjax to provide a flexible, simple mechanism for this.

Architecture

An hlisp project consists of the following components:

• The hlisp compiler. Implemented as a leiningen plugin.

• The hlisp library. Provides the hlisp.env namespace containing the implementation for the ElemNode and TextNode deftypes. All of the HTML elements, p, div, span, etc., are defined in the page namespace.

• ClojureScript libraries. Existing libraries can be used without modification in most cases. The only issue is names that are not valid HTML tags, but the solution is usually simply to alias those refs.

• Clojure macros. Macros used in ClojureScript must be written in Clojure, which has certain limiting consequences. Hopefully full cljs macro support is on the way, though.

• HTML source files. The actual HTML source files that make up the site.

• External JavaScript libraries. JavaScript libraries such as jQuery and Flapjax, for instance, which can be used in the ClojureScript environment.

Compiler

The hlisp compiler scans for HTML source files. For each source file found it does the following:

1. Extract the <script type="text/hlisp"> tag from the document <head>.
2. Use the Clojure reader to read the text contents as lisp forms.
3. Extract the namespace declaration (must be the first form).
4. Insert a cljs "prelude" in between the namespace declaration and the rest of the forms from step 2. This prelude has definitions for things that need to be provided by hlisp in the page namespace. When/if support for :use without :only is available in cljs the prelude can be removed.
5. The document <body> is parsed as a cljs form, wrapped in a function definition, and appended to the other forms. This defines the page's hlispinit function, which is marked as ^:export, so that the Closure compiler won't munge its name.
6. The cljs forms are written to a source file that will be compiled by the cljs compiler.
7. Initialization JavaScript for the page is generated and inserted into the HTML output, which is written to a file.

Then, when all HTML source files have been processed, the cljs compiler is invoked, producing a single JavaScript file, main.js.

References

1. Flapjax: A Programming Language for Ajax Applications

Related projects:

• Cappuccino unifies evaluation environment by eliminating page HTML markup. Compiles Objective-J into HTML and JavaScript.
• Hop eliminates page HTML in favor of Lisp. Encompasses back-end web services as well as front-end UI. Generates HTML, JavaScript, SQL, etc. for each tier from Lisp source code.

Previous work:

• Hlisp previous attempt at interpreted Lisp.
• Declarify.js exploration of FRP in the browser.
• Golf initial attempts.

HLisp Comments

When commenting out a form in a .hl file, you should use #_ rather than (comment ...). So this should work fine:

#_ (h3 (text "hello, ~{urpc/user}!"))

But this causes problems:

(comment (h3 (text "hello, ~{urpc/user}!")))