Adding new languages to Semantic
This document outlines the process for adding a new language to Semantic. Though the Semantic authors have architected the library such that adding new languages and syntax requires no changes to existing code, adding support for a new language is a nontrivial amount of work. Those willing to take the plunge will probably need a degree of Haskell experience.
- Find or write a tree-sitter parser for your language. The tree-sitter organization page has a number of parsers beyond those we currently support in Semantic; look there first to make sure you're not duplicating work. The tree-sitter documentation on creating parsers provides an exhaustive look at the process of developing and debugging tree-sitter parsers. Though we do not support grammars written with other toolkits such as ANTLR, translating an ANTLR or other BNF-style grammar into a tree-sitter grammar is usually straightforward.
- Create a Haskell library providing an interface to that C source. The
haskell-tree-sitterrepository provides a Cabal package for each supported language. You can find an example of a pull request to add such a package here, and a file providing:
- A bridged (via the FFI) reference to the toplevel parser in the generated file must be provided (example).
- A way to retrieve
tree-sitterdata used to auto-generate syntax datatypes using the following steps. During parser generation, tree-sitter produces a
node-types.jsonfile that captures the structure of a language's grammar. The autogeneration described below in Step 4 derives datatypes based on this structural representation. The
node-types.jsonis a data file in
haskell-tree-sitterthat gets installed with the package. The function
getNodeTypesPath :: IO FilePathis defined to access in the contents of this file, using
getDataFileName :: FilePath -> IO FilePath, which is defined in the autogenerated
- Create a Haskell library in Semantic to auto-generate precise ASTs. Create a
semantic-[LANGUAGE]package. This is an example of
semantic-python). Each package needs to provide the following API surfaces:
Language.[LANGUAGE].AST- Derives Haskell datatypes from a language and its
Language.[LANGUAGE].Grammar- Provides statically-known rules corresponding to symbols in the grammar for each syntax node, generated with the
mkStaticallyKnownRuleGrammarDataTemplate Haskell splice (example).
Language.[LANGUAGE]- Semantic functionality for programs in a language (example).
Language.[LANGUAGE].Tags- Computes tags for code nav definitions and references found in source (example).
- Add tests for precise ASTs, tagging and graphing, and evaluating code written in that language. Because tree-sitter grammars often change, we require extensive testing so as to avoid the unhappy situation of bitrotted languages that break as soon as a new grammar comes down the line. Here are examples of tests for precise ASTs, tagging, and graphing.
To summarize, each interaction made possible by the Semantic CLI corresponds to one (or more) of the above steps:
This sounds hard. You're right! It is currently a lot of work: just because the Semantic architecture is extensible in the expression-problem manner does not mean that adding new support is trivial.
What recent changes have been made? The Semantic authors have introduced a new architecture for language support and parsing, one that dispenses with the assignment step altogether. The
semantic-ast package generates Haskell data types from tree-sitter grammars. As compared with the historic process used to add new languages, these changes entire obviate the process of 1) assigning types into an open-union of syntax functors, and 2) implementing
Evaluatable instances and adding value effects to describe the control flow of your language.