ts-clojure

Testing and development bits for tree-sitter-clojure

This repository houses bits to aid in performing tests on real-world Clojure code along with some associated instructions.

See here for some background.

One-time (Mostly) Setup

There is some one-time (mostly) setup necessary before tests can be executed. This includes:

• Verifying / installing prerequisites
• Ensure this repository is available
• Verifying tree-sitter setup
• Ensure tree-sitter-clojure is available
• Fetching source code samples
• Tweaking ts-clojure's settings

Consider working with this repository in a new user account. This is not required, but you might want to read near the end of the document for potential consequences of not doing so.

Prerequisites

The prerequisites are what you typically need to work with tree-sitter, with the exception of Babashka which is used for task automation:

• tree-sitter cli
• tree-sitter dependencies
• babashka

Probably, you'll want to have git too (^^;

See here for more details.

Ensure ts-clojure is Available

If you are reading this by looking in ts-clojure as a subdirectory of tree-sitter-clojure, nothing special needs to be done and you can skip the rest of this step. Note that once Babashka is available, you can repeatedly use:

bb check-setup

from within the ts-clojure directory as you follow the steps below to see how well things are progressing.

Otherwise, clone ts-clojure (this repository) somewhere local.

Verify tree-sitter setup

Verify what version of tree-sitter you have installed and confirm that you know where it looks to find parser repositories.

See here for more details.

Ensure tree-sitter-clojure is Available

There are a few ways to set up ts-clojure to be used with tree-sitter-clojure:

1. ts-clojure is a subdirectory of tree-sitter-clojure
2. tree-sitter-clojure is a subdirectory of ts-clojure
3. no particular direct relationship

Default

Method 1 is the default and if this is chosen, nothing needs to be changed and you can skip the rest of this step.

Non-default

For the other two methods, clone tree-sitter-clojure to a location such that the tree-sitter cli can find the resulting directory. Check out an appropriate branch, tag, or commit as desired.

Note that the grammar-dir setting in conf/conf.clj should be a filesystem path that resolves to or is the tree-sitter-clojure directory. So either change the setting or make an appropriate symlink.

See here for more details.

Prepare source samples

This repository does not contain source code samples. To get meaningful testing over real-world code, it's necessary to arrange for some samples.

To fetch some source code samples:

• For clojars:

  • Change working directory to the clojars-samples subdirectory
  • Fetch 11 jars by: bb fetch-jars 11
  • Extract the jars by: bb extract-jars

• For clojuredart samples:

  • Change working directory to the clojuredart-samples subdirectory
  • Fetch some clojuredart code by: bb fetch-samples

Change the value of repos in conf/conf.clj to specify which set of samples to test against.

Note that in the case of clojars, if you are serious about testing, please consider getting more jars.

See here for more details.

Tweak settings

ts-clojure's scripts can be configured a bit via the file conf/conf.clj.

Some included settings are:

• abi - ABI number to use when generating parser.c from grammar.js
• grammar-dir - path to cloned tree-sitter-clojure directory
• repos - which set of source samples to test against
• ts-bin-path - path to or name of tree-sitter cli binary

See here for more details.

Things You Can Do

Generate parser.c

To generate tree-sitter-clojure's src/parser.c file:

bb generate-parser

This will invoke tree-sitter's generate subcommand:

• using the ABI number specified in conf/conf.clj via abi and
• using the --no-bindings argument to avoid generating binding code

Build and Install Shared Library

To build and install a shared library based on the generated parser.c:

bb corpus-test

This will run tree-sitter-clojure's corpus tests via tree-sitter test, which has a side-effect that achieves the desired aim.

See here for more details.

Run Real-World Code Tests

To test the parser on real-world code:

bb parse-samples

This will invoke tree-sitter's parse command across a set of source code samples.

Which set of samples is tested against is chosen by adjusting the repos value in conf/conf.clj appropriately. Assuming the samples have been obtained, the value can be one of:

• clojars
• clojuredart

See here for more details.

Misc Notes

About Being Future-Proof

At this time, tree-sitter cli subcommand backward compatibility does not appear to be a high priority, so it may be good to expect to have to adjust some tree-sitter invocations in the various Babashka (.clj) scripts.

As a specific example of a potential backward incompatibility, at the time of this writing, there are plans to phase out the build-wasm subcommand.

Windows Support

Have not tested yet but might work via mingw-w64 / msys2 or similar. No idea about WSL, not a fan and haven't tested.

Isolation

I had trouble at various points with tree-sitter using things that I was unaware of. I believe this was the result of a combination of testing with different versions of tree-sitter (various configuration and operation file paths changed between versions), having multiple versions of grammars under my home directory, along with tree-sitter's automatic compilation (and my not understanding how it works in detail). If you want to avoid some of that kind of thing, you might want to consider some kind of isolation / simplification mechanism such as using a fresh user account when working with this repository.