tree-saw generates random programs from tree-sitter grammars and feeds them back to tree-sitter, and/or to a compiler in syntax-checking mode. This allows it to automatically find grammar problems, tree-sitter bugs, and potentially also bugs in the compiler.
tree-saw precomputes the expected recursion depth for each rule in the grammar and uses that information to guide weighted sampling of CHOICE rules. REPEAT lengths are drawn from a Poisson distribution and regular expression matches are generated using randexp.js. The resulting probability distribution favors outputs of reasonable size that are well-suited for automated testing.
When encountering a program that causes a compiler or grammar error, tree-saw "prunes" the syntax tree in accordance with the grammar's rules to find the smallest version of the program that still generates an error. The final output is a compact failure example that lets a human quickly identify (and hopefully fix) the underlying bug.
tree-saw can expose the following types of issues:
- "Type II" grammar bugs: Bugs where a syntactically invalid program is accepted by a grammar. These manifest themselves as errors when compiling the generated output. Running tree-saw on any existing tree-sitter grammar rapidly reveals hundreds of such bugs.
- tree-sitter core bugs: Bugs inside tree-sitter itself. These can show themselves as the grammar being unable to correctly parse the output generated by itself. This occurs with some frequency, and I'm still investigating whether such failures are caused by tree-sitter or by bugs in tree-saw's generator.
- Compiler bugs: Bugs where a syntactically valid generated program results in a compiler syntax error. Since most compilers have parsers that are of much higher quality than their tree-sitter equivalents, such finds are likely to be extremely rare, and in almost all cases indicate a grammar problem instead.
tree-saw can not automatically detect "type I" grammar bugs, where a syntactically valid program is incorrectly rejected by a grammar. However, the generated programs often reveal edge cases that lead to manual identification of such bugs.
$ tree-saw --grammar tree-sitter-c --compiler 'gcc -fsyntax-only -xc -' c.json
>>>>>RESULT
>>>>>SOURCE
extern " " { }
<<<<<SOURCE
>>>>>AST
(translation_unit (linkage_specification (string_literal) (declaration_list)))
<<<<<AST
>>>>>ERROR
Compiler error:
<stdin>:1:8: error: expected identifier or ‘(’ before string constant
<<<<<ERROR
<<<<<RESULT
For a full list of options, run tree-saw without arguments.
Copyright © 2018 Philipp Emanuel Weidmann (pew@worldwidemann.com)
Released under the terms of the MIT License