whitequark/parser translation

[kddnewton]

We want to be able to translate from the Syntax Tree syntax tree to the whitequark/parser syntax tree. This should enable developers that already have syntax trees parsed with Syntax Tree to not have to parse again with the parser gem if they want to run static analysis tools like rubocop.

Note that the hypothesis with this functionality is not that going from source to tree will be faster going through this translation process. I would be shocked if that were ever true. Instead, we're guessing that going from Syntax Tree's AST to parser gem's AST is faster than going from source to parser gem's AST. This is because you would be using this API when you already have the tree in memory.

The API for this lives in lib/syntax_tree/translation/parser.rb. You use it by running:

require "syntax_tree"
require "parser"

buffer = Parser::Source::Buffer.new("(string)", 1)
buffer.source = "1 + 2"

node = SyntaxTree.parse(buffer.source)
node = SyntaxTree::Translation.to_parser(node, buffer)

pp node
                                
# s(:send,                                         
#   s(:int, 1), :+,                                
#   s(:int, 2)) 

To test this functionality, we vendor the parser gem under test/suites/parser and add "test/suites/parser/test/test_parser.rb" to our list of tests in the Rakefile. We mirror a couple of files that have to be required and override the assert_parses to effectively mirror their test suite with our own assertions.

All of the tests are passing in terms of the correct types of nodes and their children (this is checked by running bundle exec rake test). However, the parser gem has some relatively complex location information stored on every node (these are subclasses of the Parser::Source::Map class). You can see in the translation file that we have it being stored on a lot of nodes, but not all of them.

To finish this feature and close this issue, we need to match up the source maps correct. We already have location information on Syntax Tree nodes, we just need to map it to parser gem nodes. Sometimes this involves finding additional tokens in the source since the parser gem ends up storing more information than the parser gem at the moment. (You can see how this is done with the source_range_find method.)

By default the test suite doesn't check that the locations are equivalent, however you can toggle on that behavior with the PARSER_LOCATION environment variable. Toggling that on as of writing this results in 178 failures. In order to close this issue, that number needs to be 0.