Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compiled parser produces different result than tree sitter cli parse command #447

fachammer opened this issue Sep 18, 2019 · 3 comments


Copy link

commented Sep 18, 2019

So I'm developing a tree-sitter grammar for Lux. I've come across an issue where running the parser from node produces a different result than from the tree-sitter cli parse command. I've made a repo that illustrates the problem.

Steps to reproduce

  1. clone the bug-reproduction repository and checkout the problematic commit

     git clone
     cd bug-reproduction
     git checkout 56d588dd0a21de657139ba5d83517cb8c0abd9e1
  2. install dependencies. The bug reproduction repository already contains the npm-packaged tree-sitter-lux parser as a local file dependency

     npm install
  3. run the parser test file. This parses the example.lux file in the same directory and prints out the syntax tree

     node lux-parser.js
  4. this prints

     (source_file (form (inline_comment)))

    Actually it should print an ERROR node somewhere because the example.lux file is not valid Lux code. In the example.lux file the comment inside the parentheses is a line comment and should consume the rest of the line which should result in an error because the closing parenthesis is then missing.

Contrast this to parsing the same file from within the tree-sitter-lux repository via the tree-sitter cli parse command

  1. checkout tree-sitter-lux repository on proper commit and install dependencies

     git clone
     cd tree-sitter-lux
     git checkout 9f5ba967700c1cdeda95786938491aa7396f4021
     npm install
  2. run the tree-sitter parse command with the example.lux file from before

     ./node_modules/.bin/tree-sitter parse [local path to example.lux file in bug-reproduction repo]
  3. this prints

     (source_file [0, 0] - [1, 0]
         (ERROR [0, 0] - [1, 0]
             (inline_comment [0, 1] - [0, 11])))
     ../bug-reproduction/example.lux       0 ms    (ERROR [0, 0] - [1, 0])

    This is actually what I would expect from correctly parsing the example.lux file.

  4. optionally you can pack the tree-sitter-lux package and copy it into the bug-reproduction repo to see that the problem still occurs

     npm pack
     cp tree-sitter-lux-0.0.1.tgz [local path to bug-reproduction repo]

This seems like a bug to me. Regardless of whether the grammar I wrote actually correctly parses Lux code I would still expect the same results from the compiled parser and the parse command in the tree-sitter CLI.

And btw: tree-sitter is awesome! Makes it a lot easier to write proper parsers for language tooling


This comment has been minimized.

Copy link

commented Sep 18, 2019

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.95. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

@issue-label-bot issue-label-bot bot added the bug label Sep 18, 2019

This comment has been minimized.

Copy link

commented Sep 19, 2019

Ok, so I think it's not a problem with tree-sitter but there was a problem in how I packaged the tree-sitter-lux module. I originally only included the build folder that is produced by node-gyp. And apparently in the build folder there was an parser of the grammar that would allow the erroneous file. It didn't occur to me that the build folder only gets updated on the second call to npm install.

Since then I changed the way how I package the tree-sitter-lux module by including the generated src folder and the binding.gyp file and excluding the build folder. As I understand it, this way the parser gets compiled when a user installs the tree-sitter-lux module.

@fachammer fachammer closed this Sep 19, 2019

This comment has been minimized.

Copy link

commented Sep 19, 2019

Yeah, I recommend including the C source code but not the build folder, since the binaries in your build folder will only work for people who are using the same platform and architecture as you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
2 participants
You can’t perform that action at this time.