Fix parsing UTF-16 files

[tamasvajk]

Files with UTF-16 LE or BE encoding produce error nodes. Empty file saved with UTF-16 BE encoding in VS Code produces the below:

❯ tree-sitter parse a.cs
(compilation_unit [0, 0] - [0, 2]
  (ERROR [0, 0] - [0, 2]
    (ERROR [0, 0] - [0, 2])))
a.cs    0 ms    (ERROR [0, 0] - [0, 2])

The same file with UTF-8 encoding:

❯ tree-sitter parse a.cs
(compilation_unit [0, 0] - [0, 0])

[damieng]

There isn't anything in the grammar that is specifically UTF-8 and we're using tree-sitter ^0.20.0 so I'm not sure what needs to be done here - I checked a couple of the other repos but didn't see anything about UTF-16 there either. I would have expected tree-sitter itself to take care of that as it reads the file.

[tamasvajk]

Sorry for the noise, I think this issue can be closed. I think the CLI is calling parse, which has a comment Parse a slice of UTF8 text., which explains why this happens. The documentation details TSInput, which does have an encoding.