[tree-sitter] add multiple ports

[kylo252]

Description

This PR aims to provide support for building tree-sitter parsers across platforms

• add tree-sitter-common
  provides vcpkg_ts_parser_add helper functions for installing tree-sitter parsers
• add tree-sitter-{bash,c,cpp,css,javascript,json,lua,python,rust,typescript,yaml}

Background about parsers

• all of tree-sitter’s parsing functionality is exposed through C APIs
• each parser is composed of at least one C file parser.c and sometimes a scanner.{c,cc}
• parser files themselves do not include any external C/C++ dependencies
• each parser has an ABI version to indicate compatibility with libtree-sitter

List of some popular consumers of tree-sitter parsers

• https://github.com/github/linguist
• https://github.com/github/semantic
• https://github.com/neovim/neovim
• https://github.com/Wilfred/difftastic
• https://github.com/curlconverter/curlconverter
• https://github.com/simonbs/Runestone
• https://github.com/zee-editor/zee
• https://github.com/emacs-tree-sitter/elisp-tree-sitter

[github-actions]

You have modified or added at least one vcpkg.json where you should check the license field.

If you feel able to do so, please consider adding a "license" field to the following files:

• ports/tree-sitter-common/vcpkg.json

Valid values for the license field can be found in the documentation

[github-actions]

You have modified or added at least one vcpkg.json where you should check the license field.

If you feel able to do so, please consider adding a "license" field to the following files:

• ports/tree-sitter-common/vcpkg.json

Valid values for the license field can be found in the documentation

[BillyONeal]

How are these related to the port tree-sitter?

[kylo252]

How are these related to the port tree-sitter?

I think these will have two major use-cases

• dynamically loaded (probably with manifest mode)
• statically built alongside tree-sitter

My original plan was introducing these as features, but I decided that they needed dedicated ports instead of trying to lump them all in one. However, now I'm unsure if I should expose them through the main library or not. Note: it's useful that we'd be able to install parsers regardless!

I've also planned to build the tree-sitter-cli tool (used as a code-generator) internally, but it requires cargo and it turned out to be a lot of work, see #20619, so for now it's vendored through npm

https://github.com/microsoft/vcpkg/blob/002f3039c3df12b284e1fecfd5feaf110bb0d48e/ports/tree-sitter-common/vcpkg_ts_parser_add.cmake#L25-L43

[JackBoosY]

so for now it's vendored through npm

Use npm to install the dependencies automaticly is a bad idea.
We should not limit the installation tools that users can use. Users should choose system-related dependencies to install them.
For example:

• Ubuntu-apt
• CentOS-yum
• OSX-brew

[JackBoosY]

If your project is too large, we should split it into several separate PRs.

[Adela0814]

Pinging @kylo252 for response. Is work still being done for this PR?

[kylo252]

Pinging @kylo252 for response. Is work still being done for this PR?

@Adela0814, this is already complete if we keep going with installing tree-sitter-cli with npm.

I've been hoping to be able to build it from source, but bootstrapping rust is the blocker, see #20619

An alternative would be to install tree-sitter-cli as a tool with npm, it would be its own portfile in that case, maybe we could go with that for now?

[Adela0814]

An alternative would be to install tree-sitter-cli as a tool with npm, it would be its own portfile in that case, maybe we could go with that for now?

Yes, I think it's a good idea.

[github-actions]

You have modified or added at least one vcpkg.json where you should check the license field.

If you feel able to do so, please consider adding a "license" field to the following files:

• ports/tree-sitter-common/vcpkg.json

Valid values for the license field can be found in the documentation

[github-actions]

You have modified or added at least one vcpkg.json where you should check the license field.

If you feel able to do so, please consider adding a "license" field to the following files:

• ports/tree-sitter-common/vcpkg.json

Valid values for the license field can be found in the documentation

[github-actions]

You have modified or added at least one vcpkg.json where you should check the license field.

If you feel able to do so, please consider adding a "license" field to the following files:

• ports/tree-sitter-common/vcpkg.json

Valid values for the license field can be found in the documentation

[github-actions]

You have modified or added at least one vcpkg.json where you should check the license field.

If you feel able to do so, please consider adding a "license" field to the following files:

• ports/tree-sitter-common/vcpkg.json

Valid values for the license field can be found in the documentation

[github-actions]

You have modified or added at least one vcpkg.json where you should check the license field.

If you feel able to do so, please consider adding a "license" field to the following files:

• ports/tree-sitter-common/vcpkg.json

Valid values for the license field can be found in the documentation

[BillyONeal]

(Of course you don't have to include the CMake output and stuff, I just included that to show that those 3 files together build something that works)

[kylo252]

@BillyONeal, I've added a basic integration with pkg-config, along with usage instructions.

here's an example of how to use it: https://github.com/kylo252/tree-sitter-registry/tree/master/example

[Adela0814]

Convert this PR to draft since there is no progress. Please ping us if this PR is ready for review again.

[Adela0814]

Closing this PR since it seems that no progress is being made. Please ping us to reopen if work is still being done.