Update tree-sitter

[enyst]

Description

TestGenEval benchmark seems to need an upgraded tree-sitter. This PR updates tree-sitter to version 0.24.0.

Changes

• Updated tree-sitter from 0.21.3 to 0.24.0
• Added dynamic language imports to support any tree-sitter language package
• Added Python, JavaScript, and TypeScript as default languages
• Created a compatibility layer to handle the new tree-sitter API

Testing

• All tests are passing
• All pre-commit hooks pass

References

• Upgrade tree sitter OpenHands#6740

[neubig]

OK, I think with OpenHands's help I managed to make this work. It seems like the tree-sitter-languages has not been updated recently, and we should just build our own compatibility library.

[enyst]

Thank you!

[ryanhoangt]

Thanks, this is interesting and LGTM! @enyst @neubig do you want to test this PR in the OH repo before we merge this?

[neubig]

Thanks @ryanhoangt , I tried it quickly in OpenHands by providing this prompt:

We are upgrading a dependency of the OpenHands library (openhands-aci) here: https://github.com/All-Hands-AI/openhands-aci/pull/73

Please modify pyproject.toml in the OpenHands folder to install openhands-aci from the branch associated with this PR, instead of from where it is normally installed (pypi). Then follow the Python unit testing procedure described in py-unit-tests.yml exactly.

and it worked after OpenHands upgraded the tree-sitter version in OpenHands as well, so I think we're good.