Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*/tree-sitter-*: support new upstream bindings #35750

Closed
wants to merge 8 commits into from

Conversation

arthurzam
Copy link
Member

@arthurzam arthurzam commented Mar 14, 2024

So what we have in this package of patches for now:

  • bumps for the base tree-sitter stuff (lib, cli, python api)
  • tree-sitter-grammar.eclass: support for new upstream makefile
    • needed me to change the expected${S} variable.
      This is somewhat backward incompatible, but I think this is worth it (I'll fix all ebuilds which define themselves S=)
  • tree-sitter-grammar.eclass: support opt in python bindings
    • for now I added support only for python (I'll need those for future pkgcheck), but there is rust, go, swift, js...
    • the changes to deps and such were scary, so to verify them working, you can use pkgcore to dump the result:
      pquery --raw --attr alldepends --attr required_use --attr iuse -v --max --repo ~/dev/gentoo =dev-libs/tree-sitter-bash-0.21.0
    • used pkgcheck as old binding user, and it didn't break - so consumers of the libraries should feel any difference.

To see new python bindings in work, install dev-python/tree-sitter-0.21.0 and dev-libs/tree-sitter-bash-0.21.0, and the run:

>>> import tree_sitter_bash
>>> from tree_sitter import Language, Parser
>>> LANG = Language(tree_sitter_bash.language(), "bash")
>>> parser = Parser()
>>> parser.set_language(LANG)
>>> parser.parse(b"echo gentoo").root_node.sexp()
'(program (command name: (command_name (word)) argument: (word)))'

Things to do:

  • Fix all existing ebuilds which redefine S=
  • Check new tree sitter works in emacs plugins (help needed here)
  • Run ebuild *.ebuild install on all tree-sitter grammar ebuilds, and verify they all build
    for f in $(grep -r . -e 'inherit.*tree-sitter-grammar' -l); do ebuild $f clean install clean || echo "failed $f" >> /tmp/fail.log; done

@gentoo-bot
Copy link

Pull Request assignment

Submitter: @arthurzam
Areas affected: ebuilds, eclasses
Packages affected: dev-libs/tree-sitter, dev-libs/tree-sitter-bash, dev-python/tree-sitter, dev-util/tree-sitter-cli

dev-libs/tree-sitter: @thrifty-txt, @sarnex, @msva, @MatthewGentoo
dev-libs/tree-sitter-bash: @MatthewGentoo, @sarnex, @gentoo/pkgcore
dev-python/tree-sitter: @gentoo/python
dev-util/tree-sitter-cli: @MatthewGentoo, @msva, @sarnex, @thrifty-txt

Linked bugs

No bugs to link found. If your pull request references any of the Gentoo bug reports, please add appropriate GLEP 66 tags to the commit message and request reassignment.

If you do not receive any reply to this pull request, please open or link a bug to attract the attention of maintainers.

Missing GCO sign-off

Please read the terms of Gentoo Certificate of Origin and acknowledge them by adding a sign-off to all your commits.


In order to force reassignment and/or bug reference scan, please append [please reassign] to the pull request title.

Docs: Code of ConductCopyright policy (expl.) ● DevmanualGitHub PRsProxy-maint guide

@gentoo-bot gentoo-bot added assigned PR successfully assigned to the package maintainer(s). no bug found No Bug/Closes found in the commits. no signoff One or more commits do not indicate GCO sign-off. labels Mar 14, 2024
@gentoo-repo-qa-bot
Copy link
Collaborator

Pull request CI report

Report generated at: 2024-03-14 14:58 UTC
Newest commit scanned: 69ebfe7
Status: ✅ good

There are existing issues already. Please look into the report to make sure none of them affect the packages in question:
https://qa-reports.gentoo.org/output/gentoo-ci/6e62dd89db/output.html

Copy link
Member

@thesamesam thesamesam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK (besides the existing comments). Thanks for handling it.

@gentoo-repo-qa-bot
Copy link
Collaborator

Pull request CI report

Report generated at: 2024-03-18 07:08 UTC
Newest commit scanned: ac70e45
Status: ✅ good

There are existing issues already. Please look into the report to make sure none of them affect the packages in question:
https://qa-reports.gentoo.org/output/gentoo-ci/51c2136c0d/output.html

@arthurzam arthurzam force-pushed the tree-sitter branch 3 times, most recently from a013014 to c123bae Compare March 18, 2024 21:05
@gentoo-repo-qa-bot
Copy link
Collaborator

Pull request CI report

Report generated at: 2024-03-18 21:13 UTC
Newest commit scanned: a013014
Status: ✅ good

There are existing issues already. Please look into the report to make sure none of them affect the packages in question:
https://qa-reports.gentoo.org/output/gentoo-ci/b8bd341521/output.html

@gentoo-repo-qa-bot
Copy link
Collaborator

Pull request CI report

Report generated at: 2024-03-18 21:42 UTC
Newest commit scanned: c123bae
Status: ✅ good

There are existing issues already. Please look into the report to make sure none of them affect the packages in question:
https://qa-reports.gentoo.org/output/gentoo-ci/28caad435d/output.html

@arthurzam arthurzam requested a review from mgorny March 22, 2024 16:04
Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
The build system for tree-sitters now generates a much better
Makefile we can use to build the parser and grammar into a good C
library.
This also matches the build procedure used by upstream, making our
reports easier for them to debug (we hit this issue in an old bug
report on memory leak with tree-sitter-bash).

Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
New tree-sitter cli generated bindings and code around grammars and
parsers now support bulding a python wheel which supply much better
API and library for consumers in python bindings.

Currently I've added only python as a binding languages, even though
rust, swift, and go are also available. We should add them when we
see a request for them. Python will be needed for pkgcheck.

When we opt in into python bindings, we call the matching distutils
phase functions when `use python` is true.

Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
Signed-off-by: Arthur Zamarin <arthurzam@gentoo.org>
@gentoo-repo-qa-bot
Copy link
Collaborator

Pull request CI report

Report generated at: 2024-03-22 18:03 UTC
Newest commit scanned: 2741e06
Status: ✅ good

There are existing issues already. Please look into the report to make sure none of them affect the packages in question:
https://qa-reports.gentoo.org/output/gentoo-ci/7c58fcfe18/output.html

@arthurzam arthurzam deleted the tree-sitter branch March 23, 2024 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
assigned PR successfully assigned to the package maintainer(s). no bug found No Bug/Closes found in the commits. no signoff One or more commits do not indicate GCO sign-off.
Projects
None yet
6 participants