Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shim tokens in TokenizedLines returned from TreeSitterLanguageMode #19538

merged 2 commits into from Jun 14, 2019


Copy link

commented Jun 14, 2019

Fixes #18973

馃崘'd with @as-cii

Package authors are using the private tokenizedLineForRow method on language modes to get syntactic information about lines. Even though this method is private, @maxbrunsfeld provided a shim implementation in TreeSitterLanguageMode that enabled usage of some basic properties on returned TokenizedLine object, but this did not include the ability to read the tokens field. This is causing exceptions for at least one popular package, atom-aligner.

In this PR, @as-cii and I have enhanced the TokenizedLines returned by TreeSitterLanguageMode.prototype.tokenizedLineForRow to include a valid tokens array. We construct these tokens eagerly via the highlight iterator, which could impose a cost for those that call this method and don't use tokens, but making the method lazy would have been extremely complex due to the potential for the state of the buffer and the syntax tree to change after the TokenizedLine is returned.

Shim `tokens` in tokenized lines returned from `TreeSitterLanguageMode`
Co-Authored-By: Nathan Sobo <>

@nathansobo nathansobo changed the title Shim `tokens` in `TokenizedLine`s returned from `TreeSitterLanguageMode` Shim tokens in TokenizedLines returned from TreeSitterLanguageMode Jun 14, 2019


This comment has been minimized.

Copy link

commented Jun 14, 2019


@nathansobo nathansobo merged commit 2ea1f35 into master Jun 14, 2019

1 check passed

Atom Pull Requests #20190614.12 succeeded

@nathansobo nathansobo deleted the as-ns/tokenized-line-tokens-shim branch Jun 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
2 participants
You can鈥檛 perform that action at this time.