New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix difference in CRLF line ending handling between tree-sitter and text-buffer #16738

merged 1 commit into from Feb 14, 2018


None yet
1 participant

maxbrunsfeld commented Feb 14, 2018

Fixes #16642
Fixes #16643

Tree-sitter allows tokens to end between CR and LF characters. For exampe, in JavaScript, line comment tokens always end at a LF character; if the file uses CRLF line endings, the comment token will include the the preceding CR character.

Text-buffer on the other hand, does not consider the position between a CR and and LF to be a valid position. It does not let you read or write changes between those characters.

To compensate for this difference in position semantics, I've had to add a special case to the TreeSitterTextBufferInput for the situation where the Tree-sitter parser decides to seek to the position right between a CR and an LF.

@maxbrunsfeld maxbrunsfeld merged commit 2ef0bc5 into master Feb 14, 2018

3 checks passed

ci/circleci Your tests passed on CircleCI!
continuous-integration/appveyor/pr AppVeyor build succeeded
continuous-integration/travis-ci/pr The Travis CI build passed

@maxbrunsfeld maxbrunsfeld deleted the mb-tree-sitter-crlf-handling branch Feb 14, 2018

maxbrunsfeld added a commit that referenced this pull request Feb 14, 2018

Merge pull request #16738 from atom/mb-tree-sitter-crlf-handling
Fix difference in CRLF line ending handling between tree-sitter and text-buffer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment