fix(backend/sdoc_source_code): let lark scan CR to NEWLINE, not to NODE_STRING_VALUE #2575
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
haxtibal
force-pushed
the
tdmg/fix_lark_newline
branch
from
dadf96a to
c3241e8
Compare
…DE_STRING_VALUE strictdoc#2554 added dedent logic, strictdoc-project#2555 added hash generation. that hash results for same input were different on Windows and Linux even if CR LF was normalized to LF. Further investigation revealed that dedent logic didn't work as expected on Windows, because CR LF got split half into the string token, and half into the newline token, which in turn confused the dedent logic. By trial it turned out that "/[^\n\r]+/x" (verbose regex mode, real newlines) is not the same as "/[^\\n\\r]+/" (normal regex mode, symbolized newlines). The former wrongly lets lark scan a \r into NODE_STRING_VALUE. Thus change to the latter pattern.
haxtibal
force-pushed
the
tdmg/fix_lark_newline
branch
from
c3241e8 to
066f5e2
Compare
haxtibal
commented
Nov 25, 2025
stanislaw
approved these changes
Nov 25, 2025
Collaborator
stanislaw
left a comment
stanislaw
left a comment
There was a problem hiding this comment.
Thus change to the latter pattern. Note: This fixes only the dedent issue. Hash generation still expects bit-wise identical input and tests would fail if expected hash was sampled on Linux but git converted input \n to \r\n on Windows. We need to re-add code = code.replace(b"\r\n", b"\n") if hash shall be agnostic to line endings.
We didn't remove the code.replace actually, it is still there. I will try to remove the gitattributes now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#2554 added dedent logic, #2555 added hash generation. Tests revealed that hash results for same input were different on Windows and Linux even if CR LF was normalized to LF.
Further investigation revealed that dedent logic didn't work as expected on Windows, because CR LF got split half into the string token, and half into the newline token, which in turn confused the dedent logic. By trial it turned out that
"/[^\n\r]+/x"(verbose regex mode, normal string => real newlines) is not the same as"/[^\\n\\r]+/"(normal regex mode, normal string but escaped backslash => symbolized newlines). The former wrongly lets lark scan a \r into NODE_STRING_VALUE.Thus change to the latter pattern. Note: This fixes only the dedent issue. Hash generation still expects bit-wise identical input and tests would fail if expected hash was sampled on Linux but git converted input
\nto\r\non Windows. We need to re-addcode = code.replace(b"\r\n", b"\n")if hash shall be agnostic to line endings.