Make delimiters separate tokens from beginnings of '%' literals #15
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -339,7 +356,11 @@ module.exports = grammar({ | |||
| function stringBody (open, close, insert) { | |||
| var contents = [ /\\./, RegExp('[^\\\\\\' + close + ']') ]; | |||
| if (typeof insert !== 'undefined') contents.push(insert); | |||
| return seq(open, repeat(choice.apply(null, contents)), token(prec(PREC.LITERAL, close))); | |||
| return seq( | |||
| (typeof open === 'string') ? token(prec(PREC.LITERAL, open)) : open, | |||
There was a problem hiding this comment.
This ternary is a temporary hack to work around a tree-sitter bug when handling token(blank()).
| @@ -21,7 +21,7 @@ const PREC = { | |||
| LITERAL: 100, | |||
| }; | |||
|
|
|||
| const unbalancedDelimiters = '!@#$%^&*)]}>|\\=/+-~`\'",.?:;_'; | |||
| const unbalancedDelimiters = '!@#$%^&*)]}>|\\=/+-~`\'",.?:;_'.split(''); | |||
|
🙇 |
robrix
added a commit
that referenced
this pull request
Feb 22, 2016
Make delimiters separate tokens from beginnings of '%' literals
|
Nice one! Thank you! |
philipturnbull
added a commit
that referenced
this pull request
Jun 30, 2017
isalnum is only defined if the input is in the ASCII range. Nothing bad happens
on mac afaict but will cause an out-of-bounds read on Linux:
==11146== ERROR: libFuzzer: deadly signal
#0 0x649030 in fuzzer::Fuzzer::CrashCallback() /src/libfuzzer/FuzzerLoop.cpp:195:5
#1 0x648fd9 in fuzzer::Fuzzer::StaticCrashSignalCallback() /src/libfuzzer/FuzzerLoop.cpp:179:6
#2 0x7f118dd6638f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
#3 0x7f118d396d0d in isalnum (/lib/x86_64-linux-gnu/libc.so.6+0x2dd0d)
#4 0x42d44a in (anonymous namespace)::Scanner::scan_symbol_identifier(TSLexer*) /src/tree-sitter/tree-sitter-ruby/src/scanner.cc:250:12
#5 0x42be29 in (anonymous namespace)::Scanner::scan(TSLexer*, bool const*) /src/tree-sitter/tree-sitter-ruby/src/scanner.cc:727:18
#6 0x62eb83 in parser__lex /src/tree-sitter/src/runtime/parser.c:271:30
#7 0x62a05c in parser__advance /src/tree-sitter/src/runtime/parser.c:1097:21
#8 0x629789 in parser_parse /src/tree-sitter/src/runtime/parser.c:1298:9
#9 0x62617d in ts_document_parse_with_options /src/tree-sitter/src/runtime/document.c:136:16
#10 0x42b053 in LLVMFuzzerTestOneInput /src/tree-sitter/../fuzzer.cc:21:3
#11 0x64a164 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:460:13
#12 0x64a4be in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:399:3
#13 0x63c890 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:268:6
#14 0x64090b in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:683:9
#15 0x63c58c in main /src/libfuzzer/FuzzerMain.cpp:20:10
#16 0x7f118d38982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#17 0x405928 in _start (/out/ruby_fuzzer+0x405928)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This reduces the size of the parser object file from about 14mb to about 9mb. It does allow comments and whitespace between
%qand its delimiter, but I think it is ok in general to parse a superset of the language./cc @robrix