-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
char class directives delete existing classes #986
Comments
There are two issues here really: macros before char class directive, and multiple char class directives in a spec. The latter is easy to check against, and for the former, instead of trying to enforce directives before macros in the lexer spec, it is probably better to make the implementation robust against this situation. |
lsf37
added a commit
that referenced
this issue
Dec 31, 2022
- move char class creation to after the entire spec has been parsed, so we definitively know the input char set - to do that, factor out char class creation from scanner and parser into own pass - to do that, make pre-defined classes and unicode property classes parts of the syntax tree as opposed to inline unfolded char sets - to do that, factor out regexp char class normalisation into own pass (separate from macro expansion), so that this normalisation can be done after pre-defined classes are converted to char sets - add error message for declaring input char set twice - make CharClasses.init() private (use constructor instead) Fixes #986
lsf37
added a commit
that referenced
this issue
Dec 31, 2022
- move char class creation to after the entire spec has been parsed, so we definitively know the input char set - to do that, factor out char class creation from scanner and parser into own pass - to do that, make pre-defined classes and unicode property classes parts of the syntax tree as opposed to inline unfolded char sets - to do that, factor out regexp char class normalisation into own pass (separate from macro expansion), so that this normalisation can be done after pre-defined classes are converted to char sets - add error message for declaring input char set twice - make CharClasses.init() private (use constructor instead) Fixes #986
lsf37
added a commit
that referenced
this issue
Dec 31, 2022
- move char class creation to after the entire spec has been parsed, so we definitively know the input char set - to do that, factor out char class creation from scanner and parser into own pass - to do that, make pre-defined classes and unicode property classes parts of the syntax tree as opposed to inline unfolded char sets - to do that, factor out regexp char class normalisation into own pass (separate from macro expansion), so that this normalisation can be done after pre-defined classes are converted to char sets - add error message for declaring input char set twice - make CharClasses.init() private (use constructor instead) Fixes #986
lsf37
added a commit
that referenced
this issue
Dec 31, 2022
- move char class creation to after the entire spec has been parsed, so we definitively know the input char set - to do that, factor out char class creation from scanner and parser into own pass - to do that, make pre-defined classes and unicode property classes parts of the syntax tree as opposed to inline unfolded char sets - to do that, factor out regexp char class normalisation into own pass (separate from macro expansion), so that this normalisation can be done after pre-defined classes are converted to char sets - add error message for declaring input char set twice - make CharClasses.init() private (use constructor instead) Fixes #986
lsf37
added a commit
that referenced
this issue
Jan 1, 2023
- move char class creation to after the entire spec has been parsed, so we definitively know the input char set - to do that, factor out char class creation from scanner and parser into own pass - to do that, make pre-defined classes and unicode property classes parts of the syntax tree as opposed to inline unfolded char sets - to do that, factor out regexp char class normalisation into own pass (separate from macro expansion), so that this normalisation can be done after pre-defined classes are converted to char sets - add error message for declaring input char set twice - make CharClasses.init() private (use constructor instead) Fixes #986
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There is an assumption in the char class implementation that the input char set is known before regular expressions are processed, but this assumption is not enforced by the syntax.
In particular, the char class directives (
%7bit
,%8bit
,%unicode
, etc) assume they are called only once and before any character set partitions have been constructed. They reset the partitions to one partition covering the (new) whole input char set.Example:
produces
which is wrong and will match any 2-character sequence.
The text was updated successfully, but these errors were encountered: