-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable non-greedy restriction on regexes #439
base: master
Are you sure you want to change the base?
Conversation
@@ -263,47 +262,42 @@ impl NFA { | |||
}, | |||
|
|||
HirKind::Repetition(ref r) => { | |||
if !r.greedy { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't look like you use this flag anywhere in the changes from this PR so I think that this just makes the code ignore the non-greedy option altogether (non-greedy regexes act like greedy ones)?
I don't know exactly how to implement non-greedy regexes for LALRPOP but using non-greedy at the end of a token may be a bit tricky to implement as I think it would need to generate code such that repetitions only consume data if the start of any of the defined tokens would not match after it. Just implementing it for using non-greedy strictly inside of a token might be simpler however.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I took a stab at this because I saw @nikomatsakis comment #209 (comment) in the linked issue.
In my case, I was doing r"'.*?[^']'"
to capture everything in between single quotes up to the next single quote that does not have a single quote before it and r"''"
to then grab an empty string without having to or operator and specify non capturing groups in the first regex.
In my case though,that didn't give me what I needed to parse what I needed, but non-greedy regexes give a decent bit of extra control.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be that you only disabled some validation then and the actual construction of of the actual token matcher (regex) is done elsewhere and this works fine. I guess if this code only checks overlap then the flag can be ignored here as greediness shouldn't affect overlap (I think). Could you add some tests using non-greedy tokens to check that this works as expected (with non-greedy used at the start, middle and end of a token respectively)?
Thanks for the PR, we're deferring this to get a short term patch out the door, we may consider it later. |
a3aea31
to
395e3de
Compare
I don't know if this is sufficient, but it compiles, passes local tests, and when compiled it helps my grammar pass tests that highlighted the case where I wanted the regex to be nongreedy.
Fixes #209