Token patterns with flags are broken

[oxc]

Since 0e6eb41, Tokens defined by a Pattern or RegEx object are converted to a Token defined by a pattern String.

This breaks use-cases where the pattern object was created with flags set
(e.g. token("^--".toRegex(RegexOption.MULTILINE)))

Currently, this can only be worked around by specifying the flags inside the pattern string
(e.g. token("(?m)^--"))

[h0tk3y]

The aim of that commit was supporting custom tokenizers that do not depend on Java regex implementation. I'll take a look into options how regex flags can be returned to work with the default tokenizers.

[oxc]

I understand that the goal of that change was not to break patterns, but if the Pattern/RegEx parameters are internally passed as Strings, there's not too much use for the overloaded Token constructors anymore, especially if they lose information in the process.

I think in this case I personally would have preferred if the overloaded versions taking a Pattern or a RegEx would have been removed completely.

[h0tk3y]

Fixed in 0.3.2. Now, if a Token is created with a Regex or a Pattern, it is saved as a Regex inside the Token, so the Tokenizer can use the regex as well. Default tokenizer uses the Regex if it is present, and creates it from the string pattern otherwise.

[oxc]

Thanks! Works great ;)