Add regex tokenizer

[mkleen]

This adds a regex tokenizer which tokenizes the text by using a regex pattern to split. This is my first attempt and works for my usecase, but it's not ideal from a code and a configuration perspective.

Closes #1670

[Gearme]

Wow, snap! I just authored #1759 to do much the same, albeit with capture group support.
Didn't find a better solution for the cloned Regex either though ;)

[mkleen]

Wow, snap! I just authored #1759 to do much the same, albeit with capture group support. Didn't find a better solution for the cloned Regex either though ;)

Haha, great. If you have capture group support then you have probably the better version.

[codecov-commenter]

Codecov Report

Merging #1759 (0c99e75) into main (b78dc5e) will increase coverage by 0.01%.
The diff coverage is 92.85%.

@@            Coverage Diff             @@
##             main    #1759      +/-   ##
==========================================
+ Coverage   94.13%   94.14%   +0.01%     
==========================================
  Files         267      269       +2     
  Lines       50900    50985      +85     
==========================================
+ Hits        47915    48002      +87     
+ Misses       2985     2983       -2     

Impacted Files	Coverage Δ
src/tokenizer/mod.rs	95.61% <ø> (ø)
src/tokenizer/regex_tokenizer.rs	92.85% <92.85%> (ø)
src/lib.rs	96.04% <0.00%> (-0.44%)	⬇️
src/fastfield/readers.rs	89.47% <0.00%> (-0.36%)	⬇️
src/fastfield/mod.rs	99.73% <0.00%> (-0.01%)	⬇️
src/query/mod.rs	100.00% <0.00%> (ø)
src/query/range_query.rs
src/query/range_query_ip_fastfield.rs
src/query/range_query/range_query.rs	91.16% <0.00%> (ø)
src/query/range_query/range_query_ip_fastfield.rs	97.00% <0.00%> (ø)
... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[adamreichold]

I think this would work. Some nits on the formatting, one concern about the intended meaning of regular expression anchors.

I would suggest doing capture group support as a follow-up.

I think the performance question of whether to clone Regex or Arc<Regex> can only be solved via benchmarks. I would suggest keeping it as it is for now if only for simplicity until such benchmarks are written.

[mkleen]

@adamreichold Thank you for your feedback and the review. I like your idea of first merging this simpler version and then going ahead and adding the group capture support afterwards. I can wrap this up pretty quickly. @Gearme would that be OK for you as well? We could make the follow-up based on your implementation. I would also like to try out a benchmark to address #1759 (review), but also as a follow-up.

[mkleen]

Ups, @PSeitz removing you from review was not intentionally.

[mkleen]

Sorry for requesting multiple reviews. I did not realize I can only request one review at the time.

[mkleen]

I just discovered there is a off-by-one on the offsets of the second token. I will address that.

[mkleen]

I just discovered there is a off-by-one on the offsets of the second token. I will address that.

I think this is fine, wrong alarm.