Skip to content

Commit

Permalink
add optional special handling for hex in code tokenizer (#5200)
Browse files Browse the repository at this point in the history 
* better support for hex in code tokenizer

* make hex tokenization stricter

it now matches only even-lenght sequences between delimiters

* Apply suggestions from code review

---------

Co-authored-by: Paul Masurel <paul@quickwit.io>
  • Loading branch information
@trinity-1686a @fulmicoton
trinity-1686a and fulmicoton committed Jul 16, 2024
1 parent 3c7d0e0 commit 4e7dd61
Show file tree
Hide file tree
Showing 2 changed files with 466 additions and 37 deletions.
Loading

0 comments on commit 4e7dd61

Please sign in to comment.