Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Browse files

add note about regexes

  • Loading branch information...
commit 12d34eb79f55ebbd7a7aa95fa5b1fe390866bfca 1 parent 02f8b19
@Floby authored
Showing with 4 additions and 0 deletions.
  1. +4 −0
@@ -43,6 +43,10 @@ The tokenizer tries to find the longest string matching one or more rules.
When several rules match the same string, priority is given to the rule
which was added first. (this may change)
+Please note that your regular expressions should use ^ and $ in order
+to test the whole string. If these are not used, you rule will match _every_
+string that contains what you specified, this could be the whole file!
## To do
* a lot of optimisation
* being able to share rules across several tokenizer

0 comments on commit 12d34eb

Please sign in to comment.
Something went wrong with that request. Please try again.