Browse files

add note about regexes

  • Loading branch information...
1 parent 02f8b19 commit 12d34eb79f55ebbd7a7aa95fa5b1fe390866bfca @Floby committed Apr 10, 2011
Showing with 4 additions and 0 deletions.
  1. +4 −0
@@ -43,6 +43,10 @@ The tokenizer tries to find the longest string matching one or more rules.
When several rules match the same string, priority is given to the rule
which was added first. (this may change)
+Please note that your regular expressions should use ^ and $ in order
+to test the whole string. If these are not used, you rule will match _every_
+string that contains what you specified, this could be the whole file!
## To do
* a lot of optimisation
* being able to share rules across several tokenizer

0 comments on commit 12d34eb

Please sign in to comment.