Skip to content

Commit

Permalink
add note about regexes
Browse files Browse the repository at this point in the history
  • Loading branch information
Floby committed Apr 10, 2011
1 parent 02f8b19 commit 12d34eb
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ The tokenizer tries to find the longest string matching one or more rules.
When several rules match the same string, priority is given to the rule
which was added first. (this may change)

Please note that your regular expressions should use ^ and $ in order
to test the whole string. If these are not used, you rule will match _every_
string that contains what you specified, this could be the whole file!

## To do
* a lot of optimisation
* being able to share rules across several tokenizer
Expand Down

0 comments on commit 12d34eb

Please sign in to comment.