Permalink
Browse files

bugfix in tokenization: newlines in the middle of words

  • Loading branch information...
wmorgan committed Apr 9, 2012
1 parent 3de9639 commit f6a49d017b26d537ac7b6f72720b60fabf966142
Showing with 1 addition and 1 deletion.
  1. +1 −1 tokenizer.lex
View
@@ -16,7 +16,7 @@
%option extra-type="struct lexinfo*"
FIRSTWORDCHAR [^[:blank:][:punct:]<>\r\n]
-INNERWORDCHAR [^[:blank:]<>]
+INNERWORDCHAR [^[:blank:]<>\r\n]
LASTWORDCHAR [^[:blank:][:punct:]<>\r\n]
%%

0 comments on commit f6a49d0

Please sign in to comment.