Updated the RegExp to catch Strings earlier in the tokenization process. #90

Merged
merged 1 commit into from Oct 2, 2012
@@ -24,7 +24,9 @@ def tokenize string
return string if @string_cache.key?(string)
case string
- when /^[A-Za-z_~]/
+ # Check for a String type, being careful not to get caught by hash keys, hex values, and
+ # special floats (e.g., -.inf).
+ when /^[^\d\.:-]?[A-Za-z_\s!@#\$%\^&\*\(\)\{\}\<\>\|\/\\~;=]+/
if string.length > 5
@string_cache[string] = true
return string