You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add Lexer::from_str as the safe instantiation method.
Refactor Lexer::from_slice to unsafe Lexer::from_slice_unchecked. It is
assumed the byte slice is UTF-8 characters, and if they are not a valid slice
of UTF-8 bytes, then the behavior is undefined.
Dynamically determine Token type instead of storing the Ty. This reduces the Token size to 2 words (16 octets on a 64-bit machine).
Removed
Remove offset field from Token. If required, the offset should be
calculated when tokens are consumed.
Remove scan and Scanner from the public API.
Remove const from Token::ty() method for possible future optimizations.
Remove public new methods from Token and related types.
Added
Add unsafe as_str_unchecked to various types in cases where the bytes are
guaranteed to be valid UTF-8.
Add as_bytes, to_str, as_str_unchecked, and into_inner to Token.