Skip to content

v0.7.0

Choose a tag to compare

@bluk bluk released this 30 Oct 18:47
· 131 commits to trunk since this release
v0.7.0
01519db

Changed

  • Add Lexer::from_str as the safe instantiation method.
  • Refactor Lexer::from_slice to unsafe Lexer::from_slice_unchecked. It is
    assumed the byte slice is UTF-8 characters, and if they are not a valid slice
    of UTF-8 bytes, then the behavior is undefined.
  • Dynamically determine Token type instead of storing the Ty. This reduces the
    Token size to 2 words (16 octets on a 64-bit machine).

Removed

  • Remove offset field from Token. If required, the offset should be
    calculated when tokens are consumed.
  • Remove scan and Scanner from the public API.
  • Remove const from Token::ty() method for possible future optimizations.
  • Remove public new methods from Token and related types.

Added

  • Add unsafe as_str_unchecked to various types in cases where the bytes are
    guaranteed to be valid UTF-8.
  • Add as_bytes, to_str, as_str_unchecked, and into_inner to Token.