Skip to content

Step 3a: CSS tokenizer — CSS Syntax Level 3 #67

@thomasnemer

Description

@thomasnemer

Parent: #22

Goal

Implement the CSS tokenizer per CSS Syntax Module Level 3. Iterator-based, yields CssToken values.

File Changes

  • crates/ie-css/src/tokenizer.rs — new file
  • crates/ie-css/src/lib.rs — update modules
  • crates/ie-css/Cargo.toml — if needed

Implementation

  • CssToken enum: Ident, Function, AtKeyword, Hash, String, Url, Delim, Number, Percentage, Dimension, Whitespace, Colon, Semicolon, Comma, brackets, Cdo, Cdc, Eof
  • Handle escape sequences, string tokens, URL tokens, comments
  • Numbers: integer vs number distinction
  • Iterator-based

Tests

  • Simple rule tokens, strings with escapes, URL tokens, dimensions, percentages, functions
  • ~10 unit tests

Acceptance Criteria

  • cargo test -p ie-css passes
  • Clippy clean

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions