feat: define comprehensive token types for commit message compiler

[tgenericx]

PR Type

Enhancement

---

Description

• Replaced empty token enum with comprehensive Token variants for conventional commits

• Added Type, Scope, Breaking, Description, Body, Footer, Newline, and Eof token types

• Implemented utility methods: type_name(), value(), is_breaking(), is_newline(), is_eof()

• Added Display trait implementation and extensive test coverage for all token types

---

Diagram Walkthrough

flowchart LR
  A["Empty Token Enum"] -->|"Replace with variants"| B["Token Enum with 8 variants"]
  B -->|"Add methods"| C["Utility Methods"]
  B -->|"Add trait"| D["Display Implementation"]
  B -->|"Add tests"| E["13 Test Cases"]

Loading

File Walkthrough

Relevant files

Enhancement

token.rs Implement comprehensive Token enum with methods and tests src/compiler/token.rs Replaced empty TokenKind enum and Token struct with a comprehensive Token enum containing 8 variants representing different parts of conventional commit messages Implemented helper methods: type_name() for human-readable token type names, value() to extract string content, and three boolean predicates (is_breaking(), is_newline(), is_eof()) Added Display trait implementation with truncation for long descriptions and body text Added 13 comprehensive unit tests covering token instantiation, type names, value extraction, predicates, display formatting, cloning, and equality comparisons +249/-4

---

[qodo-code-review]

PR Compliance Guide 🔍

Below is a summary of compliance checks for this PR:

Security Compliance
🟢	No security concerns identified No security vulnerabilities detected by AI analysis. Human verification advised for critical code.
Ticket Compliance
🟢	🎫 #72 🟢 Define tokens that cover all conventional commit elements: type, scope, breaking indicator, description, body, and footers. Ensure tokens are distinct and self-descriptive (avoid ambiguous naming). Support optional conventional commit fields (scope, body, footer) via token structure. Keep tokens structural only (independent of semantic validation rules). Provide a public Token enum with variants like Type(String), Scope(String), Breaking, Description(String), Body(String), Footer(String). Ensure tokens are publicly usable by both lexer and parser. Add minimal documentation for each token type. Add a dummy test confirming the enum compiles and can be instantiated.
Codebase Duplication Compliance
⚪	Codebase context is not defined Follow the guide to enable codebase context checks.
Custom Compliance
🟢	Generic: Comprehensive Audit Trails Objective: To create a detailed and reliable record of critical system actions for security analysis and compliance. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Meaningful Naming and Self-Documenting Code Objective: Ensure all identifiers clearly express their purpose and intent, making code self-documenting Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Error Handling Objective: To prevent the leakage of sensitive system information through error messages while providing sufficient detail for internal debugging. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Secure Logging Practices Objective: To ensure logs are useful for debugging and auditing without exposing sensitive information like PII, PHI, or cardholder data. Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
	Generic: Security-First Input Validation and Data Handling Objective: Ensure all data inputs are validated, sanitized, and handled securely to prevent vulnerabilities Status: Passed Learn more about managing compliance generic rules or creating your own custom rules
🔴	Generic: Robust Error Handling and Edge Case Management Objective: Ensure comprehensive error handling that provides meaningful context and graceful degradation Status: Unicode slice panic: The Display implementation truncates strings with &s[..30], which can panic on non-ASCII/UTF-8 boundary inputs instead of handling this edge case safely. Referred Code let preview = if s.len() > 30 { format!("{}...", &s[..30]) } else { s.clone() }; write!(f, "Description({})", preview) } Token::Body(s) => { let preview = if s.len() > 30 { format!("{}...", &s[..30]) } else { s.clone() }; write!(f, "Body({})", preview) } Learn more about managing compliance generic rules or creating your own custom rules
Update

Compliance status legend

🟢 - Fully Compliant
🟡 - Partial Compliant
🔴 - Not Compliant
⚪ - Requires Further Human Verification
🏷️ - Compliance label

[qodo-code-review]

CI Feedback 🧐

A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

Action: Linting

Failed stage: Run Clippy [❌]

Failed test name: ""

Failure summary: The GitHub Action failed during cargo clippy -- -D warnings because Clippy/lints were treated as errors (-D warnings), and multiple items in the compiler module are flagged as dead code (-D dead-code implied by -D warnings). Errors reported include unused/never-constructed items: - src/compiler/mod.rs:9:8: function compile is never used - src/compiler/ast.rs:2:12: struct CommitAst is never constructed - src/compiler/error.rs:2:10: enum CompileError is never used - src/compiler/lexer.rs:1:12: struct Lexer is never constructed - src/compiler/parser.rs:1:12: struct Parser is never constructed - src/compiler/token.rs:10:10 and src/compiler/token.rs:49..87: enum Token and methods type_name, value, is_breaking, is_newline, is_eof are never used As a result, the crate failed to compile (could not compile commando ... due to 7 previous errors) and the step exited with code 101.

Relevant error logs: 1: ##[group]Runner Image Provisioner 2: Hosted Compute Agent ... 158: env: 159: CARGO_TERM_COLOR: always 160: targets: 161: components: clippy 162: ##[endgroup] 163: ##[group]Run : set $CARGO_HOME 164: �[36;1m: set $CARGO_HOME�[0m 165: �[36;1mecho CARGO_HOME=${CARGO_HOME:-"$HOME/.cargo"} >> $GITHUB_ENV�[0m 166: shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 167: env: 168: CARGO_TERM_COLOR: always 169: ##[endgroup] 170: ##[group]Run : install rustup if needed 171: �[36;1m: install rustup if needed�[0m 172: �[36;1mif ! command -v rustup &>/dev/null; then�[0m 173: �[36;1m curl --proto '=https' --tlsv1.2 --retry 10 --retry-connrefused --location --silent --show-error --fail https://sh.rustup.rs | sh -s -- --default-toolchain none -y�[0m 174: �[36;1m echo "$CARGO_HOME/bin" >> $GITHUB_PATH�[0m ... 237: �[36;1mif [ -z "${CARGO_REGISTRIES_CRATES_IO_PROTOCOL+set}" -o -f "/home/runner/work/_temp"/.implicit_cargo_registries_crates_io_protocol ]; then�[0m 238: �[36;1m if rustc +stable --version --verbose | grep -q '^release: 1\.6[89]\.'; then�[0m 239: �[36;1m touch "/home/runner/work/_temp"/.implicit_cargo_registries_crates_io_protocol || true�[0m 240: �[36;1m echo CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse >> $GITHUB_ENV�[0m 241: �[36;1m elif rustc +stable --version --verbose | grep -q '^release: 1\.6[67]\.'; then�[0m 242: �[36;1m touch "/home/runner/work/_temp"/.implicit_cargo_registries_crates_io_protocol || true�[0m 243: �[36;1m echo CARGO_REGISTRIES_CRATES_IO_PROTOCOL=git >> $GITHUB_ENV�[0m 244: �[36;1m fi�[0m 245: �[36;1mfi�[0m 246: shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 247: env: 248: CARGO_TERM_COLOR: always 249: CARGO_HOME: /home/runner/.cargo 250: CARGO_INCREMENTAL: 0 251: ##[endgroup] 252: ##[group]Run : work around spurious network errors in curl 8.0 253: �[36;1m: work around spurious network errors in curl 8.0�[0m 254: �[36;1m# https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/timeout.20investigation�[0m ... 273: binary: rustc 274: commit-hash: 254b59607d4417e9dffbc307138ae5c86280fe4c 275: commit-date: 2026-01-19 276: host: x86_64-unknown-linux-gnu 277: release: 1.93.0 278: LLVM version: 21.1.8 279: ##[group]Run cargo clippy -- -D warnings 280: �[36;1mcargo clippy -- -D warnings�[0m 281: shell: /usr/bin/bash -e {0} 282: env: 283: CARGO_TERM_COLOR: always 284: CARGO_HOME: /home/runner/.cargo 285: CARGO_INCREMENTAL: 0 286: ##[endgroup] 287: �[1m�[92m Checking�[0m commando v0.3.0 (/home/runner/work/commando/commando) 288: �[1m�[91merror�[0m�[1m: function `compile` is never used�[0m 289: �[1m�[94m--> �[0msrc/compiler/mod.rs:9:8 290: �[1m�[94m|�[0m 291: �[1m�[94m9�[0m �[1m�[94m|�[0m pub fn compile(_input: &str) -> Result<crate::commit_message::CommitMessage, CompileError> { 292: �[1m�[94m|�[0m �[1m�[91m^^^^^^^�[0m 293: �[1m�[94m|�[0m 294: �[1m�[94m= �[0m�[1mnote�[0m: `-D dead-code` implied by `-D warnings` 295: �[1m�[94m= �[0m�[1mhelp�[0m: to override `-D warnings` add `#[allow(dead_code)]` 296: �[1m�[91merror�[0m�[1m: struct `CommitAst` is never constructed�[0m 297: �[1m�[94m--> �[0msrc/compiler/ast.rs:2:12 298: �[1m�[94m|�[0m 299: �[1m�[94m2�[0m �[1m�[94m|�[0m pub struct CommitAst {} 300: �[1m�[94m|�[0m �[1m�[91m^^^^^^^^^�[0m 301: �[1m�[91merror�[0m�[1m: enum `CompileError` is never used�[0m 302: �[1m�[94m--> �[0msrc/compiler/error.rs:2:10 303: �[1m�[94m|�[0m 304: �[1m�[94m2�[0m �[1m�[94m|�[0m pub enum CompileError { 305: �[1m�[94m|�[0m �[1m�[91m^^^^^^^^^^^^�[0m 306: �[1m�[91merror�[0m�[1m: struct `Lexer` is never constructed�[0m 307: �[1m�[94m--> �[0msrc/compiler/lexer.rs:1:12 308: �[1m�[94m|�[0m 309: �[1m�[94m1�[0m �[1m�[94m|�[0m pub struct Lexer {} 310: �[1m�[94m|�[0m �[1m�[91m^^^^^�[0m 311: �[1m�[91merror�[0m�[1m: struct `Parser` is never constructed�[0m 312: �[1m�[94m--> �[0msrc/compiler/parser.rs:1:12 313: �[1m�[94m|�[0m 314: �[1m�[94m1�[0m �[1m�[94m|�[0m pub struct Parser {} 315: �[1m�[94m|�[0m �[1m�[91m^^^^^^�[0m 316: �[1m�[91merror�[0m�[1m: enum `Token` is never used�[0m 317: �[1m�[94m--> �[0msrc/compiler/token.rs:10:10 318: �[1m�[94m|�[0m 319: �[1m�[94m10�[0m �[1m�[94m|�[0m pub enum Token { 320: �[1m�[94m|�[0m �[1m�[91m^^^^^�[0m 321: �[1m�[91merror�[0m�[1m: methods `type_name`, `value`, `is_breaking`, `is_newline`, and `is_eof` are never used�[0m 322: �[1m�[94m--> �[0msrc/compiler/token.rs:49:12 ... 326: �[1m�[94m48�[0m �[1m�[94m|�[0m /// Returns a human-readable description of the token type 327: �[1m�[94m49�[0m �[1m�[94m|�[0m pub fn type_name(&self) -> &'static str { 328: �[1m�[94m|�[0m �[1m�[91m^^^^^^^^^�[0m 329: �[1m�[94m...�[0m 330: �[1m�[94m64�[0m �[1m�[94m|�[0m pub fn value(&self) -> Option<&str> { 331: �[1m�[94m|�[0m �[1m�[91m^^^^^�[0m 332: �[1m�[94m...�[0m 333: �[1m�[94m77�[0m �[1m�[94m|�[0m pub fn is_breaking(&self) -> bool { 334: �[1m�[94m|�[0m �[1m�[91m^^^^^^^^^^^�[0m 335: �[1m�[94m...�[0m 336: �[1m�[94m82�[0m �[1m�[94m|�[0m pub fn is_newline(&self) -> bool { 337: �[1m�[94m|�[0m �[1m�[91m^^^^^^^^^^�[0m 338: �[1m�[94m...�[0m 339: �[1m�[94m87�[0m �[1m�[94m|�[0m pub fn is_eof(&self) -> bool { 340: �[1m�[94m|�[0m �[1m�[91m^^^^^^�[0m 341: �[1m�[91merror�[0m: could not compile `commando` (bin "commando") due to 7 previous errors 342: ##[error]Process completed with exit code 101. 343: Post job cleanup.

[qodo-code-review]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	✅ Use Unicode-safe truncation Suggestion Impact: Updated Token::Description preview generation to collect the first 30 Unicode characters and append "..." when longer, replacing the unsafe &s[..30] slicing. code diff: Token::Description(s) => { - let preview = if s.len() > 30 { - format!("{}...", &s[..30]) - } else { - s.clone() - }; + let mut preview: String = s.chars().take(30).collect(); + if s.chars().count() > 30 { + preview.push_str("..."); + } write!(f, "Description({})", preview) Replace the unsafe byte-slice truncation &s[..30] with a Unicode-safe method like s.chars().take(30) to prevent potential panics. src/compiler/token.rs [99-106] Token::Description(s) => { - let preview = if s.len() > 30 { - format!("{}...", &s[..30]) - } else { - s.clone() - }; + let mut preview: String = s.chars().take(30).collect(); + if s.chars().count() > 30 { + preview.push_str("..."); + } write!(f, "Description({})", preview) } [Suggestion processed] Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical issue where slicing a string by byte index (&s[..30]) can cause a panic with multi-byte UTF-8 characters, and it proposes a safe alternative.	High
General	✅ Remove extraneous inline comment Suggestion Impact: The commit removed the inline `// ADD THIS LINE` comment after the `Token::Colon` formatting match arm. code diff: - Token::Colon => write!(f, "Colon"), // ADD THIS LINE + Token::Colon => write!(f, "Colon"), Remove the extraneous // ADD THIS LINE comment. src/compiler/token.rs [97] -Token::Colon => write!(f, "Colon"), // ADD THIS LINE +Token::Colon => write!(f, "Colon"), [Suggestion processed] Suggestion importance[1-10]: 3 __ Why: This is a minor but valid code cleanup suggestion that removes a confusing and unnecessary comment left over from development.	Low
Update