fix(command-parser)!: drop the char-level tokenizer in favor of an AST-only path

[fohte]

Purpose

• Commit commands that contain a HEREDOC inside a command substitution (the kind Claude Code's /commit skill produces) were rejected with command parse error: unclosed quote

Reproduction steps

Piping the following command into runok check --input-format claude-code-hook is rejected with command parse error: unclosed quote, even though the existing git [-C *] commit -m * rule should allow it.

git add path && git commit -m "$(cat <<'EOF'
subject

body line 1 with 'apostrophes' inside
EOF
)"

Approach

• Drop the char-level fallback tokenizer and route all bash parsing through tree-sitter-bash
• Resolve quoting and escapes per AST node so a HEREDOC body never re-enters a tokenizer
• Treat HEREDOCs whose delimiter is quoted (<<'EOF', <<"EOF", <<\EOF) as literal — skip the $(cmd) extraction inside their bodies, matching bash semantics

Design decisions

Fallback for inputs tree-sitter cannot parse

Decision	Design	Pros	Cons
Chosen	Fall back to shlex::split only when tree-sitter returns has_error	Real-world inputs that bash itself rejects (e.g. graphql query bodies as flag values) keep working; HEREDOCs stay on the AST path	Need to distinguish has_error from "not a single command" so pipelines and && lists are not silently flattened by the fallback
Rejected	Drop the fallback entirely and return SyntaxError	Simplest possible spec	Breaks practical inputs already covered by tests, e.g. gh api graphql -f query=mutation{...}
Rejected	Keep the char-level tokenizer and just add HEREDOC detection on top	Preserves existing behaviour	Leaves the dual implementation in place, so the original bug (tokenizer and AST disagreeing) reappears for other inputs

Breaking changes

• runok no longer extracts $(cmd) from inside <<'EOF' / <<"EOF" / <<\EOF HEREDOC bodies. Bash treats these bodies as literal, so any rule that relied on matching a substitution inside one needs to be rewritten to target the command outside the HEREDOC.
• Library consumers only:
  • CommandParseError::UnclosedQuote is removed and folded into SyntaxError.
  • A bare assignment (FOO=bar alone) and a trailing backslash (echo \) now return SyntaxError.

[codecov]

Codecov Report

❌ Patch coverage is 88.53211% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.04%. Comparing base (4d0467b) to head (223d798).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/rules/command_parser.rs	88.26%	25 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #330      +/-   ##
==========================================
- Coverage   89.12%   89.04%   -0.08%     
==========================================
  Files          53       53              
  Lines       11771    11871     +100     
==========================================
+ Hits        10491    10571      +80     
- Misses       1280     1300      +20     

Flag	Coverage Δ
Linux	88.85% <88.53%> (-0.09%)	⬇️
macOS	90.26% <88.53%> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gemini-code-assist]

Code Review

This pull request migrates the command parser to a tree-sitter-bash AST-based implementation, improving tokenization accuracy and shell-quoting resolution. It introduces breaking changes by removing the CommandParseError::UnclosedQuote variant and changing how bare assignments and trailing backslashes are handled, while also fixing bugs related to quoted command names and HEREDOC processing. Review feedback recommends enhancing the HEREDOC delimiter check to detect quotes at any position within the string and simplifying the node retrieval logic for command names.