fix(lexer): disable reserved-word recognition after assignment words

[mpecan]

Closes #37

Summary

Disables bash reserved-word recognition after a simple command has consumed at least one AssignmentWord, matching bash behavior.

Root cause

After a simple command has consumed an AssignmentWord, bash stops recognizing reserved words in that same simple command even though it's still at command-word position for assignment-detection purposes. Rable incorrectly kept command_start=true for reserved-word purposes too, so the lexer emitted For / Do / Then reserved tokens where plain Word is expected.

Bash verification:

bash -n -c 'foo=bar for'    -> OK (for is a plain word)
bash -n -c 'foo=bar do'     -> OK
bash -n -c 'foo=bar then'   -> OK

Implementation

Splits the LexerContext flag into two independent concepts:

• command_start — eligible to begin a new simple command or to accept an AssignmentWord (unchanged semantics).
• reserved_words_ok (new) — narrowly gates reserved-word classification in read_word_token. Cleared when an AssignmentWord is emitted; re-armed on command separators (;, ;;, ;;&, ;&, |, ||, |&, &, &&), newlines, and all parser-driven set_command_start() calls.

All sites that set command_start=true now also set reserved_words_ok=true; the only place they diverge is the AssignmentWord branch in read_word_token, which keeps command_start=true but clears reserved_words_ok.

Fixed cases (issue #37)

• reserved_word_as_word 1 — if foo= for $(pwd); then foo=$(pwd); fi
• reserved_word_as_word 3 — while arr[0]=$fo do o; do ...
• reserved_word_as_word 4 — case $x in a) x=$ then [1+2];; esac

Incidental bonus fix

• rbracket_outside_cond 3 — Cdeclare -n ref=t ]] arget. Reclassifying the post-assignment ]] as a plain Word (instead of the DoubleRightBracket reserved token) is a side effect of the same change and is documented inline in KNOWN_ORACLE_FAILURES.

Deferred cases

• reserved_word_as_word 2 — ((x\n> 0)\n). Root cause is a (( vs ( ( disambiguation bug in parse_arith_command: the raw-char scanner for )) never finds a match when the body contains a newline, and there is no fallback to nested subshells. Tracked as parser: (( ... )) should fall back to nested subshells when body is not a valid arithmetic expression #42.
• reserved_word_as_word 5 — if [ $"yes" = yes9 ][ $; then echo ok; fi. Same root cause as lexer: words containing unbalanced [...] absorb || / && / spaces #36 — read_bracket_subscript runs to EOF looking for ]. Being fixed as a side effect of lexer: ]] not tokenized as separate word outside [[ ... ]] #35.

Test plan

• New unit tests in src/lexer/tests.rs:
  • reserved_word_after_assignment_is_plain_word — locks in token classification for cases 1, 3, 4.
  • reserved_word_re_armed_after_separator — verifies reserved-word recognition comes back after ; and |.
• Existing assignment_before_command_keeps_command_start still passes unchanged.
• tests/oracle/bash_valid_divergences.tests cases 1, 3, 4 removed from KNOWN_ORACLE_FAILURES; inline comments on cases 2 and 5 point at their respective tracking issues.
• cargo fmt && cargo clippy --all-targets -- -D warnings && cargo test → 221 pass (166 unit + 1 tree-sitter + 54 integration).

🤖 Generated with Claude Code