Skip to content

Documented headless check exits 0 even on Lua errors — false green signal for agents #141

Description

@ooloth

Why

The verification command documented in CLAUDE.md gives agents incorrect confidence that their Lua changes are valid, because Neovim always exits with code 0 regardless of startup errors.

Current state

CLAUDE.md documents nvim --headless -c "qall" as a way to surface Lua errors on startup. In practice, Neovim exits 0 whether or not startup errors occurred; errors appear only on stderr, and the process exit code is always success. An agent that introduces a broken require() path or a Lua syntax error runs this command, sees exit code 0, and reports the change as verified — leaving a broken config in the repo with no signal.

Ideal state

  • CLAUDE.md includes a note explaining that nvim --headless -c "qall" exits 0 regardless of Lua errors and must not be treated as a pass/fail gate
  • CLAUDE.md documents a command that exits nonzero on Lua syntax errors — e.g., luac -p <file> as a per-file pre-flight check — and explains its scope (syntax only, not runtime require resolution)
  • Agents reading CLAUDE.md understand exactly what can and cannot be verified without an interactive Neovim session

Out of scope

  • Adding a full CI pipeline or automated test harness (tracked separately)

Starting points

  • CLAUDE.md — the "Check commands" and "Exercising real code paths" sections documenting the headless invocation
  • .stylua.toml — confirms stylua scope is formatting only, not validity

QA plan

  1. Read CLAUDE.md check commands section — expect a note explicitly stating that the headless command exits 0 on Lua errors
  2. Run luac -p lua/config/plugins/editing.lua — expect exit code 0 with no output
  3. Introduce a deliberate Lua syntax error and run luac -p on that file — expect a nonzero exit with a line-number error message
  4. Run nvim --headless -c "qall" on the same broken file — expect exit 0 (confirming the documented limitation)

Done when

CLAUDE.md accurately describes what the headless command does and does not check, and documents an alternative that reliably exits nonzero on Lua syntax errors.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions