Context
In #4048, @jonkeane noted that the "command construction smoke test" suite in quartoProjectHelper.smoke.test.ts duplicates assertions already covered by the unit tests in quartoProjectHelper.test.ts. @zackverham agreed the command-capture approach is brittle, though the tests do serve a purpose: verifying that QuartoProjectHelper successfully invokes the mocked terminal with a render command when given real filesystem inputs.
Current state
Unit tests (quartoProjectHelper.test.ts):
- Mock both
runTerminalCommand and fileExistsAt
- Assert the exact command string passed to
runTerminalCommand (e.g., quarto render ".")
- Cover project vs. document rendering, multi-level workspaces, and error cases
Smoke tests (quartoProjectHelper.smoke.test.ts, "command construction" suite):
- Use the real filesystem (create temp dirs with
_quarto.yml) but mock runTerminalCommand
- Capture the command string and assert the exact same
quarto render "..." pattern
- Overlap significantly with unit tests — the assertions are nearly identical
Problem
The smoke test command assertions are brittle because they test the exact command string, which is an implementation detail. If we change how the command is constructed (e.g., add flags, change quoting), both the unit tests and the smoke tests break, creating unnecessary maintenance burden.
The smoke tests' real value is in verifying the filesystem-to-render-mode integration: given a real directory with/without _quarto.yml, does render() invoke the terminal at all and pick the right rendering mode (project vs. document)?
Suggested improvements
-
Relax exact command matching in smoke tests — Instead of asserting the full command string, assert that:
runTerminalCommand was called (i.e., a render was attempted)
- The command starts with
quarto render
- For project mode: the command contains the project directory path (not a specific
.qmd file)
- For document mode: the command contains the source file path
Example:
expect(capturedCommand).toBeDefined();
expect(capturedCommand).toMatch(/^quarto render /);
expect(capturedCommand).toContain(tmpDir); // project mode
-
Keep exact command assertions only in unit tests — quartoProjectHelper.test.ts already covers the precise command construction with controlled mocks; that's the right place for those assertions.
-
Consider whether the command construction smoke tests are needed at all — The "real Quarto render" tests (which actually execute quarto render) already validate end-to-end behavior. The command construction smoke tests occupy a middle ground that may not add enough value over the unit tests + real render tests.
Related
Context
In #4048, @jonkeane noted that the "command construction smoke test" suite in
quartoProjectHelper.smoke.test.tsduplicates assertions already covered by the unit tests inquartoProjectHelper.test.ts. @zackverham agreed the command-capture approach is brittle, though the tests do serve a purpose: verifying thatQuartoProjectHelpersuccessfully invokes the mocked terminal with a render command when given real filesystem inputs.Current state
Unit tests (
quartoProjectHelper.test.ts):runTerminalCommandandfileExistsAtrunTerminalCommand(e.g.,quarto render ".")Smoke tests (
quartoProjectHelper.smoke.test.ts, "command construction" suite):_quarto.yml) but mockrunTerminalCommandquarto render "..."patternProblem
The smoke test command assertions are brittle because they test the exact command string, which is an implementation detail. If we change how the command is constructed (e.g., add flags, change quoting), both the unit tests and the smoke tests break, creating unnecessary maintenance burden.
The smoke tests' real value is in verifying the filesystem-to-render-mode integration: given a real directory with/without
_quarto.yml, doesrender()invoke the terminal at all and pick the right rendering mode (project vs. document)?Suggested improvements
Relax exact command matching in smoke tests — Instead of asserting the full command string, assert that:
runTerminalCommandwas called (i.e., a render was attempted)quarto render.qmdfile)Example:
Keep exact command assertions only in unit tests —
quartoProjectHelper.test.tsalready covers the precise command construction with controlled mocks; that's the right place for those assertions.Consider whether the command construction smoke tests are needed at all — The "real Quarto render" tests (which actually execute
quarto render) already validate end-to-end behavior. The command construction smoke tests occupy a middle ground that may not add enough value over the unit tests + real render tests.Related
quarto renderwhen a page uses revealjs #4021 (original issue about--to htmlbreaking revealjs)