Skip to content

Reduce brittleness of quarto command assertions in smoke tests #4058

Description

@posit-connect-projects

Context

In #4048, @jonkeane noted that the "command construction smoke test" suite in quartoProjectHelper.smoke.test.ts duplicates assertions already covered by the unit tests in quartoProjectHelper.test.ts. @zackverham agreed the command-capture approach is brittle, though the tests do serve a purpose: verifying that QuartoProjectHelper successfully invokes the mocked terminal with a render command when given real filesystem inputs.

Current state

Unit tests (quartoProjectHelper.test.ts):

  • Mock both runTerminalCommand and fileExistsAt
  • Assert the exact command string passed to runTerminalCommand (e.g., quarto render ".")
  • Cover project vs. document rendering, multi-level workspaces, and error cases

Smoke tests (quartoProjectHelper.smoke.test.ts, "command construction" suite):

  • Use the real filesystem (create temp dirs with _quarto.yml) but mock runTerminalCommand
  • Capture the command string and assert the exact same quarto render "..." pattern
  • Overlap significantly with unit tests — the assertions are nearly identical

Problem

The smoke test command assertions are brittle because they test the exact command string, which is an implementation detail. If we change how the command is constructed (e.g., add flags, change quoting), both the unit tests and the smoke tests break, creating unnecessary maintenance burden.

The smoke tests' real value is in verifying the filesystem-to-render-mode integration: given a real directory with/without _quarto.yml, does render() invoke the terminal at all and pick the right rendering mode (project vs. document)?

Suggested improvements

  1. Relax exact command matching in smoke tests — Instead of asserting the full command string, assert that:

    • runTerminalCommand was called (i.e., a render was attempted)
    • The command starts with quarto render
    • For project mode: the command contains the project directory path (not a specific .qmd file)
    • For document mode: the command contains the source file path

    Example:

    expect(capturedCommand).toBeDefined();
    expect(capturedCommand).toMatch(/^quarto render /);
    expect(capturedCommand).toContain(tmpDir); // project mode
  2. Keep exact command assertions only in unit testsquartoProjectHelper.test.ts already covers the precise command construction with controlled mocks; that's the right place for those assertions.

  3. Consider whether the command construction smoke tests are needed at all — The "real Quarto render" tests (which actually execute quarto render) already validate end-to-end behavior. The command construction smoke tests occupy a middle ground that may not add enough value over the unit tests + real render tests.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions