Skip to content

v0.4.7: Register the MCP server with one command, and every generated test names the criterion it came from

Choose a tag to compare

@github-actions github-actions released this 20 Sep 15:13
· 294 commits to main since this release

Added

  • Each generated test names the acceptance criterion it came from. Test
    generation is asked to end every test's docstring with Criteria: 3, or
    Criteria: none for a test written from the requirements, with requirement
    indices on their own Requirements: line. qikly.traceability reads those
    back out of the suite source with ast, the run summary carries
    criteria_coverage (schema 6, additive, still pools with 3, 4 and 5), the
    HTML report gains a Criteria coverage section, and
    python -m qikly.orchestrator.tuning.trace_criteria reports the same for
    suites already on disk, offline and free.

    It exists because six refinement experiments compared two suites in
    aggregate and found nothing, and an aggregate cannot tell a bar that adds
    nothing from a bar the generator ignores; the measured 54%-more-criteria,
    6%-smaller-suite result says the second happens. Per-criterion counts make
    it direct: delete a criterion, regenerate, and the change in its row prices
    that criterion with no fault model in between.

    The label is self-reported. A test naming criterion 3 is evidence the
    generator intended to cover it, not proof that it did, so read an uncovered
    criterion as a lower bound and never a coverage guarantee. Three real runs
    shaped the parser: models write the line in the function body as well as
    the docstring, and they will put a requirement's index on the Criteria:
    line unless requirements have a line of their own. A label that could mean
    two things, Criteria: 3-5, is reported rather than guessed, and a test in
    a class or inside another function, which the reader does not see, is named
    rather than left out of the counts in silence. The marker is a comment,
    # Criteria: 3, because the model writes it in the function body as well as
    the docstring and Criteria: 1, 2 there is a Python syntax error that stops
    the file importing: on CALC_TAX, eleven criteria, that failed test
    generation three times in a row and ended the run. Measured at
    8/10 convergence on ADAS_HEADWAY against 9/10 for a matched control arm
    on the same ten seeds, which is one run apart and not a detectable cost.

  • qikly --install-mcp registers the server instead of printing a block to
    paste.
    Registering qikly with Claude Code on this machine meant finding
    that claude mcp add is not on PATH, because the CLI is bundled inside the
    VS Code extension, then reading ~/.claude.json by hand to see whether the
    server was already there. That was the author, on the machine the tool was
    built on, with the documentation open.

    It writes project-local files only, .mcp.json and .vscode/mcp.json, never
    a global config, and the project file is also where the setting that actually
    goes wrong belongs: which folder the server treats as the project. It merges
    rather than overwrites, keeps every other server and key in the file, and
    backs the file up first. Running it twice changes nothing.

    It refuses three cases and prints the block instead: a file holding comments,
    because VS Code's mcp.json is JSONC and a JSON round-trip would delete
    them; an existing qikly entry that differs, because somebody changed it on
    purpose, with --force to say otherwise; and a file that is not valid JSON.
    It writes into the directory you are standing in, and says so when
    QIKLY_PROJECT_ROOT is set to somewhere else rather than quietly using it:
    on the first real use of the command that variable was left in a shell from
    an earlier session and the files landed two directories away.

    --dry-run shows the plan, and it exits non-zero when it refused everything,
    so a provisioning step can tell that apart from nothing to do. The write is
    atomic, built beside the file and moved onto it, because opening the target
    truncates it and a crash in that window would leave a config holding every
    other server you have empty. It also refuses a symlinked config rather than
    following it out of the project, and tolerates the byte order mark
    PowerShell writes by default. Cursor and Codex CLI are not covered yet;
    Codex needs a TOML writer the standard library does not have.

    One thing to know: the file is re-serialised with two-space indentation, so a
    file checked into a repository may show a whitespace diff beyond the key that
    changed.

Changed

  • --version says which qikly you are running. It printed a number, and
    the number answered the wrong question. A flag reported as unrecognised, a
    suite exercising the released package instead of the branch, an MCP server
    that would not start: each came down to which install was on PATH and
    which Python ran it, and a second install shadowing the one you are editing
    looks identical until the path is printed. It now prints the version, the
    package directory and the interpreter, stacked.
  • --explain --html puts the verdict where the evidence is. The coding
    agent's column now leads, since the withheld-from side is the one a reader
    came to see, and "No criterion text reaches the coding agent" sits in that
    column's header rather than as a banner above both. As a banner it was a
    claim the reader had to carry down to the evidence. A FAILED verdict stays
    at the top, because a leak is not a caption. The lede drops the character
    count, which is still in the terminal output: it sat at the end of the one
    sentence that has to land, and a reader who has just been told what was cut
    does not also need to know how many bytes it was.