Skip to content

v0.5.9: Fix: --score-code results from earlier versions could be wrong; re-measure them

Choose a tag to compare

@github-actions github-actions released this 10 Oct 18:28
· 94 commits to main since this release

Fixed

  • --score-code and --score-suite could print a wrong score, in either
    direction, on any project.
    Each planted fault rewrites a file, and many
    rewrites, such as == to !=, leave it the same size. Python reuses a
    cached .pyc when a file's size and whole-second modification time match,
    so two such faults written within one second ran the first one's bytecode,
    and the second was given the first one's verdict. The same suite scored 2,
    3 or 4 of 8 depending on timing, including with a fixed --score-seed;
    with the fix, ten runs of it score 3 of 8 every time. Every __pycache__
    in the workspace is now deleted before each suite run, so no run can read
    a cache an earlier one wrote.
    Scores printed by earlier versions should be re-measured before they are
    quoted.
  • A score is refused when the suite never ran the code being scored.
    When the suite imported an installed copy of the package rather than the
    workspace copy, every planted fault ran against untouched code and the
    score read near 0%. That happened with a src/ layout and an editable
    install, a plain pip install ., a namespace package, and a suite living
    inside its own package. The suite is now run once with a marker in each
    scored file and with the files it loaded recorded. A scored file counts as
    never run when its marker did not fire, or when the suite loaded the
    original or an identical copy of it instead, and the score is refused when
    that is true of every scored file. Files the suite never ran are named in
    the report. When the check itself cannot complete, the report says so above
    the score, and a run in which nothing was caught is refused.
  • A suite inside the code directory is no longer mutated itself. Its
    broken tests were counted as faults caught.

Added

  • --compare-prompts TASK groups a task's run summaries by the prompts
    that produced them and compares the arms, without a model call. --repeats N runs N more of each arm first, alternating. It refuses to compare arms
    with fewer than three runs, arms that converged differently, or arms that
    differ in model or settings, and it judges noise by the interquartile range.
    To make that possible, a run summary now records which prompts were
    overridden and a digest of each (schema 7), and QIKLY_PROMPT_OVERRIDES=0
    makes one process use the shipped prompts while overrides stay on disk.
  • --install-mcp cursor writes .cursor/mcp.json, and a bare
    --install-mcp now writes it alongside the Claude Code and VS Code files.
    The file is checked; Cursor loading it has not been, which docs/mcp.md
    says.
  • Skill 0.3.6. The Skill now also sits at .claude/skills/qikly/SKILL.md,
    the path Skill directories crawl, and its mutation-score caution no longer
    quotes a count measured before the bytecode fix.

Changed

  • The landing page says what qikly is for, functional tests for backend
    Python, and what it does not test: interfaces, performance, load, security
    or other non-functional properties. It is about a third of its former
    length; the command table moved to TASK_FILE_REFERENCE.md.
  • A failing test was described as returning only its error text, on the
    landing page and in docs/FINDINGS.md. The coding agent also sees the
    failing test's source, and both now say so.
  • docs/FINDINGS.md is published, which the README already linked to,
    with the pricing example re-measured on the fixed scorer: 3 of 8, where it
    had said 4 of 8.