You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
v0.5.9: Fix: --score-code results from earlier versions could be wrong; re-measure them
--score-code and --score-suite could print a wrong score, in either
direction, on any project. Each planted fault rewrites a file, and many
rewrites, such as == to !=, leave it the same size. Python reuses a
cached .pyc when a file's size and whole-second modification time match,
so two such faults written within one second ran the first one's bytecode,
and the second was given the first one's verdict. The same suite scored 2,
3 or 4 of 8 depending on timing, including with a fixed --score-seed;
with the fix, ten runs of it score 3 of 8 every time. Every __pycache__
in the workspace is now deleted before each suite run, so no run can read
a cache an earlier one wrote. Scores printed by earlier versions should be re-measured before they are
quoted.
A score is refused when the suite never ran the code being scored.
When the suite imported an installed copy of the package rather than the
workspace copy, every planted fault ran against untouched code and the
score read near 0%. That happened with a src/ layout and an editable
install, a plain pip install ., a namespace package, and a suite living
inside its own package. The suite is now run once with a marker in each
scored file and with the files it loaded recorded. A scored file counts as
never run when its marker did not fire, or when the suite loaded the
original or an identical copy of it instead, and the score is refused when
that is true of every scored file. Files the suite never ran are named in
the report. When the check itself cannot complete, the report says so above
the score, and a run in which nothing was caught is refused.
A suite inside the code directory is no longer mutated itself. Its
broken tests were counted as faults caught.
Added
--compare-prompts TASK groups a task's run summaries by the prompts
that produced them and compares the arms, without a model call. --repeats N runs N more of each arm first, alternating. It refuses to compare arms
with fewer than three runs, arms that converged differently, or arms that
differ in model or settings, and it judges noise by the interquartile range.
To make that possible, a run summary now records which prompts were
overridden and a digest of each (schema 7), and QIKLY_PROMPT_OVERRIDES=0
makes one process use the shipped prompts while overrides stay on disk.
--install-mcp cursor writes .cursor/mcp.json, and a bare --install-mcp now writes it alongside the Claude Code and VS Code files.
The file is checked; Cursor loading it has not been, which docs/mcp.md
says.
Skill 0.3.6. The Skill now also sits at .claude/skills/qikly/SKILL.md,
the path Skill directories crawl, and its mutation-score caution no longer
quotes a count measured before the bytecode fix.
Changed
The landing page says what qikly is for, functional tests for backend
Python, and what it does not test: interfaces, performance, load, security
or other non-functional properties. It is about a third of its former
length; the command table moved to TASK_FILE_REFERENCE.md.
A failing test was described as returning only its error text, on the
landing page and in docs/FINDINGS.md. The coding agent also sees the
failing test's source, and both now say so.
docs/FINDINGS.md is published, which the README already linked to,
with the pricing example re-measured on the fixed scorer: 3 of 8, where it
had said 4 of 8.