You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
v0.4.6: Ask it anything: docs/FAQ.md, and --explain now counts criteria rather than lines
A FAQ, from questions people actually asked.docs/FAQ.md answers
whether qikly can use classes you already have, such as hardware drivers and
protocol parsers, and whether anything runs anywhere but your own machine.
The import answer is the one worth knowing before you start: each stage runs
as python -m pytest from the project root, so a package at the project root
or installed in the same virtualenv imports from generated tests and
generated code alike, and one in a subdirectory that is not on the path stops
the run with a collection error that no amount of iterating can fix.
An opt-in check that the generated suites can be passed at all. With test_generation.check_suites: true, after the integration and system tests
are generated one model call reads every limit the acceptance criteria state,
then every test's comparison at that limit, and reports a test no correct
implementation could pass, whether it contradicts a criterion or another
test. A suite with a finding is written once more and checked again. Found on
a following-distance task whose criterion said exactly 2.00 seconds of
headway raises no warning: in seven of ten runs a generated test warned at
2.00 anyway, and all seven failed, because the coding agent sees only failure
output and cannot tell a wrong test from a right one. On those saved suites
the check flagged all seven, but a ten-run sweep with it on converged 4 in 10
against 3 without, and one wrong finding led a correct suite to be rewritten
wrong, so it is off by default. To check suites already on disk: python -m qikly.orchestrator.tuning.check_suites --tasks <TASK>.
A stall names two tests that disagree. When the failing tests alternate
between two sets, each fix breaking the other, the stop message says so and
names both, which points at the tests instead of at a correct spec.
Three example tasks from vehicle sensor data, the first bundled family
outside business data pipelines. ADAS_HEADWAY checks following distance
from forward-radar samples, ADAS_TTC computes time to collision from radar
tracks whose closing speed may be zero or negative, and ADAS_SPEED_LIMIT
checks speed against a posted limit through a unit conversion and an
enforcement tolerance. Each states its decisions in the requirements and
withholds what follows from them, including what happens exactly at every
limit, and what type each output field carries. They are not part of the
published convergence figures: on ten seeds each they converged 10, 10 and 8
times out of 10 on gemini-3.5-flash-lite, which is a first look at one task
apiece, not a rate.
A seeded run says whether the criteria predate the code. Withholding is a
property of a run qikly performed: when it writes the implementation, the
coding agent provably never saw the acceptance criteria, because one process
controlled both sides. A run seeded with seed.implementation, which is what qikly --scaffold writes for code you already have, cannot enforce that,
since it did not write the code and cannot know what its author read. It can
evidence it, within bounds it states rather than hides. If the task file was
last changed before the implementation's first commit, the criteria were not
written against code that was already in the repository. That is all it
means, and the finding is printed with its limit attached: git witnesses when
each side was committed, not when either was written, so code that sat
uncommitted in a working tree could still have shaped the criteria, and
criteria committed first say nothing about whether the code's author read
them, that order being exactly what reading them would require. The verdict
is therefore worth most when it comes out badly, in the same way a failing
test is worth more than a passing one. The run also reports whether the two
sides were committed by the same person, which is the part of separation of
duties version control really does know. Git is asked at
the start of the run, the verdict
opens the run on the console and sits under the result banner in the HTML
report, and the run summary carries it as criteria_independence
(schema_version 5, additive, still pools with 3 and 4). The conservative
reading is deliberate: the last change to the whole task file is compared
with the implementation's first commit, so a criterion revised after the
code existed loses the claim. Outside a repository, or for a file git does
not track, the run says the order could not be evidenced either way rather
than claiming anything, and no failure here can fail a run.
Changed
TROUBLESHOOTING.md is ordered by triage priority. Its eleven sections now
run in the order to work through a stall: a stronger model first, then read
the report, rule out an import error or a forgotten setting, fix the task
file, and only then give it more attempts. The triage table lists all eleven
in that order, including the contradiction, adjective and fixture checks it
used to leave out, and a new section for two generated tests that disagree.
Fixed
--explain counts criteria, not lines. It reported "12 lines removed" one
sentence after "declares 11 acceptance criteria", and the two numbers are both
right: the twelfth line is the acceptance_criteria: key the list hangs off.
A reader who stops to reconcile two numbers is a reader who distrusts them,
which is the wrong reaction to the one screen that exists to demonstrate the
claim. The terminal and the shareable page now both lead with the criteria,
and the hole in the coding agent's column reads "11 acceptance criteria
removed here".
The MCP qikly_validate reply says it is not a contradiction check.
Asked to validate CALC_TAX, GitHub Copilot relayed the counts correctly and
then summarised them as "no contradictions", which the tool never checks. The
reply's note now says it is the free offline check of the task file and that
contradictions are qikly --check-criteria. It still quotes no criterion.