v0.6.0: Three steps on one path: --score, --scaffold and --tasks, answered by number

Changed

• Three steps, each taking the same module or folder: qikly --score my_module.py scores your tests (--score-code still works), qikly --scaffold my_module.py writes the task file with the score's questions in
  it to answer, and qikly --tasks my_module.py runs it. --validate,
  --explain and --take-tests take the same path, so nobody has to learn
  that my_module.py's task is called MY_MODULE_VERIFY; task ids still
  work as before.
• --scaffold leaves only the questions to answer, for code you already
  have. Everything it can read from the code is filled in: the task name,
  the description, the interface without a TODO per function, the
  entrypoint. requirements starts as the module's docstring or a line to
  keep what the code does, marked TODO: MODIFY THIS: for you to replace in
  your own words, which --validate reminds you of. Left in place, the line
  goes to the agents without the mark. Each section sits under
  a dividing line with a short note on what it is for, with an example of
  the answer for requirements and acceptance_criteria, and the seed, the file under test, comes first. A module inside a
  package is seeded as its whole top-level package and named by its dotted
  path (acme.scoring), so a relative import such as from .util import clamp works in qikly's copy.
• Each question in the task file says what your code does there now,
  found by running it in a separate process in an empty folder, and its TODO
  names the call: TODO: what apply_discount(100, 100) should do. A
  boundary is only ever run on either side of the value in question, never
  at it: "raise ValueError, as rate 101 does now" and "return a value, as
  rate 99 does now (it returns 1.0)". Other questions show what the code
  returns now and what the changed code would, labelled. The values come in
  an order fixed by the values, not with "as your code does now" first, and
  a function qikly sees reading or writing files is never run; the code runs
  in an empty folder, so only an absolute path it builds itself could reach
  your files. Questions about the
  same call with the same values are asked once. Those lines start with
  #?, so neither agent reads them.
• Answer a question with its number. Each option qikly found is
  numbered with the criterion it becomes, and a TODO line can be answered
  2, 1: 0.0 for an option that asks for a value, keep for the option
  that is what the code does now, or a sentence, with or without quotes.
  Both agents and --validate read the answer as the full criterion, the
  file keeps what you typed, and --validate prints each short answer as
  the criterion it becomes and stops on a number that names no option.
• A criterion left as TODO is skipped, not sent to test generation or
  any other model call as written: one that starts with TODO, followed by
  a colon, a space or nothing. A TODO inside a one-line list, [...], or
  on an anchored item is left as written, since removing it would break the
  file. --validate counts with the same rule, says
  how many are answered and that a run skips the rest, and --validate and
  --tasks both stop a file with none answered before a paid run.
• --scaffold fills the data file with the values your tests already call
  the code with, so it is never only a header, and says to add any your
  answers name.
• --scaffold takes a folder: one task file per module that the
  folder's last --score asked about. With no score yet, it says to score
  the folder first and names a module to scaffold instead. A package's
  __init__.py is named for the package, and a second module of the same
  name in another folder gets its folder's name too (SUB_UTIL_VERIFY),
  never another module's task file. Each task file names its module on its
  first line, which is how a path finds it, --task-id tasks included.
• A run of a task about your own code ends with the step after it,
  qikly --take-tests my_module.py, which runs the new tests on your code.
• --score plants every fault it can, up to 60 in one module or 150 in a
  folder, and a sample only when the suite is slow, where it used to plant a
  sample of 12. The demo module is now scored 10 of 13 missed where it was 9
  of 12.
• --score finds your tests when you do not name them: the test_*.py
  and *_test.py files that import the code, searched from the project
  root, and it says which it used. --score-tests still names them, and is
  still needed with nothing to break.
• --score and --score-suite lead with what was missed: "5 of 12
  planted faults missed (42%), 7 caught" rather than "7 of 12 caught". The
  misses are the finding, and a caught count read as a pass mark. Scripts
  that parsed the old line need the new form.
• The saved report gives both counts on their own lines, caught then
  missed, and numbers the misses, the questions and the catches. The note
  that a score is a floor rather than a verdict moved to the end, after the
  lists it qualifies.
• --scaffold's next steps name each file in full and put the free check
  before the paid run.
• A run that passes says where the new tests are, that they add to the
  ones you have, and that qikly --score-suite --tasks <TASK> shows how many
  planted faults they catch.

Added

• qikly --take-tests my_module.py: tests you can take home. The tests a
  run wrote import qikly's copy of your code; this copies each stage's files
  into qikly_generated/<task>/<stage>/ beside your tests, keeping their
  names, rewrites every import of qikly's copy to yours, writes a
  conftest.py that makes your code importable from wherever it lives, and
  runs them on your code. Failures are classified from pytest's own XML: tests
  written from your criteria, to compare with each criterion; tests no
  criterion is behind, the test writer's own reading, to check; calls to a
  helper only qikly's copy has; copies that could not import your module; and
  files that could not be collected. It never says to apply qikly's change. A
  file it did not write is never overwritten, a rerun removes its own stale
  copies, the folder is remembered, and outputs/conftest.py keeps a plain
  pytest from running qikly's own copies as well.
• --score asks a question about every fault your tests missed, and
  --scaffold puts the questions where the answers belong. The report ends
  with a plain question for each miss, naming the file, line and function,
  and the comparison itself when it is the only one of its kind on the line,
  such as "your tests pass whether this is rate > 100 or rate >= 100...
  what should happen at exactly 100?". Misses that ask the same thing share
  one question. Question lines start with #? and are removed from
  everything either agent receives, wherever they end up in the file,
  because they quote the code; the answers are withheld from the coding
  agent like any criterion. A question whose line has since changed is
  dropped, and a newer score that asks nothing replaces an older one that
  did. Nothing guesses an answer and no model is called; the answers are
  yours, because tests read off the code can only agree with it.
• Skill 0.3.7 walks an assistant through the three steps, and says the
  questions are the user's to answer, word for word, never its own.
• Two guided flows were built and shelved before release: one question at a
  time in the terminal, and a separate file of numbered answers. On real
  libraries only 1 question in 63 had choices to pick from, so the task file,
  answered in sentences, stays the one place answers live.

Fixed

• A failing test no answer is behind is never called a disagreement.
  --take-tests told the user to apply qikly's change to code that was
  right, when the failing tests were the test writer's own misreading of the
  API. Such failures are now listed as tests to check before changing
  anything, and the run report says the same. A failure written from an
  answer is no longer called a disagreement either: the test can assert
  more than the answer said, and qikly's change to its copy, which on one
  library altered a documented default, is offered to read, not to apply.
• Scoring no longer hangs on a fault that loops forever. Each fault gets
  ten times the untouched suite's time, and at least a minute; one that runs
  past it is stopped and counted as caught, and the report says how many.
  cachetools' scoring had stopped for good on one.
• A function defined under an if or try is not "missing". Such a test
  was set aside as an import failure, which hid a real disagreement.
• Two different checks in one function are two questions. Grouping now
  also needs the same code on each line, not only the same question.
• No anchor fault is skipped where it matters: .match(s, pos) and a
  public module-level pattern keep the ^ fault; an escaped \$ is no
  longer stripped into a pattern that does not compile.
• Test calls are read only where the value is certain: a name changed by
  +=, a loop, a with, a tuple unpacking or a later assignment is left
  out, and so is indirect parametrize. Rows in a table that also holds
  objects, such as date(...), are read value by value, and a function
  imported through its package (from isodate import parse_date) is found.
• --score on a folder with its tests inside says to score one
  module, instead of refusing a --score-tests you never typed; tests on
  another drive no longer crash it, and the path it prints can be pasted.
• import os.path no longer counts as importing a project's path.py.
• Smaller: a timed sample is never bigger than the whole set; __main__.py
  is not a module to test; a package run from inside its own folder is told
  to run from the folder above.
• No fault is planted by removing an anchor that changes nothing. A ^
  on a pattern only given to match, which anchors at the start already, and
  a $ too for fullmatch. Those faults could never be caught, and counted
  as misses: validators' card module was scored 6 of 41 missed and is 1 of 36.
  Scores on code without such patterns do not change.
• --score asks one question where the same gap repeats. A table of 37
  rules missing the same anchor asked the same thing 37 times; it is now one
  question naming the other lines. Questions about public functions come
  first, private helpers last.
• Values beside each question from parametrized tests, including rows
  kept in a module-level table, annotated or not. Real suites pass inputs
  that way, and no concrete call was found for them.
• A test that imports a name your module lacks is no longer a
  disagreement. It failed before checking anything, and --take-tests
  said your code disagreed with a criterion it met. It is now listed apart.
• A folder holding one module is scored as that module.
• Real libraries, found by running the three steps on python-slugify,
  inflection, tabulate, validators and humanize:
  • a package's __init__.py is scored and asked about when it holds code;
    tabulate and inflection keep the whole library there, and it was skipped;
  • tests are found when they import the package by name (import humanize)
    or a module's functions from it (from validators import card_number),
    and a single module gets its own test files before the whole suite;
  • an input passed through a variable (txt = "..."; slugify(txt)) counts
    for the values beside a question, as a literal does;
  • the suite is timed once, the scoring time is said, and above five minutes
    --score offers a smaller sample of faults, or takes one without a terminal;
  • when a suite fails on the untouched code, --score shows what pytest
    said, such as a module generated at build time.
• A run's log reads plainly. Lines no longer carry the task id twice
  ([TASK] [TASK] ...), and three research-flavoured lines say what they mean:
  a seeded run "uses your own implementation ... so its results are not
  comparable with runs where qikly writes them", the git check says it
  "cannot show whether the acceptance criteria were written before the
  code", and the seed line says what the seed is for.
• --validate no longer warns that a criterion written as a call restates
  the interface. A signature qikly wrote, such as
  delivery_days(country_code, express=False), shares every word with a
  criterion like delivery_days('DE') returns 10, and the warning told a
  first-time user they had handed over the answers.
• Score percentages round instead of rounding down, so "2 of 3 caught"
  reads 67% rather than 66%, and the report's caught and missed lines sum to
  100.
• --validate warns about a data file with only its header line, the
  state --scaffold leaves it in; skipping the rows used to pass silently.
• Scoring qikly's generated suite against your own file now says what
  happened. Those tests import qikly's copy of the code, so the file is
  never run, and the message blamed an installed package. It now names
  qikly --score-suite --tasks <TASK>.
• --score no longer prints a request for feedback. It came straight
  after the result, in project jargon, and read as part of it. The request is
  now a short, optional last section of the report, below the misses it asks
  about, and QIKLY_NO_INVITE is no longer needed.
• A --score-tests path that does not exist now says where the tests are.
  With test_pricing.py beside pricing.py and --score-tests tests/ typed
  as the docs show, the message was only no suite at ...\tests. It now names
  the test files it finds beside the code and the flag that would work, for
  example try --score-tests test_pricing.py.
• --criteria-from --task-id on a freshly scaffolded task wrote invalid
  YAML. --scaffold's placeholder criterion wraps onto a second line, and
  only its first line was removed, so the second was left behind as a stray
  scalar and every later command reported the task file as unparseable. The
  documented sequence, scaffold and then import criteria, failed on every
  scaffolded task. The whole placeholder block is now removed.
• A task file with a non-ASCII character could not be read on Windows.
  The prompt builder opened it without an encoding, so Windows read it as
  cp1252, and one accented letter stopped --explain and the run while
  --validate, which reads UTF-8, said nothing. Task files are now read as
  UTF-8, and a file saved in the old local encoding is still read.
• --scaffold on a file whose name is not a Python name wrote a task no
  test could import. my-calc.py gave module: "...MY_CALC_VERIFY.my-calc".
  The module path now uses an importable name (my_calc), and a run copies
  the seeded file in under that name and says so. Your own file keeps its
  name.