Lint Rules and Errors

ask.py lint applies the rules below; serve and add run the same lint first. An error makes lint exit 1 and stops serve and add unless --force is given. A warning is reported and nothing stops. Each finding is {rule, q, msg}.

Rules for the whole set

Printed for a set with "askwright": 2, the id has space, no title, a media root that does not exist and no questions:

version: the set must start with "askwright": 1
set-id: the set needs an id of letters, digits, '-', '_' or '.'
set-title: the set needs a title
no-questions: the set has no open questions

no-questions ends the lint, so the bad media root was not reported in that run. With questions present, a missing root is the error media-root with the message "media root /work/proj/ask/renders is not a folder" (Commands shows it in full).

Two warnings look at the set as a whole. From a set of 13 copies of one question:

too-many: 13 open questions; more than 12 tires the reader, so split the set or close some
guess-always-first: the guess is always the first option; vary the option order

Rules for each question

One set broke most of the rules at once; lint printed these, in this order ([q] is the question id):

error duplicate-id [dup]: two questions have the id dup
error id [_tmp]: question id '_tmp' must be letters, digits, '-', '_' or '.' (max 64)
error kind [_tmp]: kind must be one of choice, multi, compare, rank, text, scale, confirm
error title [_tmp]: every question needs a short title
error question-mark [_tmp]: question must be a question that ends in '?' (put the label in title)
error why [_tmp]: why is missing: say what the answer changes
error options [_tmp]: a question needs at least 2 options (text questions need none)
error guess [_tmp]: every question carries the agent's guess: {option, why, confidence}
error position-label [dup]: option label 'Left' is only a position or yes/no; name the action
error option-id [dup]: option id 'unsure' is missing, repeated or reserved (unsure, other)
error option-label [dup]: option label 'Blue' is empty or repeated
error guess-confidence [dup]: guess.confidence must be "high" or "low"
error guess-why [dup]: guess.why is missing
error guess-option [dup]: the guess does not name the question's options (option, options, order or text)
error duplicate-title [dup]: open questions dup and dup share a title
error confirm-yes-no [dup]: confirm option 'Yes, keep it' starts with yes/no; name the action
error guess [dup]: every question carries the agent's guess: {option, why, confidence}
error media-missing [clip]: media: missing.mp4 does not exist
error media-type [clip]: media: notes.txt is not a video, audio or image file
error media-outside [clip]: media: ../outside.png is outside the set folder and media_roots
error sides [clip]: sides maps a side name to an option id, plus randomized: true
error show-if [clip]: show_if needs {question: <id in the set>, option: [<its option ids>]}
error one-pager [clip]: one_pager block 0 has an unknown type
error superseded-by [old]: a superseded question needs superseded_by naming a question in the set
error withdrawn-reason [gone]: a withdrawn question needs a reason
error look-for [same]: look_for is required on compare questions and questions with media
error status [later]: status must be one of open, superseded, withdrawn, closed
error duplicate-question [wordy]: open questions same and wordy ask the same thing
error guess-option [wordy]: the guess does not name the question's options (option, options, order or text)
warning double-ask [dup]: the question seems to join two asks; split it in two
warning leading [dup]: the question leads the reader; ask it neutrally
warning alt [clip]: media: image still.png has no alt text
warning vague-look-for [clip]: look_for 'quality' does not name a concrete thing to look at
warning sides-not-randomized [clip]: compare takes should sit on random sides (sides.randomized: true)
warning long-title [wordy]: the title is over 80 characters; it is a caption, keep it short
warning long-text [wordy]: why is longer than 3 sentences; move detail to a one_pager

Three more rules did not fire in that run; their messages are from the source. media (an error) is "media must be a list" or "media: every item needs a src". question (an error) is "every question must be an object". codec (a warning, only with ffprobe installed and without --no-probe) names the codecs found: "clip.mp4 is hevc/aac; H.264 with AAC plays in every browser" for an HEVC video.

What each rule looks for:

Rule	Level	Fires when
id, duplicate-id	error	The id breaks the pattern (a leading _, - or ., a space, more than 64 characters), or two questions share one.
kind, status	error	Not one of the listed values.
title, long-title	error, warning	The title is empty; over 80 characters.
question-mark	error	The question does not end in ?.
why, long-text	error, warning	why is empty; why or hint has more than three sentences.
duplicate-title, duplicate-question	error	Two open questions share a title or a question, ignoring case.
double-ask	warning	and or or followed by a verb such as "should", "keep", "ship" or "merge" ("Which colour and should we ship it?").
leading	warning	Words such as "obviously", "surely", "don't you think", "isn't it".
look-for, vague-look-for	error, warning	look_for is missing on a compare question or one with media, or is not a list of strings; an item under three words or like "quality", "overall", "the vibe".
options, option-id, option-label	error	Fewer than two options on a kind other than text; an id that is missing, repeated, badly formed or reserved; an empty or repeated label.
position-label	error	A label that is only a position or a yes/no: left, right, top, first, a to e, a number, option 2, choice 3, yes, no, ok, sure.
confirm-yes-no	error	A confirm label that starts with "Yes" or "No".
guess, guess-confidence, guess-why, guess-option	error	No guess; confidence not high or low; no why; the guess does not name the question's options in the form its kind needs.
sides, sides-not-randomized	error, warning	sides maps to something that is not an option id; a compare question without "randomized": true.
show-if	error	show_if names a question not in the set, itself, or options that question lacks.
one-pager	error	A block with an unknown type.
media-missing, media-outside, media-type, alt	error, warning	A file that cannot be found, one outside the set folder and media roots, an extension that names no video, audio or image type, and an image without alt.
superseded-by, withdrawn-reason	error	A superseded question without a valid superseded_by; a withdrawn one without reason.

Rules for study sets

Added in 0.2.0. One study set broke them; lint printed:

error study-kind [rank]: a study set asks choice (multiple choice) or text (written) questions
error points [two]: points must be a number above 0
error study-correct [two]: a multiple-choice question needs exactly one option with correct: true
error study-why [two]: option b needs a why: why it is right, or the mistake it stands for
error study-rubric [essay]: a written question needs a model_answer or a rubric for whoever grades it
error figure [pic]: media: figure type must be graph, forces or shapes
warning topic [two]: give every study question a topic; the review scores by topic
warning study-few-options [two]: two options is a coin toss; add wrong answers that each stand for a real mistake
warning study-few-options [coin]: two options is a coin toss; add wrong answers that each stand for a real mistake
warning alt [pic]: media: the figure has no alt text; say what it shows
warning study-longest-right [None]: the right answer is the longest option in 4 of 5 questions; students learn to pick the longest, so trim it or lengthen a wrong one

study-longest-right looks at the whole set: it warns when, over at least four multiple-choice questions with three or more options, the right answer is the longest in more than half. figure and its alt warning apply in any set; Figures lists every figure message.

A decision set with correct, rubric or model_answer in it gets a warning, since only a quiz uses them:

study-fields [music]: correct, model_answer and rubric only work in a set with "mode": "study"

In a study set two rules are relaxed: question-mark only needs a ? somewhere in the question, and option-label compares labels with their capitals. The guess rules, the question's why, the look_for requirement, double-ask and position-label do not apply.

Errors from the commands

A folder with no set, and a file that is not JSON:

$ python ask.py lint ask/nothing-here
{
  "ok": false,
  "error": "no questions.json at ask/nothing-here/questions.json"
}
$ echo $?
1
$ python ask.py lint ask/broken
{
  "ok": false,
  "error": "questions.json is not valid JSON: Expecting property name enclosed in double quotes: line 1 column 17 (char 16)"
}
$ echo $?
1

The JSON message is Python's own and was the same on 3.9 and 3.14. The other command errors are on Commands: new on an existing set, add without --replace, close with an unknown id, serve on a set that fails the lint, and usage mistakes (exit 2). The server's errors are on Server and security.

Things the lint does not check

• Whether the guess is right, or the question worth asking: that is the skill's job, in writing-questions.md.
• Whether randomized: true is true. It takes your word for it.
• look_for on a question whose only media is in its one_pager.
• The codecs of audio files, and of video when ffprobe is not installed.