How Images Are Judged

How artwright-judge turns model judgements into a ranking, what aw.py decides and what it leaves to the model, and what the measured numbers mean. The skill does the looking; the script keeps the record and does the arithmetic.

Why two at a time, in both orders

The judge skill rests on findings in the repository's RESEARCH.md: models judging beauty are weak and swayed by position. Swapping the order of two images flips up to 61% of verdicts, and moving from two candidates to four or more hurts models about seven times as much as people. So the skill:

• shows one pair per call, and each pair twice, in fresh calls, once per order;
• labels the candidates 1 and 2 and never shows file names, the model, the seed, the user's guess or other votes;
• treats a split between the two orders as a tie for a person to see, never a coin toss;
• says when a pick is taste rather than craft (both images intact and skilled), names the taste axis, and never calls one image "objectively" better.

aw.py compare prepare makes the two order images (see Commands); the model judges each; aw.py compare record stores each pick.

How a pair resolves

Each compare record appends one row to matches.jsonl and answers with the pair's state. The first record for each order counts; later records for the same order are re-queries.

Recorded	status	Notes
only one order so far	waiting	need_order says which order is missing
both orders picked the same image	decided	winner is a or b, the first or second file given to prepare; flipped: false
the orders picked different images	tie	flipped: true, flag: "orders disagree: show a human"
a split, then a re-query agreeing with one side	decided	flipped: true, tiebreak: true: the winner is the side the first re-query that is not a tie names
one order a pick, the other tie	tie	flipped: false; a re-query naming the picked side breaks it
both orders tie	tie	flipped: false; since 0.1.1, see Versions and upgrading

The orders disagreeing, run on dusk.png and noon.png in a work folder named flip:

python aw.py compare record 7313d500b577 1 1 --work flip
python aw.py compare record 7313d500b577 2 1 --work flip

The second record prints:

{
 "pair": "7313d500b577",
 "resolved": {
  "status": "tie",
  "winner": null,
  "flipped": true,
  "flag": "orders disagree: show a human"
 },
 "ok": true
}

Order 1 showed dusk.png on the left and order 2 showed it on the right, so picking the left image both times is the position bias the method is built to catch. The rubric allows a third run to break a tie only by agreeing with one of the two earlier verdicts:

python aw.py compare record 7313d500b577 1 1 --work flip

prints:

{
 "pair": "7313d500b577",
 "resolved": {
  "status": "decided",
  "winner": "a",
  "flipped": true,
  "tiebreak": true
 },
 "ok": true
}

The ranking

aw.py rank fits a Bradley-Terry model over every resolved pair: a decided pair is a win and a loss, a tie is half a win each. Each image also gets one virtual tie against an average player, which keeps an unbeaten image's strength finite. Strengths are normalised so their geometric mean is 1.

The flip folder ranked after the re-query above:

python aw.py rank --work flip

prints:

{
 "ranking": [
  {
   "rank": 1,
   "file": "dusk.png",
   "sha256": "01069b7b7057f04bb65c47b1bd01c995bcae247570c53c24e6b7dde6053350c0",
   "strength": 2.13,
   "wins": 1,
   "losses": 0,
   "ties": 0,
   "close_to_next": false
  },
  {
   "rank": 2,
   "file": "noon.png",
   "sha256": "1139a0a24d75eabd11ce68c29b4c1c62c69782d5046fd09d1f2d4028088a3834",
   "strength": 0.469,
   "wins": 0,
   "losses": 1,
   "ties": 0,
   "close_to_next": null
  }
 ],
 "games": 1,
 "abstained": [],
 "cycles": [],
 "cycle_count": 0,
 "top_in_cycle": false,
 "note": "close_to_next: strengths within 1.5x, never compared directly, or inside a cycle (A beat B, B beat C, C beat A); do not claim an order there. top_in_cycle: the winner is close, say so",
 "ok": true
}

Reading it:

• close_to_next is true when a row's strength is less than 1.5 times the next row's, when the two never met directly, or when both sit in a cycle; the order between those two is not established, and the skill says so. The last row has no next row, so it is null.
• cycles, cycle_count and top_in_cycle (0.3.0) report three-way loops among decided pairs, described in Cycles below. Here there are none.
• abstained lists the pairs whose orders disagreed. A caller must not silently pick between them.
• Equal strengths (two images with the same record) come out in an order set by their SHA-256, which says nothing about the images.
• A pair waiting for its second order is left out of games, wins and losses.

A four-image round robin with one split pair is on the Verdict contract page, where rank --write puts the ranking into verdict.json.

Swiss rounds. Above about 12 candidates the skill does not judge every pair. rank --next N suggests up to N pairs down the current ranking whose two images have no resolved result yet, each image in at most one pair.

Cycles

Bradley-Terry always produces an order, even when the picks go round in a loop. Since 0.3.0, rank finds three-way loops among decided pairs (A beat B, B beat C, C beat A; longer loops are not searched), sets close_to_next on every row from the loop's highest-ranked member down to the row above its lowest, rows of other images between them included, and says whether the top image is in a loop. Three renders judged in a work folder named loop, with the same pick in both orders each time: render_01 beat render_02, render_02 beat render_03, and render_03 beat render_01.

python aw.py rank --work loop

prints:

{
 "ranking": [
  {
   "rank": 1,
   "file": "renders/render_02.png",
   "sha256": "666fa6e5311dae41e94c6a76419982ae459480b39f498676d2267f92a609dfaa",
   "strength": 1.0,
   "wins": 1,
   "losses": 1,
   "ties": 0,
   "close_to_next": true
  },
  {
   "rank": 2,
   "file": "renders/render_01.png",
   "sha256": "a281f69710d4b9e2c7c204ee01e0b99def061bf9092bad6f060ed1a77ad60aa2",
   "strength": 1.0,
   "wins": 1,
   "losses": 1,
   "ties": 0,
   "close_to_next": true
  },
  {
   "rank": 3,
   "file": "renders/render_03.png",
   "sha256": "f152d4d30a807dc9d6b4ed2fe8ad398d2fe16c7e8db4e96c0c3b45505beb5307",
   "strength": 1.0,
   "wins": 1,
   "losses": 1,
   "ties": 0,
   "close_to_next": null
  }
 ],
 "games": 3,
 "abstained": [],
 "cycles": [
  {
   "files": [
    "renders/render_02.png",
    "renders/render_03.png",
    "renders/render_01.png"
   ],
   "ranks": [
    1,
    2,
    3
   ]
  }
 ],
 "cycle_count": 1,
 "top_in_cycle": true,
 "note": "close_to_next: strengths within 1.5x, never compared directly, or inside a cycle (A beat B, B beat C, C beat A); do not claim an order there. top_in_cycle: the winner is close, say so",
 "ok": true
}

The three strengths are equal and the order among them comes from their SHA-256, so it means nothing. cycles lists at most 20 loops; cycle_count counts them all. When top_in_cycle is true, the judge skill reports the winner as close, whatever its strength, and the rubric treats the members of a cycle as having no order.

The rubric

The judge skill scores every pair, brief check and critique against one rubric, artwright-judge, whose version is part of the cache key and of verdict.json. Version 3 (0.5.0) checks, in this order:

1. Brief fit, as yes/no checks split from the brief, with hard fails listed apart. A word that names a look becomes the traits that show it: "golden hour" is a low orange sun, blue shadows and light only on the tops of tall things; "overcast" is no cast shadows and the sky as the lightest note.
2. Read and composition, each check tied to a region: the three-value read, the brief's subject against the measured focal peak (aw.py comp --subject), the armature, edges and corners, tangents and mergers, horizon and camera height, depth planes, the main silhouette. A focal_off_subject flag must be answered in the notes.
3. Light logic, one plan for the whole image, with aw.py light and depth pointing at where to look. An artificial or stylised brief may ask for broken daylight rules.
4. Value and colour.
5. Rendering integrity, a gate and not a matter of taste; two light directions in one composite fail it.
6. Mechanism, a gate like integrity, for any image whose subject or brief includes a working machine: the power is traced from its source to the tool and the broken link is named with its region. Skipped for a ruin, a toy, an ornament or a brief that asks for an impossible machine. Over a 3D render, the render shows the working layout.
7. Style coherence.
8. Expressive intent.

Version 3 added the mechanism criterion; verdicts cached under version 2 no longer match. Version 1 had no separate light-logic criterion and no fixed order for the composition checks. Version 2 also added a winner check: a batch winner is the best of what was made, not proof that it fits, so the skill runs the brief-fit checks on the top image before reporting it, moves down the ranking when it is DOES NOT FIT, and says plainly when nothing fits. The full text is rubric.md in the repository.

Calibration numbers

aw.py cache agreement measures the judge rather than the images, from one or more work folders and the person's labelled pairs (cache label, see Commands). After the four-image batch on the Verdict contract page and three labels, one of them recorded with --shown-first a:

python aw.py cache agreement batch/artwright

prints:

{
 "pairs_both_orders": 6,
 "both_orders_agreement": 0.833,
 "target": 0.8,
 "labelled_pairs_judged": 3,
 "accuracy_vs_labels": 0.667,
 "requery_disagreement": null,
 "labels_with_order": 1,
 "labels_first_shown_share": 1.0,
 "note": "recompute whenever the judge model changes; below 0.8 both-orders agreement, show ties to a human",
 "ok": true
}

Field	Meaning
both_orders_agreement	share of pairs judged in both orders whose two first picks match. The target is 0.8; below it the skill says so in the verdict and leans on ties
accuracy_vs_labels	share of labelled pairs, among those judged, where the resolved verdict matches the person's pick (a tie must match a tie label)
requery_disagreement	share of re-queries of the same order that changed the pick: the noise floor. null with no re-queries
labels_with_order	labels that recorded which image was shown first (cache label --shown-first) and are not ties (0.3.0)
labels_first_shown_share	among those, the share where the person picked the image shown first: the person's own side bias. null with none

Here one of the three labels was b on the pair the judge left as a tie, so accuracy is 0.667. The one label with an order picked the side shown first, so the share is 1.0; it means something only over many labels. The curator skill decides which image goes first by a coin flip and records it, so a share far from 0.5 shows the person's position bias, the same bias both orders check for in the judge. Recompute whenever the judge model changes.

The cache

The judge never judges an unchanged file twice with the same rubric: cache get before, cache put after. The key is the image's SHA-256, the rubric id and version (default 3 since 0.5.0, 2 in 0.3.0 and 0.4.x), the judge model and the prompt version, plus the temperature and seed when they are given (0.3.0), so changing any of them is a fresh judgement (examples on Commands).

What the numbers mean

aw.py stats describes; it never scores. Value, contrast, colourfulness, edges, symmetry, balance, a focal cell and a fractal dimension are evidence the critique can cite with the command that made them. The same holds for light, gamut, accents, depth and comp: their flags point at a region to look at and are never a reason on their own (Light, colour and composition). The JSON says it itself: "descriptions, not a beauty score; thirds_distance is never a reason". The skills never give the golden ratio as a reason, and the rule of thirds is a description, not a score.

No aw.py command says whether an image is AI-made, who made it or what it is worth, and the skills decline those questions from pixels alone (Skills).

What clean pixel art means

aw.py pixel passes an image as clean_pixel_art only when all of these hold, and lists each one that fails in problems:

Rule	Fails as
one integer scale with 100% lattice alignment	off-grid pixels (lattice alignment N%)
every block a single colour	blocks not uniform (N%)
no blend pixels (0.1% or less)	transitional (blend) pixels N%
colours within --budget; without a budget, at most 256	N colours over a budget of M, or N colours: not pixel art, or AI output never reduced to a palette
binary alpha	N semi-transparent pixels

The skill's rule of thumb: a raw AI "pixel art" image typically shows thousands of colours, a wandering block size and about 2% alignment, and needs a grid fixer before any critique. The resampled test sprite on Commands fails four of the five rules. Orphans, jaggies, outline colours and palette match are reported but do not decide the pass; the critique weighs them. So are the 0.7.0 craft numbers (banding, corner doubles, interior jaggies, pillow shading, edge_lost_on_bg, tile repetition): none of them has a pass line yet.

Two rules soften at a stated scale. With --scale 1 (0.4.1), blend pixels are reported but not failed, since a native sprite's one-pixel shading steps are not blends; and runs that touch a fully transparent pixel are left out of transitional_pixels (0.4.2).