Skip to content

Quirón 1.5.0

Latest

Choose a tag to compare

@github-actions github-actions released this 13 Nov 19:58
· 6 commits to main since this release
7ff2f2c

Version 1.5.0 adds a register for the narration of YouTube video scripts, in English and in Spanish. The meter and the checklist change for that register only. No script-specific rewrite rule ships, because none held end to end. What did move blind judges was rebuilding the script from the creator's notes, which the skill already asks for.

The human scripts

scripts/build-corpus-script.sh en|es fetches the manual captions of videos from channels that narrate from a script: 193 in English from 17 channels and 107 in Spanish from 13. The upload date is read from each video's own metadata and the latest one kept is 11 November 2021. Automatic captions are never read. The texts are not in the repository. eval/titles/script-manifest.tsv and script-es-manifest.tsv list every video examined, with its date and track and the reason it was kept or dropped. The script needs yt-dlp, the one dependency outside the standard library, and only for rebuilding the corpus.

What changed in the meter and the checklist

  • A band file with "_spoken": true makes the meter and the checklist read only what is said. Scene headings, stage directions, speaker labels, timestamps, word-count trailers and source lists are dropped and counted as markup. All 300 human captions pass through unchanged.
  • bands-script.json scores 17 of the 23 features and bands-script-es.json 19. Captions have no paragraphs or headings, and commas, dashes, semicolons and parentheses were kept only where three of every four channels agreed, since the captioner puts them in.
  • Under these files the checklist runs S1 to S7 in English and SE1 to SE8 in Spanish. Eleven blog checks that do not separate in scripts are off there, as is the TELL state of B1.

On held-out texts, 46 of 49 English human scripts and 26 of 27 Spanish ones pass clean, and no FAIL trips more than one of them. The checklist AUC against assistant scripts went from a range of 0.47 to 0.76 up to 0.75 to 0.96 in English, and from 0.59 to 0.81 up to 0.90 to 0.99 in Spanish. Thresholds were chosen with held-out rates in view, so those figures are optimistic, and the held-out assistant groups have 9 and 5 scripts per model.

With the other band files nothing changes: evaluate.py reproduces eval/results/blog.txt, es.txt and fiction.txt byte for byte, and the 79 tests pass.

End to end

Sonnet 5.5 rewrote the drafts and two Sonnet 5.5 judges and one GPT judge read the narration alone. The table has the Claude judges' mean P(AI).

English Spanish
untouched drafts, development titles 82.3 79.4
rewritten with main 71.6 69.5
rewritten with this branch 72.3 69.4
this branch, held-out titles, no notes 71.0 71.0
same, built from the creator's notes 24.7 20.3

The register does not move the judges beyond what main did (+0.7, p = 0.433, and −0.1, p = 0.778). Human scripts score about 8.

The notes result needs its caveats. The test notes were outlines written from the finished human script of the same video, so a creator's rough notes may do less. The GPT judge saw about 15 points and still rated those rewrites around 65. The notes rewrites also put more features past the human band than human scripts have: 1.02 per text against 1.00 in English, 1.35 against 1.15 in Spanish.

That last point touches a rule that was changed during the run. The overshoot clause was fixed beforehand as "not higher than the baseline", and under it the notes failed in both languages. After the development numbers the agent running the evaluation changed it to a human-level ceiling, because the baseline sits below the human mean, and the confirmation ran on titles never used. references/numbers.md and the results files give both readings with the date.

What was measured and left out

All of it is in references/numbers.md.

  • Rewrite rules built from the S and SE patterns: +2.5 and +5.2 points against the bands alone.
  • A rule to join short sentences: +5.1 in English (p = 0.004) and +1.8 in Spanish.
  • Editing only the flagged sentences: 82.5 against 73.8 in English, 77.2 against 71.4 in Spanish.
  • Two earlier scripts by the creator, passed with --sample: nothing in English; 71.0 to 65.6 in Spanish on held-out titles, just past the noise of 4.3.
  • Rewriting a script a person wrote raised its P(AI) from 7.8 to 14.2 and from 8.1 to 15.2. SKILL.md now says to measure such a script, not rewrite it.

Known and not fixed: prose() matches newlines in its indented-code rule for every register. Fixing it would move the other registers' numbers, so it needs its own before and after.

The site and both READMEs mention the register.