Release v0.8.4 · tamnd/papers-reader

The release that found out the publish gate was not running.

Publishing

papers publish runs the hard audit on every batch. It only ran under --id, so a run that named one paper went through the gate and a run that named none staged every root whole and went straight past it. The help has said all along that a paper a hard rule refuses is not published, and for a batch of fourteen papers with twelve of them refused that was simply untrue. A batch that names no papers now reads the changed papers off the working tree, audits them, drops the ones a rule refuses with the rule printed, and ships the rest.
This is why the corpus repository was not getting content. The pipeline was pushing whole trees, the gate was not looking, and the batches that did get through were the ones a person ran by hand.

A page number in a running head is no longer part of the section. The Paxos paper prints its title and the folio as one two line paragraph, which the whole paragraph match could not see, so three English sections kept a page number that rule T10 refuses and no amount of re-running could remove it. The match reads lines now. A remainder under ten words goes with the number, and a Markdown table row is left alone, because a row of dashes and pipes is not a folio.
-prune sweeps a translation whose English is gone whether or not anything split. The sweep was inside the splitter, so the one paper that needed it, the one whose extracted pages are missing and which therefore splits to nothing, was the one paper it never ran on. The GPT-3 paper had a Vietnamese references section with no English beside it causing T04, T07 and G04 at once.
-redo acts on rule L19 as well as on the hard rules.

Audit rule G04 counts a section number a paper used twice, the way the assigner already did. The assigner numbers a repeated key from two, s1 then s1-2; the rule computed the bare key and called the difference an error. Jacobson numbers Slow-start 1 after numbering three problems 1, 2 and 3, Metcalfe restarts at 1 in the appendix, and Paxos has two references sections. Five hard findings that tags assign could never clear, because the assigner was right and the rule was wrong. The counting lives in tags.Keys and both sides call it.

A file of nameless bitmaps has no text layer to check an extraction against, so the acceptance rules that compare against one stand down instead of refusing every page.
A figure too small to measure still records its page fraction, so the cap that keeps whole pages out of the corpus has a number to work with.

scripts/papers-full.sh is in the repository. It is the script that takes one paper from the PDF to a merged pull request before starting the next one, and it had been living in a home directory on one host. Nothing in it names a host. CI runs shellcheck over it.

No prompt changed, so nothing already translated is stale for this release.

translate: -redo acts on L19 as well as on the hard rules by @tamnd in #187
extract: a file of nameless bitmaps has no layer to check against by @tamnd in #188
split: drop bare page numbers and clear up after a re-split by @tamnd in #189
figures: a tiny figure still records a fraction by @tamnd in #190
publish: run the hard audit on every batch, and take the page number out of a running head by @tamnd in #191
tags: count a section number a paper used twice in the rule too by @tamnd in #192
split: sweep a translation with no English whether or not anything split by @tamnd in #193
scripts: put the autonomous pipeline in the repository by @tamnd in #194

Full Changelog: v0.8.3...v0.8.4

translate: -redo acts on L19 as well as on the hard rules by @tamnd in #187
extract: a file of nameless bitmaps has no layer to check against by @tamnd in #188
split: drop bare page numbers and clear up after a re-split by @tamnd in #189
figures: a tiny figure still records a fraction by @tamnd in #190
publish: run the hard audit on every batch, and take the page number out of a running head by @tamnd in #191
tags: count a section number a paper used twice in the rule too by @tamnd in #192
split: sweep a translation with no English whether or not anything split by @tamnd in #193
scripts: put the autonomous pipeline in the repository by @tamnd in #194

Full Changelog: v0.8.3...v0.8.4