Skip to content

v0.11.1

Latest

Choose a tag to compare

@jordan-gibbs jordan-gibbs released this 11 Sep 21:15
· 4 commits to main since this release
1367c9b

Seven fixes from the backlog sweep

  • Markdown links no longer mint stub notes (#93, reported by @ThiagoMafra-Integrare). [[Label]](https://…) — everywhere on GitHub READMEs, awesome-lists and Wikipedia-shaped pages — parsed as a wikilink, and repair --stub (default on, and mandated after every session) turned each one into a real note that then topped _most-linked and inflated PageRank. WIKI_LINK_RE now refuses a ]] immediately followed by (; the hygiene filter also rejects unsubstituted {…} placeholders and unbalanced brackets. Resolver-minted stubs are excluded from both rankers and have their stale centrality zeroed, so vaults polluted before this fix recover on the next graph rank. The stub marker is centralized so the minting sites and the filter cannot drift.
  • claims ingest finds the claims (#69, reported by @fduple). The default scan read only the legacy research/temp/, while every producer and consumer had moved to research/runs/<tag>/temp/ — so the claims table was empty in every real run and claims matrix / targets were dead on arrival. The default now scans both; --tag narrows to the run. A zero-file default scan prints a hint instead of a bare zero.
  • A Semantic Scholar rate limit is reported as a rate limit (#70, reported by @fduple). A 429 was folded into "No metadata found" — an affirmatively wrong diagnosis. 429s now retry three times with backoff (honouring Retry-After), and an exhausted budget surfaces as rate_limited in sources score output rather than missing. Nothing is cached for a throttle, so the next run retries. Optional S2_API_KEY support, scoped to semanticscholar.org hosts only — the fetch helper is shared with Unpaywall and Europe PMC, and an unscoped header would have shipped the key to them.
  • run resume names a skill that exists, chapters reach the manifest, step 8 keeps its preflight gaps (#100, from @darlingm's contract audit in #88). resume printed hyperresearch-2 for a skill named hyperresearch-2-width-sweep, and CLAUDE.md advertises resume as the recovery path — the slug now comes from the installer's own step table. set_chapter() had zero callers; it is gone, and the chapter-plan event the partition skill already emits now registers the chapter, so a resumed dissertation reports what is actually pending. Step 8's subagent wrote to the same file the preflight had just written; it gets its own output and a merge step, and gap records carry the id the dispatch already referenced.
  • Every pipeline constant is authored once (#101, from @darlingm's contract audit in #88). citation_density_min sat unread on the profile while two gates hardcoded their own number; the draft orchestrator hardcoded word targets the synthesizer templated, so on premier the two halves of a run were 60% apart; step 4 templated loci_analysts and then said "both" and wrote loci-a.json / loci-b.json. A new test asserts every Profile field is read somewhere outside profiles.py — and it found eight more dead fields (comparisons_tensions, source_tensions, tension_survey, tension_full_reads, corpus_critic_fetchers, citation_totals, utility_scoring, vault_check_interval_s), all now wired into the prose that had been hardcoding them. char_targets_no_word_boundary is authored per gear at the 3:1 ratio the profile already used, which moved full's argumentative range from 20000–25000 to 15000–30000 — the old span was 1.25× against a 2× word span.
  • Citation density is per 1000 effective words, not per 1000 characters (#76). The floor meant something roughly 3× stricter for CJK than for English. The denominator reuses #64's boundary detection: whitespace tokens where they exist, characters ÷ chars_per_word_no_word_boundary (3.0) where they don't. The floor is re-derived as citation_density_min = 9.0 per 1000 words, which is what 1.5 per 1000 characters worked out to for English prose — English verdicts are unchanged at the boundary; CJK reports are now held to the same standard per unit of content rather than a looser one.
  • hpr serve binds its port exclusively on Windows and reports the port it bound (#104). SO_REUSEADDR means "reclaim TIME_WAIT" on POSIX and "share a live port" on Winsock; a second bind on a running server's port succeeded on Windows. The server class now leaves it off there. run_server returns and prints the actually-bound port instead of http://127.0.0.1:0, so the tests stopped monkeypatching server_activate. CI gains a single windows-latest job on 3.11 — the platform this project is developed on — with the required-check names held stable.

Security review of the sweep

A review pass over the seven fixes above found six medium-severity problems — two of them pre-existing on main — and fixed each with a regression test.

  • A hostile page could hang hpr sync. WIKI_LINK_RE was quadratic on bracket floods: 20 KB of [[ took 4.8 s, 40 KB took 19 s, and 100 KB did not finish in two minutes — and the pattern runs on the whole body of every fetched page. Excluding [ from both character classes makes it linear (a 300 KB flood scans in under a millisecond). Pre-existing; surfaced because #93 touched the pattern.
  • The Semantic Scholar key followed cross-host redirects. httpx strips Authorization on a cross-origin hop but not x-api-key, so a 302 from semanticscholar.org would have carried the key to any host. Redirects are followed manually now, at most five, http(s) only, with headers recomputed from each hop's URL. Separately, the host check was a bare endswith on the raw netloc, which evilsemanticscholar.org and a userinfo trick both passed; it is now an exact-or-subdomain match on the parsed hostname.
  • claims ingest --tag ../../.. scanned outside the vault, because pathlib replaces the base on an absolute segment. A tag narrows the scan only when the resolved directory sits under research/runs/; files that resolve outside the vault root — symlinks — are skipped.
  • One malformed claims file aborted the whole ingest. Claims JSON is agent-written from fetched content: a dict where a string was expected raised out of .strip(), a list confidence raised out of sqlite, and 200 000 nested brackets hit RecursionError. Files are size-capped at 8 MB before reading, text fields are typed and bounded, and one bad claim becomes one error entry instead of a dead run.
  • A crafted chapter-plan event could corrupt the manifest. The fold introduced by #100 stringified any chapter value into a key and stored any title verbatim; a non-dict chapters crashed resume_position. Only a string or int id and a string title fold, both bounded; everything else stays in events.jsonl.
  • Retry-After was already clamped to sixty seconds — confirmed with a parametrized test over nan, inf, negatives and forty-digit values, all of which fall back to the 2 s / 4 s ladder.

Deferred to its own issue: hpr run init applies no validation to the vault tag at all, so ../../x scaffolds outside the vault. Same bug class as the claims fix, but it touches every run command.