Skip to content

backdraft 0.11.0

Choose a tag to compare

@spencerbraun spencerbraun released this 11 Sep 13:27

A week on sizing up a source before spending on it — what it will be called, how much of it a read costs, and whether there is anything in it.

  • backdraft ingest <source> --dry-run says what each source would be called before that is permanent: the slug and media type it would take, whether it is already ingested (under which slug, and whether withdrawn), and the numbered slug a taken name would land on. Nothing is fetched, written or minted. A slug is fixed once a token carries it, so this is how to see the default before --slug has to overrule it.
  • A page read has a budget. One backdraft read shows 12,000 characters — 200 rows of a sheet — unless --limit says otherwise, so a scraped article no longer spends a context window in a single call with nothing to say so. A read that stops short closes with the total and the exact command that continues it. The cut never lands inside a chunk, so every token still names text you were shown whole, and ordinary pages print exactly what they did.
  • A web page's table of contents is its chunks: backdraft read <slug> on a one-page source lists p1.c1, p1.c2, … with what each opens with, under the name the page gave itself in its <title> — the map for finding where the article starts past the site's navigation. The artifact names a fetched page by that title too, wherever the record carries one.
  • A source that came back a shell says so wherever it is listed. backdraft read's document list, that source's table of contents and backdraft ls all mark it little text: N chars, so a registry ingested in another session still tells the writer which sources are login walls and scans.
  • Every hint the gate prints — search's, show's, the table of contents', session show's and a short read's continuation — now carries a --session you typed, so following one no longer reads into a different ledger from the one you bind against.
  • ingest --dry-run refuses a malformed --config the way ingest does, rather than predicting a run that cannot start.

1440 tests.