0.3.0

Upgrade steps

• Rename release to vintage in sources.toml. The old key is now refused, with a
  message naming the new one (see Removed below).
• A source with headers needs an https url now (see Security below). Switch an
  http:// URL to https://, which most APIs serve.
• Run build again for JSON sources: numbers are now written as they appear in the file,
  so such an output can change (see Changed below).

Added

• --manifest PATH on every command, and manifest= on main, read a sources.toml
  other than the one in the current directory, such as one a project ships inside its
  package. The data still goes to data/ in the current directory.
• A weekly CI job runs the Save Page Now contract test against the live archive, so the
  stand-in the other tests use is checked against the real service. Run it locally with
  uv run pytest -m live.
• Two docs pages: How sourceledger works, on the stages and the
  files each one writes, and Use sourceledger safely. The README
  shows how to verify a release with gh attestation verify or pypi-attestations.

Changed

• sourceledger is for Linux and macOS (POSIX), and its package metadata now says so. On
  Windows it runs, but fetch takes no lock against a second run.
• Reading a table with thousands of columns, or a sources.toml with thousands of sources,
  is faster: the check for repeated names no longer grows with the square of their number.
• JSON sources keep numbers as written: 1.10 stays 1.10 and 1e5 stays 1e5, at the
  top level and inside nested values. Before, they were read as numbers and written back
  as 1.1 and 100000.0. An output built from such a file changes on the next build.
• build reads a CSV source and writes its output one row at a time, with the default
  parser, so memory use no longer grows with the size of the file. For a 161 MiB source it
  peaked at 2.5 GiB before, and it is about a quarter faster. The output and
  datapackage.json are the same, byte for byte. JSON, Excel, gzipped CSV and a custom
  parse are still read whole.
• validate reads each output one row at a time, and checks the raw file's hash without
  reading it into memory, so memory use no longer grows with the size of the data. For a
  161 MiB source it peaked at 2.5 GiB before. build on the command line counts the rows
  it prints the same way.
• fetch checks a raw file's hash before a conditional request without reading the whole
  file into memory, and reads the fetch log once per run. Before, each source with
  archive = true that came back with the same bytes read the whole log again.

Removed

• Breaking: the old key release is no longer read as vintage. A manifest that
  still uses it fails to load and names the new key, rather than being kept as an unknown
  key and leaving the vintage "not stated".

Fixed

• Bytes in the fetch log that are not UTF-8 no longer stop fetch, status and revisions
  with a bare decoding error. They read as U+FFFD, so a line they break is reported by
  its number, as any other damaged line is.
• archive = true now waits up to 120 s for Save Page Now, up from 15 s. The archive
  answers only once the snapshot is taken, 49 s for a GitHub page in one test, so a fetch
  whose snapshot took longer than 15 s recorded an empty archive_url although the
  archive had the page.
• A fetch log whose last line is a whole record without its final newline, as an editor
  or a merge can leave it, keeps that record. Before, the next fetch took it for a line
  cut off by an interrupted fetch and deleted it from the history. Appending now reads
  only the end of the log, not all of it.
• A source's last record in the fetch log, or its .fetch.json, with a missing field or a
  field of the wrong type (a SHA-256 that is not 64 hex digits, a size that is not a whole
  number), is now reported as not a fetch record. Before, fetch, build, validate and revisions
  could stop with a traceback, and fetch skipped the sources after it. fetch now
  downloads such a source again, and validate lists it as a problem.
• revisions fails when the current log does not exist, as it already did for the
  earlier copy. Run from the wrong directory, it used to report no revisions.
• fetch and build called from Python with a sha256 pin in upper case, bypassing
  load_manifest, now treat it the same way: both compare it as written.
• A cut-off or damaged gzip, a file that is not an Excel workbook, and JSON nested too
  deeply to read now fail build and validate with one line that names the source,
  not a traceback.
• An Excel sheet with a value right of its header is refused, as a CSV row with more
  fields than its header is. Before, it became a column with an empty name, and two such
  values gave a misleading "appear more than once" error. Empty header cells at the end
  of the first row are no longer read as columns.

Security

• Breaking: headers on a source whose url is not https is a manifest error, since
  the API key would cross the network in clear text.
• A redirect that changes the scheme, such as https to http on the same host, now drops
  the headers too, as a redirect to another host already did. Before, the key went out
  in clear text after the downgrade.
• A redirect from a public host is now checked against the address the connection
  actually reached, not a separate lookup before it. A name that resolved to a public
  address for the check and to an internal one for the connection could get past it.
  Through a proxy the check is skipped, since the proxy resolves the name.
• The 300 s limit on one source's fetch now holds against a server that sends a few
  bytes at a time. Each read used to wait for a full 1 MiB, so a slow trickle kept a
  fetch going far past the limit. It now stops within one 30 s socket timeout of it.
• fetch no longer follows a symlink at data/raw/.lock or data/fetches.jsonl. A
  checkout could plant one, and taking the lock emptied the file it pointed at. The lock
  file is never truncated now, and a symlinked log fails the fetch.
• An Excel workbook is refused when it unzips past max_bytes, as a gzipped file
  already was. A small workbook could otherwise expand to more memory than the machine
  has.