v0.5.1

Changed

• The release SBOM lists the dependencies of the xlsx, parquet and release extras, at the versions the release was tested with. The 0.5.0 SBOM listed none, because the core has no dependencies.
• The release docs point to citesync 0.5.0, whose check compares a release folder's RO-Crate and Croissant files with CITATION.cff.

Fixed

• release copies keywords, publisher and contributor from codemeta.json into the RO-Crate and Croissant files; before, it dropped them. A publisher given as text, or as an object without @type, is written as an Organization, which both validators require. In the RO-Crate, a person listed twice is one entity.
• validate and release check that datapackage.json has the name set in sources.toml, so a name added after the last build no longer ships the old one. validate() and release() take name=. A script that gates on validate exits 1 until the next build.
• A pyproject.toml that is not UTF-8, or that cannot be read, now gives the one-line pyproject.toml: ...; fix it, or set name in sources.toml that a TOML error already gave, before build writes any output.
• release --check refuses --codemeta even when it names ./codemeta.json.
• The reference says to put name above the first [[source]], where TOML would otherwise make it that source's title.

The SBOM lists the dependency versions this release was tested with, not what an install will resolve.

v0.5.0

Upgrade note for scripts that call the CLI:

• Exit code 3 now means a stage could not run. It was 1.
• Errors on stderr read error: ... and help: ... instead of STAGE: ....
• validate and verify print their problems on stdout.
• --json reports have a schema_version and error objects.
• Options go after the stage name.
• revisions prints no revisions when no source changed. It used to print nothing.
• In Python, main() also returns 3 and 130. 0.4.0 returned only 0 or 1.

See options, output and exit codes.

Upgrade note for Python callers and manifests:

• read_output() now yields rows instead of returning a list. Wrap it in list() where the
  code indexes or counts them.
• sources.toml refuses top-level keys other than output_format, release_formats and
  [[source]].
• A dataset with a recorded file needs 0.5.0 to build.

Details are under Changed.

Added

• A top-level name in sources.toml names the data package in datapackage.json. Without it, build uses the [project] name in pyproject.toml, and only then the folder's name, so another clone, worktree or CI checkout writes the same file. load_package_name reads it, and build takes name=.
• release --codemeta PATH reads the dataset's CodeMeta file in place of ./codemeta.json. Its license is the dataset's licence, so a project whose data is not under the code's licence keeps a separate file.
• entry() takes argv, as main() does, so a project's tests can run its command without patching sys.argv.
• A custom reader may return any mappings as rows, with numbers or None as cells: the Parser type now says so.
• record, and record() in the library, take in a file that another tool such as pooch, DVC or your own script downloaded. It copies the file into data/raw/, checks the sha256 pin, and writes the same log line and fetch record as fetch, with a recorded_at time and optional request parameters, stored as text in name order. The download time is the one the other tool reports, or empty. datapackage.json and the HTML report show these outputs as recorded, not fetched, and status shows the record time when there is no download time. A recorded URL other than the source's loses its query, and a parameter named like a credential (api_key, apiKey, authKey, sessionid, sig) is refused, since both are published with the dataset. Recording the same bytes again leaves the fetch record as logged; from another URL or download time it adds a log line, and validate reports a datapackage.json built before such a record. For a source with archive = true, record keeps the snapshot a fetch logged for the same bytes and asks for no new one. A dataset with a recorded file needs this version to build: 0.4.0 refuses its fetch records.
• sourceledger STAGE --help lists only that stage's options. --dry-run on fetch, record and build says what they would do and writes nothing. fetch, record and verify accept --json. On a terminal, fetch shows which source it is fetching.
• Parquet outputs through sourceledger[parquet] and the project-wide output_format = "parquet" setting. Tables use zstd compression and an explicit string schema, with the same provenance columns and validation as CSV. build and validate accept output_format= in the library, and read_output reads either format. output_format inside a [[source]] table, where a line appended to sources.toml lands, is refused, as is release_formats.
  Writing and validation use batches of at most 10,000 rows through PyArrow, including full-column decoding to detect corruption.
• load_output_format() reads the project's output_format, for library callers to pass to build and validate. validate in the wrong format names the setting instead of asking you to delete the outputs.
• entry() runs the command for a project with its own reader, as sourceledger does: point the console script at a function that calls it with your parse and prog (recipe). After Ctrl-C it ends by SIGINT, so a shell loop around it stops.
• release OUT_DIR, and release() in the library, write the folder you publish from a build that validates: data/ with the outputs of the sources whose redistribute is true, a datapackage.json without the others, and a SHA256SUMS in the format of sha256sum -b. With release_formats = ["ro-crate", "croissant"] in sources.toml, it also writes ro-crate-metadata.json (RO-Crate 1.3) and croissant.json (Croissant 1.1), with the dataset's name, authors, licence, version and date from codemeta.json; a format not asked for is reported as skipped. load_release_formats() reads the key, for library callers to pass to release. release --check DIR checks a folder: every file against SHA256SUMS, and each metadata file with its official validator, rocrate-validator or mlcroissant, from the new sourceledger[release] extra. check_release() is the same check in the library. The report names each check that ran and its version, and a refused release lists its findings under problems with exit code 1, as --check does. Without the extra, a check that is needed exits 3 with the install line; a validator that stops without findings, such as offline, or runs past 10 minutes also exits 3. Above 100 files, the Zenodo limit, it warns. ReleaseError, RestrictedSourceError and ValidatorFailed are ValueErrors, and ValidatorMissing and ValidatorCouldNotRun OSErrors, so code that catches the base classes still works.
• A recipe follows a publisher's revision across two PUDL releases: revisions flags the new file and vintage names the release and DOI, then compare.py diffs the rows and carries the change into a direct operational CO2 estimate.

Changed

• Breaking: sources.toml refuses two ids that differ only in case, such as Pudl and pudl, since on Windows and macOS they name the same file. It also refuses a Windows device name as an id, such as CON, NUL or COM1, in any case. Rename the source, as con_data.
• Breaking: sources.toml refuses a url with a non-ASCII character outside its host. Percent-encode it as UTF-8, as d%C3%A4.csv for dä.csv. A non-ASCII host such as bücher.example is still accepted. Before, fetch failed with only 'ascii' codec can't encode character.
• Breaking: options go after the stage name. sourceledger --manifest x status now exits 2, where 0.4.0 took it. A stage that takes no operand refuses one: sourceledger status extra and build foo exit 2 instead of ignoring it. Option prefixes are no longer accepted. validate --j used to run as --json, so write the full option name.
• Breaking: sources.toml accepts only output_format, release_formats and [[source]] at the top level. A misspelt key such as [[sources]] is refused, with the intended key when one is close. Before, it read as a manifest with no sources, so build wrote an empty datapackage.json over a good one. A project that kept its own table there, for its reader's settings, moves it into its own file or into each [[source]], whose other keys stay allowed. output_format and release_formats are refused inside a [[source]].
• Breaking: read_output() is a generator that yields one row at a time, so reading a table of 1.5 million rows no longer holds them all in one list. It reads CSV row by row and Parquet in batches of at most 10,000 rows. Wrap the call in list() where the code indexes the rows, takes their length, or reads them twice. Every error, a missing file included, now raises on the first next(). An unfinished iterator keeps the file open, which on Windows blocks replacing the output.
• Windows is supported and tested in CI. 0.4.0 said it was not.
• build, validate and parse_csv take 26%, 38% and 25% less time than in 0.4.0 on a 200,000-row CSV table (best of 21 runs on one machine). load_manifest takes about 0.6 ms longer for 200 sources, for its new checks.
• The code behind the public names moved from sourceledger.ledger into manifest, ledger, fetcher, builder and report modules. Import from sourceledger itself: the submodules may change in any release, as the stability policy says.
• The CLI's stages, options, exit codes and --json reports are part of the public API in the stability policy.
• The HTML report puts the integrity result in a box at the top, lists the outputs with links under one heading, and names hashes and times in plain words ("SHA-256 of the downloaded file", "Served from"). It follows the reader's dark mode, prints without the in-page links, wraps long ids, and shows a recorded file's request parameters. A script that reads the report's HTML needs updating.
• The CLI help ends with a first run, the exit codes and a link to the docs, and --version prints the installed version, after the project's name when a project calls main(prog=...). Ctrl-C ends a stage with an error, the command to finish it and exit code 130 instead of a traceback, and the command ends by SIGINT so a shell loop around it stops (on Windows it exits 130); under --json the report still prints, with the sources fetched so far. init says what to do next, and a count of one reads "1 problem" or "1 row".
• An unexpected error, such as a project's parser raising KeyError, prints one error: line with its type and exits 3 instead of a traceback. A TOML syntax error names the manifest and the line to fix. A missing record, revisions or verify path says what to pass. A `...

v0.4.0

Move manually added files out of data/processed/ before running validate.

Added

• report creates a standalone HTML provenance report that recipients can read offline. It links outputs to their sources and lists integrity problems without including raw data.
• An assurance case explains the trust assumptions, safeguards and limits behind integrity and provenance checks.
• status, validate and revisions accept --json for scripts and CI. See JSON reports.
• Per-source timeout_s sets the fetch budget, including retries and waits. The default
  remains 300 seconds; see download limits and retries.
• format = "file" keeps files unchanged and validates their bytes, provenance and
  reuse metadata. Build describes them as non-tabular resources in datapackage.json;
  see How build works.

Changed

• The documentation website uses a navy and lime reading layout, responsive navigation and copyable code. A connected provenance diagram and revision comparison use an example checked against the pipeline. The full tutorial has its own page.

• Documentation separates the quick start from the manifest and command reference, custom readers and release verification.

• Breaking: validate reports unclaimed non-CSV files in data/processed/, alongside
  unclaimed CSV files. See validation.

Fixed

• JSON status reports render non-text status values as text instead of crashing on TOML dates and times.

• File validation checks each output hash once while preserving the checks of its provenance and package metadata.

• The HTML provenance report flags missing, duplicate or no-longer-implemented outputs instead of reporting successful checks for an incomplete package.

• The citation helper removes old Zenodo version, date and DOI overrides so each release can receive its own archive metadata. See release verification.

• A malformed resource name in datapackage.json produces validation problems instead of an uncaught error.

• Command options can appear between the stage and its path on Python 3.12, including
  revisions --json OLD_LOG. See the command reference.

Security

• Fetch follows redirects without reading their unused response bodies, avoiding an
  unbounded read. See Use sourceledger safely.

v0.3.1

Added

• The README shows how to rebuild a release from its tag and get the same wheel and sdist
  as on PyPI, byte for byte.

Changed

• init writes a #:schema line that names the schema at your installed version's tag,
  not the one on main. An editor then checks the manifest against the rules your version
  applies, not against keys a later release adds or refuses. Update the line when you
  upgrade. The schema's $id, and SchemaStore's entry, stay on main.
• Each GitHub release now carries the wheel, the sdist and their signed provenance
  bundle (sourceledger-<version>.sigstore.json) as assets. The maintainer pushes a tag,
  and the release workflow creates the release.

0.3.0

Upgrade steps

• Rename release to vintage in sources.toml. The old key is now refused, with a
  message naming the new one (see Removed below).
• A source with headers needs an https url now (see Security below). Switch an
  http:// URL to https://, which most APIs serve.
• Run build again for JSON sources: numbers are now written as they appear in the file,
  so such an output can change (see Changed below).

Added

• --manifest PATH on every command, and manifest= on main, read a sources.toml
  other than the one in the current directory, such as one a project ships inside its
  package. The data still goes to data/ in the current directory.
• A weekly CI job runs the Save Page Now contract test against the live archive, so the
  stand-in the other tests use is checked against the real service. Run it locally with
  uv run pytest -m live.
• Two docs pages: How sourceledger works, on the stages and the
  files each one writes, and Use sourceledger safely. The README
  shows how to verify a release with gh attestation verify or pypi-attestations.

Changed

• sourceledger is for Linux and macOS (POSIX), and its package metadata now says so. On
  Windows it runs, but fetch takes no lock against a second run.
• Reading a table with thousands of columns, or a sources.toml with thousands of sources,
  is faster: the check for repeated names no longer grows with the square of their number.
• JSON sources keep numbers as written: 1.10 stays 1.10 and 1e5 stays 1e5, at the
  top level and inside nested values. Before, they were read as numbers and written back
  as 1.1 and 100000.0. An output built from such a file changes on the next build.
• build reads a CSV source and writes its output one row at a time, with the default
  parser, so memory use no longer grows with the size of the file. For a 161 MiB source it
  peaked at 2.5 GiB before, and it is about a quarter faster. The output and
  datapackage.json are the same, byte for byte. JSON, Excel, gzipped CSV and a custom
  parse are still read whole.
• validate reads each output one row at a time, and checks the raw file's hash without
  reading it into memory, so memory use no longer grows with the size of the data. For a
  161 MiB source it peaked at 2.5 GiB before. build on the command line counts the rows
  it prints the same way.
• fetch checks a raw file's hash before a conditional request without reading the whole
  file into memory, and reads the fetch log once per run. Before, each source with
  archive = true that came back with the same bytes read the whole log again.

Removed

• Breaking: the old key release is no longer read as vintage. A manifest that
  still uses it fails to load and names the new key, rather than being kept as an unknown
  key and leaving the vintage "not stated".

Fixed

• Bytes in the fetch log that are not UTF-8 no longer stop fetch, status and revisions
  with a bare decoding error. They read as U+FFFD, so a line they break is reported by
  its number, as any other damaged line is.
• archive = true now waits up to 120 s for Save Page Now, up from 15 s. The archive
  answers only once the snapshot is taken, 49 s for a GitHub page in one test, so a fetch
  whose snapshot took longer than 15 s recorded an empty archive_url although the
  archive had the page.
• A fetch log whose last line is a whole record without its final newline, as an editor
  or a merge can leave it, keeps that record. Before, the next fetch took it for a line
  cut off by an interrupted fetch and deleted it from the history. Appending now reads
  only the end of the log, not all of it.
• A source's last record in the fetch log, or its .fetch.json, with a missing field or a
  field of the wrong type (a SHA-256 that is not 64 hex digits, a size that is not a whole
  number), is now reported as not a fetch record. Before, fetch, build, validate and revisions
  could stop with a traceback, and fetch skipped the sources after it. fetch now
  downloads such a source again, and validate lists it as a problem.
• revisions fails when the current log does not exist, as it already did for the
  earlier copy. Run from the wrong directory, it used to report no revisions.
• fetch and build called from Python with a sha256 pin in upper case, bypassing
  load_manifest, now treat it the same way: both compare it as written.
• A cut-off or damaged gzip, a file that is not an Excel workbook, and JSON nested too
  deeply to read now fail build and validate with one line that names the source,
  not a traceback.
• An Excel sheet with a value right of its header is refused, as a CSV row with more
  fields than its header is. Before, it became a column with an empty name, and two such
  values gave a misleading "appear more than once" error. Empty header cells at the end
  of the first row are no longer read as columns.

Security

• Breaking: headers on a source whose url is not https is a manifest error, since
  the API key would cross the network in clear text.
• A redirect that changes the scheme, such as https to http on the same host, now drops
  the headers too, as a redirect to another host already did. Before, the key went out
  in clear text after the downgrade.
• A redirect from a public host is now checked against the address the connection
  actually reached, not a separate lookup before it. A name that resolved to a public
  address for the check and to an internal one for the connection could get past it.
  Through a proxy the check is skipped, since the proxy resolves the name.
• The 300 s limit on one source's fetch now holds against a server that sends a few
  bytes at a time. Each read used to wait for a full 1 MiB, so a slow trickle kept a
  fetch going far past the limit. It now stops within one 30 s socket timeout of it.
• fetch no longer follows a symlink at data/raw/.lock or data/fetches.jsonl. A
  checkout could plant one, and taking the lock emptied the file it pointed at. The lock
  file is never truncated now, and a symlinked log fails the fetch.
• An Excel workbook is refused when it unzips past max_bytes, as a gzipped file
  already was. A small workbook could otherwise expand to more memory than the machine
  has.

v0.2.0

Upgrade steps:

• Rename any source id that uses a non-ASCII letter: ids must be ASCII letters, digits,
  _ and - (see the Breaking change below). Fetch again afterwards, since the id names
  the raw file and the output.

Added

• Python 3.15 is supported and tested in CI.
• JSON (an array of objects), Excel (the xlsx extra) and gzipped CSV and JSON sources,
  read by the new default reader parse_source from a source's format or its URL's
  extension. A gzipped file is refused when it expands past max_bytes.
• headers in sources.toml for API keys: each value names an environment variable
  ("env:NAME"), literal values are refused, the record keeps the header names only, and a
  redirect to another host drops them.
• build writes data/processed/datapackage.json, a Frictionless Data Package (v2) for
  the outputs: columns, size, SHA-256, source URL, licence and fetch record per output.
  It has no timestamp of its own, so the same records build the same bytes. validate
  reports a package that no longer matches the outputs.
• A JSON Schema for sources.toml, shipped in the package, for editors to check a manifest
  as it is typed. A test keeps it in step with what load_manifest accepts.
• sourceledger fetch marks each line new or unchanged, against the fetch log.
• sourceledger init writes a starter sources.toml with the schema line and a commented
  example source. It never overwrites an existing one.
• sourceledger status lists every source with its status, vintage, last fetch, checksum,
  licence and whether it may be republished, from the manifest and the log, offline.
• sourceledger attribution prints the implemented sources' reuse terms as a Markdown
  table, for a dataset's README or release notes.
• sourceledger verify DIR checks a downloaded dataset against its datapackage.json:
  every file it lists must exist inside DIR, symlinks included, with the size and
  SHA-256 it gives.
• --only ID for fetch, build and validate, and an only argument for the same
  functions, to work on some sources only. An unknown id is an error.
• status, attribution and verify are also functions in the public API.
• archive = true in sources.toml asks the Internet Archive's Save Page Now to snapshot
  a source's URL after a fetch that brings new bytes. The snapshot's address is kept in the
  fetch record and the log as archive_url, and kept when the same bytes are downloaded
  again. Each connect and read times out after 15 s,
  and a failure is a warning, never a failed fetch. Records written before this field
  still load.

Changed

• Breaking: source ids must be ASCII (letters, digits, _, -). They name files,
  which must mean the same file on every system. Before, any Unicode letter passed.

Fixed

• A line in data/fetches.jsonl that is JSON but not a fetch record is reported by line
  number instead of crashing fetch and revisions.
• A second fetch in the same project while one is running exits at once instead of
  interleaving writes with it (Linux and macOS).
• Downloads stream to disk in 1 MiB chunks, so a 200 MiB source no longer needs 200 MiB
  of memory, plus copies, to fetch.

Security

• CI gives the read-only GitHub token to the zizmor hook only, not to every hook.
• Each release file on PyPI has a signed GitHub build-provenance attestation:
  gh attestation verify <file> --repo simonvanlierde/sourceledger shows where it was built.
• CI installs a set prek version, 0.5.5, instead of the newest one on each run.

v0.1.0

chore(release): 0.1.0 (#4)

* chore(release): 0.1.0

* chore(release): version and date in the citation files