Skip to content

Releases: QuietFlare/clew

0.7.0

Choose a tag to compare

@ramanse ramanse released this 15 Oct 14:52

Added

  • Two ways to try Clew with nothing installed. A Codespace opens with
    Nextflow, Docker and two recorded nf-core/demo runs already in the
    image, and docker run -p 8770:8770 ghcr.io/quietflare/clew serves the
    page with the same runs. docs/try.md walks through both.
    In a Codespace, triage and the agent go through a proxy QuietFlare runs
    unless you set your own keys.
  • Drift and Reclaim tabs on the page. Drift compares two runs of the
    picked record, roots with their causes and the rest by cause. Reclaim
    weighs the run's work directories from the record and the disk, with
    the proof and the causes that withheld the rest, and deletes nothing.
    "Explain in words" has a model read the plan on the page and say what
    it reports, never whether it matters.
  • Browse on the page uploads a run from your own machine: a launch
    folder, a .lineage folder, or a zip or tgz of either, in every
    browser.
  • The Nextflow lineage extractor reads the copies publishDir made, with
    their checksums, into the graph's published files. On a store recorded
    with NXF_CACHE_MODE=DEEP, clew reclaim proves a work directory
    redundant from the record alone, with no clew digest pass.
  • Each command ends with the next one: --list-runs with extract,
    extract with impact, impact with seal, seal with verify.
  • The agreed design lives in spec/: a module diagram, the rules and a
    module map, reviewed on every pull request.
  • examples/weblog_receiver.py collects the events the nf-weblog
    plugin posts into the JSONL file clew extract nextflow-work reads, so
    a run made without the lineage store still gets a graph from its work
    tree. docs/try.md walks through it: receiver up, then
    the run, then the extractor.
  • The plan table on the page shows each task's hash beside its process
    and tag, so three rows of one process read apart, as they do in the
    terminal.

Changed

  • clew demo is gone. The recorded runs and docs/try.md replace it.

Fixed

  • from clew.provider.nextflow import NextflowAdapter, SheetKind, the
    import the providers guide shows, now works; the package exported only
    NextflowAdapter.

  • A .history entry Nextflow did not write, such as a macOS ._* file
    left by a tarball, is skipped with a warning instead of crashing the
    extractor.

  • A content-mode checksum label in a Nextflow lineage store is believed
    only when the run record names Nextflow 26.09.0-edge or later. Earlier
    versions wrote a standard-mode value under whatever label
    NXF_CACHE_MODE gave (nextflow-io/nextflow#7579), and the extractor
    took the label at its word. The coverage note names the version that
    wrote the store and what to do.

  • The store's sha256 mode is Nextflow's own 128-bit hash over the
    content, not a SHA-256 of the file. Its digests are now labelled
    nextflow-sha256: beside nextflow-deep:, so they never read as
    comparable with a clew digest value.

  • The sources doc said process.cache = 'deep' makes the store record
    content checksums. It does not; the environment variable does, on the
    fixed versions.

0.6.1

Choose a tag to compare

@ramanse ramanse released this 09 Oct 07:51

Release 0.6.1, as published to PyPI. The attached clew-demo-runs.tgz holds two recorded nf-core/demo runs with a Nextflow lineage store carrying content checksums, for docs/try.md and the Codespace.

0.4.0

Choose a tag to compare

@ramanse ramanse released this 07 Sep 09:33

Added

  • clew drift: where two runs of the same workflow part ways, and why.
    Tasks paired by name, outputs compared by content digest, the cause of
    each root read from the record.
  • --runs on digest, reclaim and drift: read a run straight from a
    .lineage store, a horus-lineage root, or a directory of graphs. The
    digests clew digest computes go to a sidecar under <runs>/.clew/.

0.3.0

Choose a tag to compare

@ramanse ramanse released this 06 Sep 18:18

Changed

  • Modules are grouped by layer: clew.graph, clew.ledger,
    clew.extract, clew.questions, clew.views. clew.core is gone. The public API is
    clew.load_graph, blast_radius, classify, parse_trigger and
    resolve_trigger. Commands are unchanged.
  • clew stitch joins runs by content digest. --results is gone.

Added

  • clew reclaim: which work directories are safe to delete, with the
    proof for each. Nothing is removed without --apply and a receipt.
  • Content digests in the graph, digest fields of the form
    <algorithm>:<value>, recorded by the Horus, Snakemake and Nextflow
    store extractors where the engine has them.
  • clew digest: hash a run's files once, for runs the engine recorded
    without content digests.
  • DNAnexus support: clew extract-dnanexus builds a graph from an
    analysis, over the API or from saved describe output. Edges join on
    file ID. Optional price and duration_s per task.
  • Latch support: clew extract-latch builds a graph from an execution,
    over the API or from saved records. Edges join on latch:// path.
  • Cromwell support: clew extract-cromwell builds a graph from workflow
    metadata, from a file or a server. Scatters become one node per shard,
    subworkflows are flattened, edges join on path. Verified on Cromwell 92.
  • Snakemake support: clew extract-snakemake reads the .snakemake
    metadata store, file or SQLite backend, and carries Snakemake's own
    input checksums onto the edges. Verified on Snakemake 9.26.
  • A graph contract in core, contract_violations, run by the tests over
    every shipped and fixture graph.

Fixed

  • The runtime version now matches the package version. 0.2.0 reported
    itself as 0.1.1.

v0.2.0

Choose a tag to compare

@ramanse ramanse released this 05 Sep 12:40

Added

  • Horus support: clew extract-horus reads a horus-lineage run directory
    and joins edges by content digest, so a graph closes across machines.
  • A trigger registry and the generic --trigger kind:value form. An
    unknown kind is read as a label key, so engines that record labels answer
    label queries with no adapter.
  • --html on clew impact writes a self-contained impact report.
  • The lineage store extractor follows resumed runs: tasks cached from an
    earlier session in the same chain stay in the graph, replaced tasks are
    marked rather than dropped, and an unknown store version is refused.
  • A comparison of what each lineage source can prove, in the docs.
  • Output sizes in the work-directory extractor.

Changed

  • --donor is now --subject. The old flag still works.
  • Generic graph queries moved from the domain adapters into core.
  • Shipped demo graphs are anonymised.
  • The README is a front page. The design documentation moved to docs/.

Fixed

  • External inputs that nf-prov crates record as URLs are kept rather than
    dropped.
  • The work-directory extractor documents that forwarded files lose their
    producer, and the lineage store is preferred for that reason.

v0.1.1

Choose a tag to compare

@ramanse ramanse released this 05 Sep 12:40

Added

  • Python 3.9 support, which is what macOS ships.

Fixed

  • The work-directory extractor refuses copy-staged work directories instead
    of returning an empty graph that would report every task as clean.

v0.1.0

Choose a tag to compare

@ramanse ramanse released this 05 Sep 12:40

First release on PyPI as clew-lineage.

  • Lineage extraction from Nextflow's native lineage store, nf-prov RO-Crates
    and work-directory symlinks, all emitting one graph format.
  • Blast radius for subject, container and external-input triggers, with
    remove and distrust modes.
  • Contribution classes with fail-closed defaults and a versioned,
    content-hashed remediation policy.
  • Storage checked against a work root, never assumed.
  • Cross-run stitching at published files.
  • An append-only, hash-chained event log on Postgres with two clocks.
  • Sealed evidence bundles that replay offline, witnessed back into the log
    and countersigned with OpenSSH keys.
  • A CI gate that fails closed on unknown subjects.
  • A self-contained dashboard and a read-only MCP server for auditors.