Skip to content

da-validator v0.2.0

Choose a tag to compare

@flop-github-release-publisher flop-github-release-publisher released this 18 Nov 06:49
21278a5

Added

  • First versioned da-validator image for linux/amd64.
  • submit --inflight <1-4> (default 2) uploads that many chunks at once and reads and encodes
    the next one meanwhile. Each provider still gets one shard PUT at a time.
  • submit --provider-fail-limit <K> (default 8) stops sending shards to a provider after K
    refused PUTs in a row for the rest of the run, unless fewer than 3 providers would remain.
    Its shards stay unconfirmed and the custody check after the push still decides.
  • Opt-in source pull (#4263). serve --self-ingest
    accepts a publisher-signed source hint at PUT /blob/{c}/source. It then builds this
    validator's shards of an assigned blob from the hint's mirrors with HTTP Range reads. Each
    chunk is checked against its manifest leaf before its shard is stored, and a refused chunk
    tries the next mirror. Chunks no mirror serves are left to peer re-replication.
    --self-ingest-hosts (default huggingface.co,hf.co) bounds the hosts it may contact, and
    --self-ingest-concurrency (default 2) the chunks fetched at once.
  • submit --source-files <table> --source-mirror <url> offers that hint first, waits while
    providers self-ingest (--source-stall-secs, default 600), then pushes only what they lack.
  • A custody worker submits DaRegistry::confirm_custody once this validator's assigned index
    of a custody-latched blob is custody-pinned. It is inert while the chain latches no blob.

Changed

  • submit logs why a provider refuses a shard (its HTTP status or error) when it starts
    refusing, and bounds each request (connect 10 s, 150 s in all, past the validator's own
    60 s admission plus 60 s body deadline).

  • A blob may have up to 65,536 chunks (4 TiB at the unchanged 64 MiB chunk size), up from
    16,384 (1 TiB). The manifest body limit grows to 2 MiB to match. A validator on an older
    image rejects these manifests, so roll every validator before such a blob is registered.

Fixed

  • retrieve no longer drops shards whose body takes longer than 30 s to arrive. The client bounds
    connect (10 s) and each read (30 s) instead of the whole request, so --chunk spot checks pass
    behind the public DA edge.