Latest-file-per-job validation + whole-job-snapshot uploads.
Validators and miners should both pull the new version.
PRs
data-universe: #851
What changed
What changed
-
S3 validation moves to an auditor model. Starting with this release, macrocosmos validator (UID 89) is the designated auditor — it runs S3 validation locally and publishes per-miner scores to data-universe-api. All other validators fetch those scores and apply them. Cuts duplicate scraper/listing load across the fleet. Additional auditor validators may be added on request — reach out if you want to participate.
-
Validator scores only the latest file per job_id. Older files in the same job folder are ignored. Stale files will be reaped server-side.
Miner uploader rewritten to whole-job snapshots. Each cycle now writes one parquet per job (capped at max_rows) instead of incremental chunks. -
S3 credibility / boosts / effective_size reset (scorer migration v14→v15). Old scores are not comparable to the new model.
What you do -
Validators: pull the new release and restart. You'll see State migration v14 -> v15 once on startup, then either latest-only filter dropped N stale files (UID 89) or Result fetched from data-universe-api (everyone else).