Skip to content

swath 0.4.0

Latest

Choose a tag to compare

@github-actions github-actions released this 17 Nov 09:10
· 41 commits to main since this release
7e75c77

swath 0.4.0

User-facing changes

swath lists object stores and filesystems in parallel: it shows where the space went, and saves a sorted
Parquet inventory you can query without listing again. It reads listing metadata only and never reads, changes
or deletes what it lists. 0.4.0 is a new tool under the same name: one static binary, a new command line and
a new dataset format. Start here; the
documentation is also at swath.varve.io.

  • Sources. Amazon S3 general-purpose buckets and S3-compatible stores (s3://), Google Cloud Storage
    (gs://), Azure Blob Storage flat namespaces (az://), and local or mounted filesystems (a path).
    --all-versions lists S3 versions and delete markers, GCS generations, and Azure versions, snapshots and
    soft-deleted blobs (Sources).
  • Where the space went. swath du SOURCE on a terminal draws the largest prefixes and objects at any
    depth, each with its size, count, share of the total and newest date. -d N, a pipe or --format print
    totals per prefix as a table, TSV or JSONL
    (Total a source).
  • An inventory you can query. swath list SOURCE -o DIR writes a key-sorted Parquet dataset whose
    manifest.json commits its parts and records how complete the listing was. Query it with DuckDB, pandas,
    Spark or any Parquet reader; swath show DIR summarizes it, and swath du --from-dataset DIR totals it
    without listing again. -o a bucket prefix with --work-dir DIR writes the dataset to S3, GCS or Azure
    Blob. swath list also prints rows as a table, TSV or JSONL, or writes them to a .tsv or .jsonl file
    (optionally .gz or .zst)
    (Query an inventory,
    Output).
  • Resume and refresh. An object-store dataset run that is stopped or interrupted keeps its published
    parts and its journal, and swath resume DIR continues it. --overwrite keeps the previous dataset
    readable until the new one commits, and --seed-from PREVIOUS divides a relisting along the previous
    dataset's keys
    (Stop, resume and refresh).
  • The run report. A run accounts for itself in one JSON document (--report FILE, and run.json beside
    a dataset): the build, the command, the outcome and exit code, the timings, the requests made and their
    estimated cost. progress.json holds a running dataset's live counts, and --log-format json writes
    stderr as events (Run report).
  • Install. One archive per target (linux-amd64, linux-arm64, darwin-amd64, darwin-arm64,
    windows-amd64) holding static swath and swath-replay binaries, with SHA256SUMS, keyless cosign
    signatures, an SPDX SBOM and build provenance; images ghcr.io/varveio/swath:0.4.0 and
    ghcr.io/varveio/swath-replay:0.4.0 for linux/amd64 and linux/arm64. go install is not supported in
    0.4.0: build from a checkout (Install).

Moving from 0.3.2

  • The command. The commands are swath list, swath du, swath show and swath resume, and options
    are renamed or removed, without aliases. An option only one provider has carries its prefix
    (--s3-region, --s3-profile). Most old options name what replaced them when you give them
    (--force-path-style is --s3-path-style)
    (Unknown flag). Seven
    are refused as a plain unknown flag: --text-part-size and --text-writers have no replacement (text rows
    go to one file, -o NAME.tsv), nor has --writeback-size; --object-listing-queue-size is
    --sort-buffer, in bytes; --engine-toggle is swath diag list's options; --metrics-endpoint and
    --no-metrics are under "Metrics" below.
  • Sorted output. A dataset written with -o DIR is always sorted Parquet, so 0.3.2's
    swath list SOURCE --format parquet --sort -o out/ is swath list SOURCE -o out/. --sort is now
    swath du's and takes a value (name or size): the old spelling reads -o as that value and fails with
    swath: unexpected argument "out/" (exit 2). --format parquet is refused, naming -o DIR.
  • Filters. --include and --exclude take globs; 0.3.2's regular expressions go to --include-regex
    and --exclude-regex. Common regular-expression spellings given as a glob ('\.parquet$') are refused
    with exit 2, naming the regex option; others are valid globs that match something else: --include parquet
    keeps only entries named parquet, at any depth, not every key containing parquet as in 0.3.2. So move
    every regular expression to --include-regex or --exclude-regex
    (Filters). They match the key below
    SOURCE, where 0.3.2 matched the whole key, so rewrite an anchored pattern relative to SOURCE: for
    s3://b/logs/, 0.3.2's --exclude '^logs/private/' is --exclude-regex '^private/'. In sizes KB, MB,
    GB and TB are decimal (0.3.2 read them as binary); K, KiB and the like are binary.
  • Times. --modified-since and --modified-until read a date or a time without Z or an offset as
    local time (TZ), where 0.3.2 read it as UTC. To keep 0.3.2's meaning, give the UTC time with seconds and
    Z: 2026-01-01T00:00:00Z for 2026-01-01
    (Filters).
  • Durations are a number with a unit (90s, 1h30m, 2d). An ISO-8601 duration (PT30S), none,
    off and upper-case units are refused with exit 2; 0 turns off --max-duration, --stall-timeout and
    --part-interval.
  • Parts. A sorted dataset's parts were about 1 GiB in 0.3. They are 64 MiB by default now, and a part
    open for 30 seconds with rows closes although not full, so part boundaries depend on timing. For parts
    about as large as before, give --part-size 1GiB --part-interval 0.
  • --report FILE no longer creates its directory: a name in a directory that does not exist is refused
    with exit 2, before anything is listed.
  • The dataset. A dataset is a SeekParquet dataset with the swath profile: manifest.json, written last,
    commits it (there is no _SUCCESS), run.json is its run report, and its columns are named per source
    under a schema id such as swath/s3-objects/v1
    (The swath dataset format). 0.4.0 neither
    reads a 0.3 dataset nor resumes a 0.3 run; --overwrite replaces the directory an interrupted 0.3 run
    left. List again with 0.4.0, and update queries to the new columns. symlink.txt, the part list for
    Athena and Trino, is no longer written: the manifest's parts lists the parts
    (Query an inventory).
  • Metrics. 0.4.0 exports no OTLP metrics. --metrics-endpoint and --no-metrics are refused as unknown
    options, but SWATH_OTLP_ENDPOINT, SWATH_OTLP_INTERVAL, OTEL_RESOURCE_ATTRIBUTES and
    OTEL_SERVICE_NAME are not read, so metrics configured only in the environment stop without an error. A
    run's counts are in its report, progress.json and --log-format json events.
  • S3 client setup. The region comes from --s3-region, AWS_REGION, AWS_DEFAULT_REGION or the
    profile, else us-east-1, never from EC2 instance metadata: on EC2 without one of them, a bucket in
    another region fails with exit 1 naming the option to give. An endpoint_url in ~/.aws/config is not
    read (give --endpoint-url or AWS_ENDPOINT_URL_S3), and requests to https:// endpoints, AWS's
    included, go through HTTPS_PROXY, not HTTP_PROXY
    (S3).
  • The image ghcr.io/varveio/swath is distroless, without a shell: its entrypoint is /opt/swath/swath,
    it runs as 65532:65532, and it cannot run --s3-bearer-token-command or a profile's
    credential_process (Container images).

The 0.3 documentation stays at the v0.3.2 tag.

Evidence

The release pipeline publishes a tag only after these gates pass
(Releasing swath); the
results are in the tag's ci.yml and release.yml runs.

  • ci.yml on main at that exact commit (go-full-gate): go vet and the race tests of the three Go
    modules, the default tier on macOS, Windows and Linux ARM64, cross-builds of the five targets, SeekParquet
    conformance, the replay fixture tier, and both images built for two architectures and smoked offline
    (testing).
  • release.yml on the tag: go vet and the whole race suite of the main module unsharded, the five targets
    built once with each linux/amd64 binary's --version checked, and a smoke of the pushed image digests
    before they are tagged: an anonymous listing of a public S3 prefix into a dataset, read back with DuckDB
    from the manifest's part list, resumed as a no-op, and served by the replay image. The release stays a
    draft until its own signatures and attestations verify against the published assets.
  • The license gate that generates THIRD_PARTY_NOTICES fails the build on a linked module whose license is
    not recognized or not allowed.
  • Measured runs, each with its build, date, machine and command, are in
    measurements.md. In one
    live listing with --workers 1024 swath counted 1.09 billion S3 objects in 177 seconds, and in another,
    also at 1,024 workers, wrote them as a sorted inventory in 846 seconds, on one 32-vCPU VM; at the default 64
    workers neither finished within 40 minutes. 35 production listings of public buckets, 50.6 billion objects
    in all, completed. None was taken on this tag.
  • What has run against the real services, all on Linux VMs and recorded in measurements.md: anonymous
    listings of public S3 buckets, the billion-object runs included; the 35 production listings, of public S3
    buckets with 0.4.0-rc.2, each writing its dataset to a GCS bucket with default credentials; anonymous
    listings of a public Azure Blob container; and a private GCS bucket of 5.1 million objects, listed by an
    earlier build. New in 0.4.0 and tested only against emulators, fakes and CI runners: S3 credential refresh
    (#577); Azure Blob with a SAS, Shared Key or Entra ID
    (#346); GCS with a key file; datasets written to S3 and
    Azure Blob buckets; and journaled runs and swath resume on Windows, which run on CI's Windows runners.

Limits and known issues

  • Not listed: Azure hierarchical namespaces (Data Lake Storage Gen2), S3 directory buckets, access points
    and Outposts, and GCS soft-deleted objects.

  • Resume: only object-store runs that write a dataset resume. A filesystem listing, text output and
    swath du are run again.

  • Windows: dataset runs and swath resume work as on Linux and macOS, on an output whose filesystem
    syncs directories (checked when the run starts). Power loss is safe on local NTFS; on ReFS or an SMB share
    a crash or kill is recovered, but a power loss may lose the latest part names. An --overwrite blocked by
    a file another program holds open fails naming that cause and is rolled back: the previous dataset stays
    readable and swath resume DIR installs the replacement later. If the rollback is blocked too, the
    directory holds no dataset until swath resume DIR finishes the replacement
    (Known limits).

  • S3-compatible stores: swath trusts a store's empty last page; a store that returns one too early makes
    a listing miss keys (#265). Check a first run against a new
    store with an independent count
    (S3-compatible stores).

  • macOS Gatekeeper. The binaries are not signed with an Apple Developer ID or notarized
    (#420). A browser download is refused until you clear its
    quarantine attribute, after verifying and extracting the archive (darwin-amd64 on Intel):

    xattr -d com.apple.quarantine swath-0.4.0-darwin-arm64/swath swath-0.4.0-darwin-arm64/swath-replay
  • Archives are built for the five targets above only; images are linux only.

  • swath publishes the complete result of the live listing it performed. It is not a point-in-time snapshot
    of a bucket that changes during the run.

  • The rest, and what has not been measured, is in
    Known limits.

What's Changed

  • Release v0.3.2 by @sagiba in #212
  • site: field guide model shows owner-side splits; name the output and latency-valve floors by @sagiba in #207
  • site: homepage captions say what the run did before the provenance caveats by @sagiba in #208
  • Link the project story from the README by @sagiba in #213
  • Add GCS and Azure listing replay over one shared fixture by @sagiba in #230
  • Cut per-key and first-touch cost from sorted delimiter listing by @sagiba in #236
  • Add a bounded live provider conformance campaign with GCS and Azure evidence by @sagiba in #243
  • ci: bump the actions group across 1 directory with 11 updates by @dependabot[bot] in #246
  • Serve Azure listings as the live service does: BOM, UTF-16 order, startFrom with a delimiter by @sagiba in #244
  • Inject the --inject-latency profile on the GCS and Azure routes too by @sagiba in #250
  • Archive replay receipts for the merged Azure fidelity work and promote the matching rows by @sagiba in #249
  • Go port of swath: S3, GCS and local filesystem providers on a shared range engine, sorted dataset format 2, durable resume by @sagiba in #248
  • go: recovery refuses a linked data/, numeric suspect ordinals, bounded empty-page liveness, resume and reader docs corrected by @sagiba in #251
  • go: sparse filters release their pages, unsafe cursors decline one split, --report cannot replace managed output, stricter flags, honest resource fields by @sagiba in #252
  • go/order: a failed range stops the ordered stream at its node by @sagiba in #254
  • go: range engine campaign: balanced splits, seed descent, admission, memory, instrumentation by @sagiba in #255
  • Go port of swath-replay: sorted-Parquet reader, S3, GCS and Azure routes, differential driver, image and CI by @sagiba in #253
  • Go replay follow-ups: S3 Owner and fetch-owner as live S3, GCS time_created, fuzz targets, the live matrix recorded and replayed by @sagiba in #257
  • docs: field investigations: no machine-local paths in the replay server command by @sagiba in #259
  • go: phase 2: text sink, delta spill, warm start, red-team fixes, CLI by @sagiba in #258
  • docs: schema notes before 0.4 (GCS last_modified/etag meaning, checksum, roadmap) by @sagiba in #277
  • go: spill encoders keep 1.5 MiB, not 8.6; the pinning test holds the least of three peaks by @sagiba in #278
  • go: durable spill: sealed ranges survive a crash and a resume replays them instead of relisting by @sagiba in #282
  • ci: a 30-minute test timeout for the Go race run by @sagiba in #283
  • go/listing: make TestCancellationLeavesNothing deterministic on one core by @sagiba in #276
  • ci: on-demand Go stress workflow by @sagiba in #290
  • ci: skip the Java tiers when only the Go port changed by @sagiba in #285
  • go: fix two flaky tests (TestDurableUnknownCodec, TestPipelinedSmallResponsesStayWithinTheCeiling) by @sagiba in #291
  • go: the segment-parallel drain tail, a shared codec pool and priced writer admission by @sagiba in #281
  • ci: split the go job — lint, sharded race, non-race, build by @sagiba in #294
  • go: --all-versions lists S3 versions and delete markers and GCS generations by @sagiba in #280
  • go: shape suite, scalefake service behaviours, nightly shape baseline by @sagiba in #289
  • go/listing: TestDurableRepeatedRecovery requires a snapshot that replays (#292) by @sagiba in #296
  • ci: run the Go default tier on macOS, Windows and linux/arm64 by @sagiba in #286
  • go/keyrange: TestSeedFirstPageLargeFirstTile asserts the seed's tiles, not a timing ratio by @sagiba in #300
  • ci: Java image public-bucket smoke is dispatch-only; offline smoke on main merges by @sagiba in #302
  • docs: testing map — every test method, where it lives, how to run it, when CI runs it by @sagiba in #303
  • go: switch the dataset format to SeekParquet (manifest, footer stamps, exact key bounds, replay reader) by @sagiba in #299
  • go/order: TestDistinctCountsOnlyHandedRows waits on no gauge (#306) by @sagiba in #307
  • go: byte identity holds per encoder path; TestFooterOnlyChange pins both (#305) by @sagiba in #308
  • go/order: the durable-cache tests run on Windows; their directory listings were the failure (#301) by @sagiba in #310
  • go/listing: versions x segments over durable files; K inside a certificate at the claim kill (#297) by @sagiba in #312
  • ci: Go reliability items from #304 (nightly condition, tidy check, download retries, per-SHA cache, stress inputs) by @sagiba in #311
  • go: a failed directory listing is an error, not an empty directory (#310 review) by @sagiba in #314
  • go/objfile: open an object-store object as a file with one version pinned by @sagiba in #316
  • go/order: TestDurableLedgerFailureEvicts holds the run until the append is torn and a certificate evicted (#309) by @sagiba in #317
  • go/order: the segment budget tests hold the frontier until every other range is done (#313) by @sagiba in #315
  • go/objfile: bump go-jose/v4 for three Dependabot alerts by @sagiba in #327
  • go/s3: TestAttemptsAreRequestEconomics answers after a clock tick (#319) by @sagiba in #320
  • go/order: TestSegmentsTrailerFailureClosesNoPart damages the trailer when the segments are tried (#321) by @sagiba in #323
  • go/order: TestAheadWaitWakes holds the frontier until every other range is done (#325) by @sagiba in #329
  • go/order: TestFetchWaitWakes holds the frontier until every other range is done by @sagiba in #331
  • go: consume the seekparquet module (writer, manifest, inspection); SHA-256 digests; replay meters from ReadStats by @sagiba in #318
  • go/order: TestSpillDiskBudget holds the frontier until the run signals, not a slow page (#326) by @sagiba in #330
  • go/keyrange: TestPriorStale holds the grown tile until every other range is done (#332) by @sagiba in #334
  • go/cmd/swath: TestSegmentsReplayDurableFiles kills at a seal after the first record is synced (#333) by @sagiba in #336
  • go/reader: the swath reader module: shared schemas, metadata and typed rows (plan step 8a) by @sagiba in #335
  • go/reader: Open, CheckSchema, AcceptWholeSource and Stream (plan step 8b) by @sagiba in #340
  • go/reader: Totals and Aggregates over seekparquet's selection plans (plan step 8d) by @sagiba in #341
  • go/s3: TestThrottleStormLowersConcurrency ends the clean window at the limiter's first increase (#342) by @sagiba in #343
  • go: test infrastructure for the Azure Blob provider (account information route, long names, azblobtest corpus and server, UTF-16 comparator) by @sagiba in #324
  • go/azblob: Azure Blob list client on azidentity, objhttp and tokensource by @sagiba in #328
  • go/azblob: Azure Blob provider (az://, admission, schema, staged refusal of names at U+E000 and above) by @sagiba in #337
  • go/azblob: list names at U+E000 and above in byte order through the order adapter by @sagiba in #344
  • go: SDK-backed listing clients for S3, GCS and Azure Blob behind the swath_sdkclient build tag (experiment) by @sagiba in #348
  • go/docs: split internals/providers.md into a hub, a page per provider, the list clients and the range engine by @sagiba in #349
  • go/s3: read the web-identity token file's presence before the lookup, not after a failed one by @sagiba in #350
  • go/replay: mark the legacy fixture readers (java-era, format 2) as transitional, deleted after the Go/Java replay comparison by @sagiba in #354
  • ci: shared Go module cache, per-job build caches, bounded retention by @sagiba in #355
  • go: delete swath-dataset format 2 (readers, replay profile, docs) by @sagiba in #357
  • ci: split the rest race shard, macOS on main and nightly only, no cross-OS runners for Java-only changes by @sagiba in #356
  • docs: CI architecture and the reasoning behind it by @sagiba in #358
  • ci/setup-go: download the module graph on a copy of go.mod and go.sum by @sagiba in #360
  • go/reader: typed List, ValidateDigests and seekparquet aliases (varve imports reader and objfile only) by @sagiba in #359
  • go: swath du — per-prefix totals, like du, for buckets and local trees by @sagiba in #347
  • ci: BuildKit caches are written from main only by @sagiba in #361
  • docs/measurements: CI caches, race shards and cross-OS legs after #355 and #356 by @sagiba in #366
  • go: swath/local-entries/v1, extensions.swath.version and layered show output by @sagiba in #367
  • go/localfs: huge directories and cold counts (parallel sort, pointer-free entries, count processors) by @sagiba in #339
  • docs: go/docs/dataset.md owns the swath dataset profile by @sagiba in #369
  • go/localfs: a count does not sort; NFS stays on READDIRPLUS across large directories by @sagiba in #351
  • go/objfile: S3 objects at version "null" are identified by ETag by @sagiba in #370
  • go/localfs: on macOS, stat each entry relative to its directory and fill every stat column by @sagiba in #368
  • go/reader: List pairs the object at a common prefix's exact key onto the prefix entry by @sagiba in #371
  • ci: module-cache download copies go.mod/go.sum in the repository layout (fixes main after #371) by @sagiba in #375
  • go/localfs: a count does not sort on the portable engine either (Windows, macOS) by @sagiba in #372
  • go/reader: AdmitWhole/ReopenWhole with Identity, IsRefusal, and Dataset.Page by @sagiba in #373
  • go/objfile: durable pins (Pin/Restore/Snapshot), ParseLocation, objfile/remote.Open, served-offset range guarantee by @sagiba in #374
  • go/reader: typed local entries, nullable local totals, Azure exact totals, parallel Scan; localfs records --no-dirs as a coverage filter by @sagiba in #377
  • go/objfile: remote.Open's GCS client uses the new auth library; ReopenWhole counts note by @sagiba in #379
  • go: one SDK-backed listing client per provider; explicit anonymity; one normalized failure value; swath owns only the swath-specific parts by @sagiba in #383
  • docs: describe Go listing and inventory across storage backends by @sagiba in #378
  • Go port: a provider is one directory plus one import (issue 338) by @sagiba in #386
  • Go port: provider audit and docs pass by @sagiba in #387
  • go/localfs: default --workers is 4 per CPU, at most 64 by @sagiba in #389
  • go/order: clear the popped stack slot so expanded ranges can be collected by @sagiba in #390
  • Go port: no credential-bearing endpoint or location reaches an error or a record by @sagiba in #393
  • Go port: remove dead code and redundant tests in go/internal/source by @sagiba in #395
  • go/localfs: a count reads large directories in slabs, handing chunks to other workers by @sagiba in #399
  • ci: publish images to GHCR only from releases by @sagiba in #398
  • go/localfs: warn when a listing skips a directory too large to sort by @sagiba in #401
  • release: ship the Go binaries and images by @sagiba in #403
  • go/s3: pin the fast timestamp parser to the standard one (test, fuzz, benchmark) by @sagiba in #400
  • ci: shorten PR feedback and bound shared cache growth by @sagiba in #402
  • release: prepare v0.4.0-rc.1 by @sagiba in #407
  • ci: parallelize isolated crash tests and narrow Java inputs by @sagiba in #410
  • ci: Dependabot for Go modules and Dockerfile.go; neutral OSV exception label by @sagiba in #409
  • swath-core: make FinalizationPipelinePropertyTest's roll precondition deterministic by @sagiba in #411
  • go/cli: GNU-style parsing, provider-neutral text rows, prefixed provider flags, new help by @sagiba in #394
  • go/keyrange: remove the floors split policy and --floors by @sagiba in #415
  • ci: scope Go PR validation and require full release qualification by @sagiba in #414
  • release: version check expects 'swath X.Y.Z (commit)'; swath-replay prints the same form by @sagiba in #418
  • release: name the Go tools swath/swath-replay; run images as distroless nonroot by @sagiba in #419
  • go/docs: user docs by task, one reference page per command, internals moved out by @sagiba in #417
  • release: darwin/amd64, SBOM root name, cosign v3 + macOS quarantine docs; deflake part-interval test by @sagiba in #423
  • go/reader: open and read only; remove admission and digest validation (#397) by @sagiba in #424
  • replay: build the SeekParquet list boundary index at open by @sagiba in #425
  • deps: bump the go-objfile group in /go/objfile with 7 updates by @dependabot[bot] in #413
  • ci: bump the actions group across 1 directory with 6 updates by @dependabot[bot] in #405
  • ci: setup-go v7 in the composite action; Dependabot scans .github/actions by @sagiba in #427
  • deps: Go 1.27.3 toolchain (go.mod toolchain lines and Dockerfile.go together) by @sagiba in #428
  • go/listing: the default row budget follows memory; count spill frames and opens by @sagiba in #426
  • reader: baseline Scan and Stream benchmarks and an allocation guard by @sagiba in #433
  • go/cli: --row-budget is --sort-buffer by @sagiba in #429
  • deps: azblob 1.8.3 and Go dependency refresh; Azure wire snapshot by @sagiba in #434
  • deps: Go 1.27.4 and the release dependency refresh by @sagiba in #438
  • reader: borrowed column-batch scans with field selection by @sagiba in #441
  • ci: bump the actions group across 1 directory with 2 updates by @dependabot[bot] in #432
  • go/azblob: list in the Arrow format, XML when the service answers XML by @sagiba in #440
  • ci: DuckDB 2.0.0, OSV-Scanner 2.7.1, Syft 1.57.x by @sagiba in #442
  • go/replay: fill the window cache from SeekParquet's borrowed batches by @sagiba in #443
  • ci: pin Buildx and the BuildKit image in release; correct two dependency doc statements by @sagiba in #444
  • go/clients: wire snapshots for S3 and GCS, Arrow column types through the decoder by @sagiba in #446
  • reader: Stream in one ordered pass of RangeBatches by @sagiba in #445
  • go: one-line credential refusals; refuse a SAS naming a request parameter by @sagiba in #447
  • go/azblob: 1,000-entry default page by @sagiba in #448
  • go/azblob: pin x-ms-version 2026-06-06, which Azurite answers by @sagiba in #451
  • CHANGELOG: 2026-11-10 changes by @sagiba in #450
  • reader: keep TestStreamMatchesPages inside the race job's time by @sagiba in #454
  • reader: scan shards from a queue, eight per worker by @sagiba in #455
  • reader: Totals in shards over borrowed batches; certify key order across shard boundaries by @sagiba in #456
  • replay: compare the Go and Java servers on converted fixtures (#352) by @sagiba in #452
  • go/s3: cut the list scanner's and the density step's CPU per object by @sagiba in #458
  • go/cli: split SWATH_CPUPROFILE and SWATH_MEMPROFILE at the end of listing by @sagiba in #463
  • order: reuse spill read handles across frame and trailer reads; count them and frame sizes by @sagiba in #464
  • go: write sorted datasets directly to GCS (#266, phase 1) by @sagiba in #465
  • go: write sorted datasets directly to S3 (#266) by @sagiba in #466
  • go: write sorted datasets directly to Azure Blob (#266) by @sagiba in #467
  • listing: admit encoders against their own memory allowance by @sagiba in #468
  • parquetout: compress pages at zstd SpeedFastest by @sagiba in #471
  • parquetout: build each column's null value once and reuse a repeated string's value by @sagiba in #472
  • go/azblob: read once per page in the byte-order adapter (#449) by @sagiba in #469
  • go/azblob: admit a flat account whose Data Lake host refuses it for soft delete (#457) by @sagiba in #459
  • objfile: clarify upload contracts and streamline library docs by @sagiba in #473
  • go/azblob: --all-versions (versions, snapshots, soft-deleted blobs) and the blob properties every listing returns by @sagiba in #460
  • go/keyrange, azblob: a counted prior lists in whole pages of 5,000 (#449) by @sagiba in #470
  • go: read through the redesigned SeekParquet Go API; ListRequest.ReadAhead by @sagiba in #474
  • objfile: batched range reads, verified sized opens, whole-object reads by @sagiba in #475
  • go/cmd/swath: the failed-after-parts test waits for a saved part before it fails the listing by @sagiba in #476
  • docs: the Go docs as swath's docs — concepts, generated CLI reference checks, link check, library examples by @sagiba in #477
  • objfile: request observer, cache mode, identity-keyed byte cache, sized connection pool by @sagiba in #478
  • go: read SeekParquet datasets through objfile (spsource adapter, request budgets, zero-read count) by @sagiba in #479
  • release: notes for 0.4.0-rc.2 by @sagiba in #480
  • go/reader: browse page counts from row spans and budgeted value-only sums by @sagiba in #481
  • docs: release-candidate install page, Windows limits, interim dataset validation; readbudget name check by @sagiba in #486
  • go/azblob: zero SDK timestamps are missing times; Totals docs (#487) by @sagiba in #488
  • Report committed counts and object bytes on resume; not_found for missing buckets (#482) by @varve-steno in #489
  • go: follow SeekParquet's API cleanup; callers keep bulk reads out of objfile's cache (#484) by @sagiba in #490
  • docs: where not_found applies (source buckets; not anonymous Azure or output buckets) by @sagiba in #491
  • go/cmd/swath: refuse a bare -o NAME.parquet; NAME.parquet/ selects a directory (#142) by @sagiba in #493
  • go/reader: keep only swath's meaning; scanning, aggregation and pages move to SeekParquet by @sagiba in #495
  • docs: macOS quarantine note in the release notes; what swath trusts on S3-compatible stores by @sagiba in #496
  • go/internal/listing: an Azure Blob row in the crash matrix (#241) by @sagiba in #497
  • go/localfs: prune subtrees covered by recursive excludes by @sagiba in #504
  • go: a local listing records its filesystem's space in the manifest by @sagiba in #503
  • go/replay: fixtures are generated by Go; go-fixtures needs no JDK (#353) by @sagiba in #507
  • go: tests for scenarios the Java suite covered and Go did not by @sagiba in #508
  • Retire the Java implementation: delete the tree, its build and its CI (#73) by @sagiba in #510
  • go: a bucket run resumes from a copied work directory (test); Spot guidance and stale resume notes in the docs (#511) by @sagiba in #515
  • Move the Go module to the repository root (#293) by @sagiba in #529
  • docs: spill at billion scale, and what a spill cap too small does by @sagiba in #534
  • Hands-on fixes: text output excludes itself, recovery advice that works, partial-sample resume by @sagiba in #531
  • go: swath du --from-dataset DIR [PREFIX] totals a saved dataset by @sagiba in #532
  • Go run trace (swath diag list --trace) and the explainer on it by @sagiba in #536
  • CLI: exit 3 for incomplete output, unreadable naming, signal death, usage errors, du hard links by @sagiba in #537
  • site: VitePress site from docs/, with a new home and Start pages by @sagiba in #533
  • First-run fixes: stop early when nothing is reachable, dump to a file, tmpfs warning at first spill by @sagiba in #538
  • ci: a docs tier for pull requests that change only pages under docs/ by @sagiba in #539
  • site: development-docs notice, rc.2 differences, install for every platform, reviewer fixes by @sagiba in #543
  • README: a one-minute page that hands off to swath.varve.io by @sagiba in #542
  • docs: one page per source; remove the 0.3 migration page by @sagiba in #547
  • deps: Go dependency refresh across the three modules; one Dependabot entry for all of them by @sagiba in #546
  • go: a spill-capped run no longer dies with 'spill: corrupt frame' (a refused range's trailer at offset 0) by @sagiba in #551
  • go: instrument the spill-cap stall (wait sites, frontier split eligibility, refusal time, writer during listing) (#535) by @sagiba in #549
  • test: TestPhaseProfilesRun no longer fails when a short listing ends before the profile's start is stamped by @sagiba in #552
  • go: say so when the spill cap is too small (error event after sustained refusal; --spill-wait-timeout exits resumably) (#535) by @sagiba in #553
  • go: durable spill certificate limits scale with the run (#513) by @sagiba in #554
  • deps: seekparquet to 8ceebf4 (read-ahead and forward routing fixes) by @sagiba in #550
  • go: a diag knob for the segment tail's worker count (--segment-workers) by @sagiba in #556
  • deps: seekparquet to 41e4319 (listing read-ahead sized by the entries its pages yield) by @sagiba in #559
  • go: durable spill certificate slots are bounded by their metadata, not by spill size (#513) by @sagiba in #560
  • go: a resume that reuses certificates writes its tail in parallel segments (#288) by @sagiba in #561
  • go: a seeded run lists front to back inside a window at the output frontier (#535) by @sagiba in #555
  • docs: one pass over the guide and internals after the spill, seeded-window and resume changes by @sagiba in #562
  • go: every sorted dataset carries its cut points for the next listing (a side file beside the manifest) by @sagiba in #557
  • test: TestSegmentsReplayDurableFiles accepts a certificate the drainer reaches before the cut rule by @sagiba in #564
  • deps: bump the go group across 3 directories with 1 update by @dependabot[bot] in #548
  • go: format versions start at 1 (report, journal, durable spill); a report schema change no longer needs a bump by @sagiba in #565
  • go: untrusted names are escaped on every human-readable stderr line (#521) by @sagiba in #572
  • docs: first-run commands, the benchmark sentence and copy fixes from the site review by @sagiba in #570
  • docs: command options render as grouped, wrapping lists generated from the flag definitions by @sagiba in #574
  • go: the frontier window shrinks to what the spill cap can serve instead of crawling on forced admissions (#535) by @sagiba in #563
  • go: a regular expression given to a glob filter is refused; a 0.3 run directory is recognised (#526, #527) by @sagiba in #571
  • objfile/remote: one Options for opening a location, with OpenWith (closes #384; step 1 of #567) by @sagiba in #569
  • go: the default spill cap follows the disk, so a disk grown under a live run is used by @sagiba in #566
  • go/s3: credentials refresh before they expire, and an expired-token answer refreshes and retries once (#523) by @sagiba in #573
  • site: the home page leads with a real run on a public bucket and its replay by @sagiba in #575
  • test: TestPlanStaleManyWorkersSmallCap allows a slow runner's refusals by @sagiba in #581
  • test: the fake credential source's expiry advances with each fetch by @sagiba in #583
  • deps: seekparquet at its release go/v0.2.0 by @sagiba in #580
  • go: swath du rows carry both sizes under the dataset's names; the list table names its zone by @sagiba in #582
  • go: the swath du table shows each prefix's newest time by @sagiba in #591
  • go: main green again: an abort race in part publication, five flaky tests by @sagiba in #590
  • go: list and du command-line grammar: -h, -o names, filters, SOURCE forms, one error shape by @sagiba in #585
  • report, progress, log: one vocabulary for the machine surfaces by @sagiba in #587
  • release: tag reader and objfile with every version by @sagiba in #592
  • go: complete means nothing is missing; one verdict; --overwrite keeps the previous dataset until commit by @sagiba in #589
  • du --from-dataset DIR [SOURCE]: the positional is the dataset's source or a path below it by @sagiba in #593
  • go: bare swath du on a terminal draws the largest prefixes at any depth by @sagiba in #588
  • go: what a person sees: the closing lines state the result, swath show reads in order, one table grammar by @sagiba in #595
  • site: a field guide to the Go engine, and the home page's run report by @sagiba in #597
  • du --from-dataset on a file, quoted-dot globs, cuts.txt after an overwrite resume; drop --segment-workers by @sagiba in #598
  • spill_pressure: with the default cap, say to grow the disk by @sagiba in #600
  • site: the field guide's visuals, rebuilt for the Go engine by @sagiba in #599
  • go: the du view fills the terminal, reads size, count, share, bar, date, path, and marks what it misses by @sagiba in #596
  • docs: split the user guide by the question asked by @sagiba in #601
  • go: the du view spends no line under a floor and opens the largest prefixes by @sagiba in #603
  • localfs: list directories deeper than PATH_MAX by @sagiba in #602
  • site: home page leads with the swath du view by @sagiba in #594
  • test: the CI flakes for 0.4.0 made deterministic; Azure admission waits out a busy account by @sagiba in #607
  • Windows: journaled runs and swath resume by @sagiba in #605
  • test: keyrange's stale-prior and first-page tests gate on the engine's decisions, not wall time by @sagiba in #609
  • ci: Windows go-xos in three legs along the race shards by @sagiba in #608
  • test: Windows flakes from journaled runs; GCS retries a connection reset on Windows by @sagiba in #611
  • dataset: a refused --overwrite never restores the old manifest without its data/ by @sagiba in #610
  • go: an --overwrite installation syncs the commit record apart from the data; du rows give an unreadable directory null; NOTICE files; documentation corrections by @sagiba in #612
  • test: journaled tests that must complete get a long watchdog; the nightly Windows job runs in legs by @sagiba in #613
  • release: the 0.4.0 copy (release notes, changelog, no release-candidate wording) by @sagiba in #604

New Contributors

Full Changelog: v0.3.2...v0.4.0