Repository navigation
swath 0.4.0
User-facing changes
swath lists object stores and filesystems in parallel: it shows where the space went, and saves a sorted
Parquet inventory you can query without listing again. It reads listing metadata only and never reads, changes
or deletes what it lists. 0.4.0 is a new tool under the same name: one static binary, a new command line and
a new dataset format. Start here; the
documentation is also at swath.varve.io.
- Sources. Amazon S3 general-purpose buckets and S3-compatible stores (
s3://), Google Cloud Storage
(gs://), Azure Blob Storage flat namespaces (az://), and local or mounted filesystems (a path).
--all-versionslists S3 versions and delete markers, GCS generations, and Azure versions, snapshots and
soft-deleted blobs (Sources). - Where the space went.
swath du SOURCEon a terminal draws the largest prefixes and objects at any
depth, each with its size, count, share of the total and newest date.-d N, a pipe or--formatprint
totals per prefix as a table, TSV or JSONL
(Total a source). - An inventory you can query.
swath list SOURCE -o DIRwrites a key-sorted Parquet dataset whose
manifest.jsoncommits its parts and records how complete the listing was. Query it with DuckDB, pandas,
Spark or any Parquet reader;swath show DIRsummarizes it, andswath du --from-dataset DIRtotals it
without listing again.-oa bucket prefix with--work-dir DIRwrites the dataset to S3, GCS or Azure
Blob.swath listalso prints rows as a table, TSV or JSONL, or writes them to a.tsvor.jsonlfile
(optionally.gzor.zst)
(Query an inventory,
Output). - Resume and refresh. An object-store dataset run that is stopped or interrupted keeps its published
parts and its journal, andswath resume DIRcontinues it.--overwritekeeps the previous dataset
readable until the new one commits, and--seed-from PREVIOUSdivides a relisting along the previous
dataset's keys
(Stop, resume and refresh). - The run report. A run accounts for itself in one JSON document (
--report FILE, andrun.jsonbeside
a dataset): the build, the command, the outcome and exit code, the timings, the requests made and their
estimated cost.progress.jsonholds a running dataset's live counts, and--log-format jsonwrites
stderr as events (Run report). - Install. One archive per target (
linux-amd64,linux-arm64,darwin-amd64,darwin-arm64,
windows-amd64) holding staticswathandswath-replaybinaries, withSHA256SUMS, keyless cosign
signatures, an SPDX SBOM and build provenance; imagesghcr.io/varveio/swath:0.4.0and
ghcr.io/varveio/swath-replay:0.4.0for linux/amd64 and linux/arm64.go installis not supported in
0.4.0: build from a checkout (Install).
Moving from 0.3.2
- The command. The commands are
swath list,swath du,swath showandswath resume, and options
are renamed or removed, without aliases. An option only one provider has carries its prefix
(--s3-region,--s3-profile). Most old options name what replaced them when you give them
(--force-path-styleis--s3-path-style)
(Unknown flag). Seven
are refused as a plain unknown flag:--text-part-sizeand--text-writershave no replacement (text rows
go to one file,-o NAME.tsv), nor has--writeback-size;--object-listing-queue-sizeis
--sort-buffer, in bytes;--engine-toggleisswath diag list's options;--metrics-endpointand
--no-metricsare under "Metrics" below. - Sorted output. A dataset written with
-o DIRis always sorted Parquet, so 0.3.2's
swath list SOURCE --format parquet --sort -o out/isswath list SOURCE -o out/.--sortis now
swath du's and takes a value (nameorsize): the old spelling reads-oas that value and fails with
swath: unexpected argument "out/"(exit 2).--format parquetis refused, naming-o DIR. - Filters.
--includeand--excludetake globs; 0.3.2's regular expressions go to--include-regex
and--exclude-regex. Common regular-expression spellings given as a glob ('\.parquet$') are refused
with exit 2, naming the regex option; others are valid globs that match something else:--include parquet
keeps only entries namedparquet, at any depth, not every key containingparquetas in 0.3.2. So move
every regular expression to--include-regexor--exclude-regex
(Filters). They match the key below
SOURCE, where 0.3.2 matched the whole key, so rewrite an anchored pattern relative to SOURCE: for
s3://b/logs/, 0.3.2's--exclude '^logs/private/'is--exclude-regex '^private/'. In sizesKB,MB,
GBandTBare decimal (0.3.2 read them as binary);K,KiBand the like are binary. - Times.
--modified-sinceand--modified-untilread a date or a time withoutZor an offset as
local time (TZ), where 0.3.2 read it as UTC. To keep 0.3.2's meaning, give the UTC time with seconds and
Z:2026-01-01T00:00:00Zfor2026-01-01
(Filters). - Durations are a number with a unit (
90s,1h30m,2d). An ISO-8601 duration (PT30S),none,
offand upper-case units are refused with exit 2;0turns off--max-duration,--stall-timeoutand
--part-interval. - Parts. A sorted dataset's parts were about 1 GiB in 0.3. They are 64 MiB by default now, and a part
open for 30 seconds with rows closes although not full, so part boundaries depend on timing. For parts
about as large as before, give--part-size 1GiB --part-interval 0. --report FILEno longer creates its directory: a name in a directory that does not exist is refused
with exit 2, before anything is listed.- The dataset. A dataset is a SeekParquet dataset with the swath profile:
manifest.json, written last,
commits it (there is no_SUCCESS),run.jsonis its run report, and its columns are named per source
under a schema id such asswath/s3-objects/v1
(The swath dataset format). 0.4.0 neither
reads a 0.3 dataset nor resumes a 0.3 run;--overwritereplaces the directory an interrupted 0.3 run
left. List again with 0.4.0, and update queries to the new columns.symlink.txt, the part list for
Athena and Trino, is no longer written: the manifest'spartslists the parts
(Query an inventory). - Metrics. 0.4.0 exports no OTLP metrics.
--metrics-endpointand--no-metricsare refused as unknown
options, butSWATH_OTLP_ENDPOINT,SWATH_OTLP_INTERVAL,OTEL_RESOURCE_ATTRIBUTESand
OTEL_SERVICE_NAMEare not read, so metrics configured only in the environment stop without an error. A
run's counts are in its report,progress.jsonand--log-format jsonevents. - S3 client setup. The region comes from
--s3-region,AWS_REGION,AWS_DEFAULT_REGIONor the
profile, elseus-east-1, never from EC2 instance metadata: on EC2 without one of them, a bucket in
another region fails with exit 1 naming the option to give. Anendpoint_urlin~/.aws/configis not
read (give--endpoint-urlorAWS_ENDPOINT_URL_S3), and requests tohttps://endpoints, AWS's
included, go throughHTTPS_PROXY, notHTTP_PROXY
(S3). - The image
ghcr.io/varveio/swathis distroless, without a shell: its entrypoint is/opt/swath/swath,
it runs as65532:65532, and it cannot run--s3-bearer-token-commandor a profile's
credential_process(Container images).
The 0.3 documentation stays at the v0.3.2 tag.
Evidence
The release pipeline publishes a tag only after these gates pass
(Releasing swath); the
results are in the tag's ci.yml and release.yml runs.
ci.ymlonmainat that exact commit (go-full-gate):go vetand the race tests of the three Go
modules, the default tier on macOS, Windows and Linux ARM64, cross-builds of the five targets, SeekParquet
conformance, the replay fixture tier, and both images built for two architectures and smoked offline
(testing).release.ymlon the tag:go vetand the whole race suite of the main module unsharded, the five targets
built once with each linux/amd64 binary's--versionchecked, and a smoke of the pushed image digests
before they are tagged: an anonymous listing of a public S3 prefix into a dataset, read back with DuckDB
from the manifest's part list, resumed as a no-op, and served by the replay image. The release stays a
draft until its own signatures and attestations verify against the published assets.- The license gate that generates
THIRD_PARTY_NOTICESfails the build on a linked module whose license is
not recognized or not allowed. - Measured runs, each with its build, date, machine and command, are in
measurements.md. In one
live listing with--workers 1024swath counted 1.09 billion S3 objects in 177 seconds, and in another,
also at 1,024 workers, wrote them as a sorted inventory in 846 seconds, on one 32-vCPU VM; at the default 64
workers neither finished within 40 minutes. 35 production listings of public buckets, 50.6 billion objects
in all, completed. None was taken on this tag. - What has run against the real services, all on Linux VMs and recorded in measurements.md: anonymous
listings of public S3 buckets, the billion-object runs included; the 35 production listings, of public S3
buckets with 0.4.0-rc.2, each writing its dataset to a GCS bucket with default credentials; anonymous
listings of a public Azure Blob container; and a private GCS bucket of 5.1 million objects, listed by an
earlier build. New in 0.4.0 and tested only against emulators, fakes and CI runners: S3 credential refresh
(#577); Azure Blob with a SAS, Shared Key or Entra ID
(#346); GCS with a key file; datasets written to S3 and
Azure Blob buckets; and journaled runs andswath resumeon Windows, which run on CI's Windows runners.
Limits and known issues
-
Not listed: Azure hierarchical namespaces (Data Lake Storage Gen2), S3 directory buckets, access points
and Outposts, and GCS soft-deleted objects. -
Resume: only object-store runs that write a dataset resume. A filesystem listing, text output and
swath duare run again. -
Windows: dataset runs and
swath resumework as on Linux and macOS, on an output whose filesystem
syncs directories (checked when the run starts). Power loss is safe on local NTFS; on ReFS or an SMB share
a crash or kill is recovered, but a power loss may lose the latest part names. An--overwriteblocked by
a file another program holds open fails naming that cause and is rolled back: the previous dataset stays
readable andswath resume DIRinstalls the replacement later. If the rollback is blocked too, the
directory holds no dataset untilswath resume DIRfinishes the replacement
(Known limits). -
S3-compatible stores: swath trusts a store's empty last page; a store that returns one too early makes
a listing miss keys (#265). Check a first run against a new
store with an independent count
(S3-compatible stores). -
macOS Gatekeeper. The binaries are not signed with an Apple Developer ID or notarized
(#420). A browser download is refused until you clear its
quarantine attribute, after verifying and extracting the archive (darwin-amd64on Intel):xattr -d com.apple.quarantine swath-0.4.0-darwin-arm64/swath swath-0.4.0-darwin-arm64/swath-replay
-
Archives are built for the five targets above only; images are linux only.
-
swath publishes the complete result of the live listing it performed. It is not a point-in-time snapshot
of a bucket that changes during the run. -
The rest, and what has not been measured, is in
Known limits.
What's Changed
- Release v0.3.2 by @sagiba in #212
- site: field guide model shows owner-side splits; name the output and latency-valve floors by @sagiba in #207
- site: homepage captions say what the run did before the provenance caveats by @sagiba in #208
- Link the project story from the README by @sagiba in #213
- Add GCS and Azure listing replay over one shared fixture by @sagiba in #230
- Cut per-key and first-touch cost from sorted delimiter listing by @sagiba in #236
- Add a bounded live provider conformance campaign with GCS and Azure evidence by @sagiba in #243
- ci: bump the actions group across 1 directory with 11 updates by @dependabot[bot] in #246
- Serve Azure listings as the live service does: BOM, UTF-16 order, startFrom with a delimiter by @sagiba in #244
- Inject the --inject-latency profile on the GCS and Azure routes too by @sagiba in #250
- Archive replay receipts for the merged Azure fidelity work and promote the matching rows by @sagiba in #249
- Go port of swath: S3, GCS and local filesystem providers on a shared range engine, sorted dataset format 2, durable resume by @sagiba in #248
- go: recovery refuses a linked data/, numeric suspect ordinals, bounded empty-page liveness, resume and reader docs corrected by @sagiba in #251
- go: sparse filters release their pages, unsafe cursors decline one split, --report cannot replace managed output, stricter flags, honest resource fields by @sagiba in #252
- go/order: a failed range stops the ordered stream at its node by @sagiba in #254
- go: range engine campaign: balanced splits, seed descent, admission, memory, instrumentation by @sagiba in #255
- Go port of swath-replay: sorted-Parquet reader, S3, GCS and Azure routes, differential driver, image and CI by @sagiba in #253
- Go replay follow-ups: S3 Owner and fetch-owner as live S3, GCS time_created, fuzz targets, the live matrix recorded and replayed by @sagiba in #257
- docs: field investigations: no machine-local paths in the replay server command by @sagiba in #259
- go: phase 2: text sink, delta spill, warm start, red-team fixes, CLI by @sagiba in #258
- docs: schema notes before 0.4 (GCS last_modified/etag meaning, checksum, roadmap) by @sagiba in #277
- go: spill encoders keep 1.5 MiB, not 8.6; the pinning test holds the least of three peaks by @sagiba in #278
- go: durable spill: sealed ranges survive a crash and a resume replays them instead of relisting by @sagiba in #282
- ci: a 30-minute test timeout for the Go race run by @sagiba in #283
- go/listing: make TestCancellationLeavesNothing deterministic on one core by @sagiba in #276
- ci: on-demand Go stress workflow by @sagiba in #290
- ci: skip the Java tiers when only the Go port changed by @sagiba in #285
- go: fix two flaky tests (TestDurableUnknownCodec, TestPipelinedSmallResponsesStayWithinTheCeiling) by @sagiba in #291
- go: the segment-parallel drain tail, a shared codec pool and priced writer admission by @sagiba in #281
- ci: split the go job — lint, sharded race, non-race, build by @sagiba in #294
- go: --all-versions lists S3 versions and delete markers and GCS generations by @sagiba in #280
- go: shape suite, scalefake service behaviours, nightly shape baseline by @sagiba in #289
- go/listing: TestDurableRepeatedRecovery requires a snapshot that replays (#292) by @sagiba in #296
- ci: run the Go default tier on macOS, Windows and linux/arm64 by @sagiba in #286
- go/keyrange: TestSeedFirstPageLargeFirstTile asserts the seed's tiles, not a timing ratio by @sagiba in #300
- ci: Java image public-bucket smoke is dispatch-only; offline smoke on main merges by @sagiba in #302
- docs: testing map — every test method, where it lives, how to run it, when CI runs it by @sagiba in #303
- go: switch the dataset format to SeekParquet (manifest, footer stamps, exact key bounds, replay reader) by @sagiba in #299
- go/order: TestDistinctCountsOnlyHandedRows waits on no gauge (#306) by @sagiba in #307
- go: byte identity holds per encoder path; TestFooterOnlyChange pins both (#305) by @sagiba in #308
- go/order: the durable-cache tests run on Windows; their directory listings were the failure (#301) by @sagiba in #310
- go/listing: versions x segments over durable files; K inside a certificate at the claim kill (#297) by @sagiba in #312
- ci: Go reliability items from #304 (nightly condition, tidy check, download retries, per-SHA cache, stress inputs) by @sagiba in #311
- go: a failed directory listing is an error, not an empty directory (#310 review) by @sagiba in #314
- go/objfile: open an object-store object as a file with one version pinned by @sagiba in #316
- go/order: TestDurableLedgerFailureEvicts holds the run until the append is torn and a certificate evicted (#309) by @sagiba in #317
- go/order: the segment budget tests hold the frontier until every other range is done (#313) by @sagiba in #315
- go/objfile: bump go-jose/v4 for three Dependabot alerts by @sagiba in #327
- go/s3: TestAttemptsAreRequestEconomics answers after a clock tick (#319) by @sagiba in #320
- go/order: TestSegmentsTrailerFailureClosesNoPart damages the trailer when the segments are tried (#321) by @sagiba in #323
- go/order: TestAheadWaitWakes holds the frontier until every other range is done (#325) by @sagiba in #329
- go/order: TestFetchWaitWakes holds the frontier until every other range is done by @sagiba in #331
- go: consume the seekparquet module (writer, manifest, inspection); SHA-256 digests; replay meters from ReadStats by @sagiba in #318
- go/order: TestSpillDiskBudget holds the frontier until the run signals, not a slow page (#326) by @sagiba in #330
- go/keyrange: TestPriorStale holds the grown tile until every other range is done (#332) by @sagiba in #334
- go/cmd/swath: TestSegmentsReplayDurableFiles kills at a seal after the first record is synced (#333) by @sagiba in #336
- go/reader: the swath reader module: shared schemas, metadata and typed rows (plan step 8a) by @sagiba in #335
- go/reader: Open, CheckSchema, AcceptWholeSource and Stream (plan step 8b) by @sagiba in #340
- go/reader: Totals and Aggregates over seekparquet's selection plans (plan step 8d) by @sagiba in #341
- go/s3: TestThrottleStormLowersConcurrency ends the clean window at the limiter's first increase (#342) by @sagiba in #343
- go: test infrastructure for the Azure Blob provider (account information route, long names, azblobtest corpus and server, UTF-16 comparator) by @sagiba in #324
- go/azblob: Azure Blob list client on azidentity, objhttp and tokensource by @sagiba in #328
- go/azblob: Azure Blob provider (az://, admission, schema, staged refusal of names at U+E000 and above) by @sagiba in #337
- go/azblob: list names at U+E000 and above in byte order through the order adapter by @sagiba in #344
- go: SDK-backed listing clients for S3, GCS and Azure Blob behind the swath_sdkclient build tag (experiment) by @sagiba in #348
- go/docs: split internals/providers.md into a hub, a page per provider, the list clients and the range engine by @sagiba in #349
- go/s3: read the web-identity token file's presence before the lookup, not after a failed one by @sagiba in #350
- go/replay: mark the legacy fixture readers (java-era, format 2) as transitional, deleted after the Go/Java replay comparison by @sagiba in #354
- ci: shared Go module cache, per-job build caches, bounded retention by @sagiba in #355
- go: delete swath-dataset format 2 (readers, replay profile, docs) by @sagiba in #357
- ci: split the rest race shard, macOS on main and nightly only, no cross-OS runners for Java-only changes by @sagiba in #356
- docs: CI architecture and the reasoning behind it by @sagiba in #358
- ci/setup-go: download the module graph on a copy of go.mod and go.sum by @sagiba in #360
- go/reader: typed List, ValidateDigests and seekparquet aliases (varve imports reader and objfile only) by @sagiba in #359
- go: swath du — per-prefix totals, like du, for buckets and local trees by @sagiba in #347
- ci: BuildKit caches are written from main only by @sagiba in #361
- docs/measurements: CI caches, race shards and cross-OS legs after #355 and #356 by @sagiba in #366
- go: swath/local-entries/v1, extensions.swath.version and layered show output by @sagiba in #367
- go/localfs: huge directories and cold counts (parallel sort, pointer-free entries, count processors) by @sagiba in #339
- docs: go/docs/dataset.md owns the swath dataset profile by @sagiba in #369
- go/localfs: a count does not sort; NFS stays on READDIRPLUS across large directories by @sagiba in #351
- go/objfile: S3 objects at version "null" are identified by ETag by @sagiba in #370
- go/localfs: on macOS, stat each entry relative to its directory and fill every stat column by @sagiba in #368
- go/reader: List pairs the object at a common prefix's exact key onto the prefix entry by @sagiba in #371
- ci: module-cache download copies go.mod/go.sum in the repository layout (fixes main after #371) by @sagiba in #375
- go/localfs: a count does not sort on the portable engine either (Windows, macOS) by @sagiba in #372
- go/reader: AdmitWhole/ReopenWhole with Identity, IsRefusal, and Dataset.Page by @sagiba in #373
- go/objfile: durable pins (Pin/Restore/Snapshot), ParseLocation, objfile/remote.Open, served-offset range guarantee by @sagiba in #374
- go/reader: typed local entries, nullable local totals, Azure exact totals, parallel Scan; localfs records --no-dirs as a coverage filter by @sagiba in #377
- go/objfile: remote.Open's GCS client uses the new auth library; ReopenWhole counts note by @sagiba in #379
- go: one SDK-backed listing client per provider; explicit anonymity; one normalized failure value; swath owns only the swath-specific parts by @sagiba in #383
- docs: describe Go listing and inventory across storage backends by @sagiba in #378
- Go port: a provider is one directory plus one import (issue 338) by @sagiba in #386
- Go port: provider audit and docs pass by @sagiba in #387
- go/localfs: default --workers is 4 per CPU, at most 64 by @sagiba in #389
- go/order: clear the popped stack slot so expanded ranges can be collected by @sagiba in #390
- Go port: no credential-bearing endpoint or location reaches an error or a record by @sagiba in #393
- Go port: remove dead code and redundant tests in go/internal/source by @sagiba in #395
- go/localfs: a count reads large directories in slabs, handing chunks to other workers by @sagiba in #399
- ci: publish images to GHCR only from releases by @sagiba in #398
- go/localfs: warn when a listing skips a directory too large to sort by @sagiba in #401
- release: ship the Go binaries and images by @sagiba in #403
- go/s3: pin the fast timestamp parser to the standard one (test, fuzz, benchmark) by @sagiba in #400
- ci: shorten PR feedback and bound shared cache growth by @sagiba in #402
- release: prepare v0.4.0-rc.1 by @sagiba in #407
- ci: parallelize isolated crash tests and narrow Java inputs by @sagiba in #410
- ci: Dependabot for Go modules and Dockerfile.go; neutral OSV exception label by @sagiba in #409
- swath-core: make FinalizationPipelinePropertyTest's roll precondition deterministic by @sagiba in #411
- go/cli: GNU-style parsing, provider-neutral text rows, prefixed provider flags, new help by @sagiba in #394
- go/keyrange: remove the floors split policy and --floors by @sagiba in #415
- ci: scope Go PR validation and require full release qualification by @sagiba in #414
- release: version check expects 'swath X.Y.Z (commit)'; swath-replay prints the same form by @sagiba in #418
- release: name the Go tools swath/swath-replay; run images as distroless nonroot by @sagiba in #419
- go/docs: user docs by task, one reference page per command, internals moved out by @sagiba in #417
- release: darwin/amd64, SBOM root name, cosign v3 + macOS quarantine docs; deflake part-interval test by @sagiba in #423
- go/reader: open and read only; remove admission and digest validation (#397) by @sagiba in #424
- replay: build the SeekParquet list boundary index at open by @sagiba in #425
- deps: bump the go-objfile group in /go/objfile with 7 updates by @dependabot[bot] in #413
- ci: bump the actions group across 1 directory with 6 updates by @dependabot[bot] in #405
- ci: setup-go v7 in the composite action; Dependabot scans .github/actions by @sagiba in #427
- deps: Go 1.27.3 toolchain (go.mod toolchain lines and Dockerfile.go together) by @sagiba in #428
- go/listing: the default row budget follows memory; count spill frames and opens by @sagiba in #426
- reader: baseline Scan and Stream benchmarks and an allocation guard by @sagiba in #433
- go/cli: --row-budget is --sort-buffer by @sagiba in #429
- deps: azblob 1.8.3 and Go dependency refresh; Azure wire snapshot by @sagiba in #434
- deps: Go 1.27.4 and the release dependency refresh by @sagiba in #438
- reader: borrowed column-batch scans with field selection by @sagiba in #441
- ci: bump the actions group across 1 directory with 2 updates by @dependabot[bot] in #432
- go/azblob: list in the Arrow format, XML when the service answers XML by @sagiba in #440
- ci: DuckDB 2.0.0, OSV-Scanner 2.7.1, Syft 1.57.x by @sagiba in #442
- go/replay: fill the window cache from SeekParquet's borrowed batches by @sagiba in #443
- ci: pin Buildx and the BuildKit image in release; correct two dependency doc statements by @sagiba in #444
- go/clients: wire snapshots for S3 and GCS, Arrow column types through the decoder by @sagiba in #446
- reader: Stream in one ordered pass of RangeBatches by @sagiba in #445
- go: one-line credential refusals; refuse a SAS naming a request parameter by @sagiba in #447
- go/azblob: 1,000-entry default page by @sagiba in #448
- go/azblob: pin x-ms-version 2026-06-06, which Azurite answers by @sagiba in #451
- CHANGELOG: 2026-11-10 changes by @sagiba in #450
- reader: keep TestStreamMatchesPages inside the race job's time by @sagiba in #454
- reader: scan shards from a queue, eight per worker by @sagiba in #455
- reader: Totals in shards over borrowed batches; certify key order across shard boundaries by @sagiba in #456
- replay: compare the Go and Java servers on converted fixtures (#352) by @sagiba in #452
- go/s3: cut the list scanner's and the density step's CPU per object by @sagiba in #458
- go/cli: split SWATH_CPUPROFILE and SWATH_MEMPROFILE at the end of listing by @sagiba in #463
- order: reuse spill read handles across frame and trailer reads; count them and frame sizes by @sagiba in #464
- go: write sorted datasets directly to GCS (#266, phase 1) by @sagiba in #465
- go: write sorted datasets directly to S3 (#266) by @sagiba in #466
- go: write sorted datasets directly to Azure Blob (#266) by @sagiba in #467
- listing: admit encoders against their own memory allowance by @sagiba in #468
- parquetout: compress pages at zstd SpeedFastest by @sagiba in #471
- parquetout: build each column's null value once and reuse a repeated string's value by @sagiba in #472
- go/azblob: read once per page in the byte-order adapter (#449) by @sagiba in #469
- go/azblob: admit a flat account whose Data Lake host refuses it for soft delete (#457) by @sagiba in #459
- objfile: clarify upload contracts and streamline library docs by @sagiba in #473
- go/azblob: --all-versions (versions, snapshots, soft-deleted blobs) and the blob properties every listing returns by @sagiba in #460
- go/keyrange, azblob: a counted prior lists in whole pages of 5,000 (#449) by @sagiba in #470
- go: read through the redesigned SeekParquet Go API; ListRequest.ReadAhead by @sagiba in #474
- objfile: batched range reads, verified sized opens, whole-object reads by @sagiba in #475
- go/cmd/swath: the failed-after-parts test waits for a saved part before it fails the listing by @sagiba in #476
- docs: the Go docs as swath's docs — concepts, generated CLI reference checks, link check, library examples by @sagiba in #477
- objfile: request observer, cache mode, identity-keyed byte cache, sized connection pool by @sagiba in #478
- go: read SeekParquet datasets through objfile (spsource adapter, request budgets, zero-read count) by @sagiba in #479
- release: notes for 0.4.0-rc.2 by @sagiba in #480
- go/reader: browse page counts from row spans and budgeted value-only sums by @sagiba in #481
- docs: release-candidate install page, Windows limits, interim dataset validation; readbudget name check by @sagiba in #486
- go/azblob: zero SDK timestamps are missing times; Totals docs (#487) by @sagiba in #488
- Report committed counts and object bytes on resume; not_found for missing buckets (#482) by @varve-steno in #489
- go: follow SeekParquet's API cleanup; callers keep bulk reads out of objfile's cache (#484) by @sagiba in #490
- docs: where not_found applies (source buckets; not anonymous Azure or output buckets) by @sagiba in #491
- go/cmd/swath: refuse a bare -o NAME.parquet; NAME.parquet/ selects a directory (#142) by @sagiba in #493
- go/reader: keep only swath's meaning; scanning, aggregation and pages move to SeekParquet by @sagiba in #495
- docs: macOS quarantine note in the release notes; what swath trusts on S3-compatible stores by @sagiba in #496
- go/internal/listing: an Azure Blob row in the crash matrix (#241) by @sagiba in #497
- go/localfs: prune subtrees covered by recursive excludes by @sagiba in #504
- go: a local listing records its filesystem's space in the manifest by @sagiba in #503
- go/replay: fixtures are generated by Go; go-fixtures needs no JDK (#353) by @sagiba in #507
- go: tests for scenarios the Java suite covered and Go did not by @sagiba in #508
- Retire the Java implementation: delete the tree, its build and its CI (#73) by @sagiba in #510
- go: a bucket run resumes from a copied work directory (test); Spot guidance and stale resume notes in the docs (#511) by @sagiba in #515
- Move the Go module to the repository root (#293) by @sagiba in #529
- docs: spill at billion scale, and what a spill cap too small does by @sagiba in #534
- Hands-on fixes: text output excludes itself, recovery advice that works, partial-sample resume by @sagiba in #531
- go: swath du --from-dataset DIR [PREFIX] totals a saved dataset by @sagiba in #532
- Go run trace (swath diag list --trace) and the explainer on it by @sagiba in #536
- CLI: exit 3 for incomplete output, unreadable naming, signal death, usage errors, du hard links by @sagiba in #537
- site: VitePress site from docs/, with a new home and Start pages by @sagiba in #533
- First-run fixes: stop early when nothing is reachable, dump to a file, tmpfs warning at first spill by @sagiba in #538
- ci: a docs tier for pull requests that change only pages under docs/ by @sagiba in #539
- site: development-docs notice, rc.2 differences, install for every platform, reviewer fixes by @sagiba in #543
- README: a one-minute page that hands off to swath.varve.io by @sagiba in #542
- docs: one page per source; remove the 0.3 migration page by @sagiba in #547
- deps: Go dependency refresh across the three modules; one Dependabot entry for all of them by @sagiba in #546
- go: a spill-capped run no longer dies with 'spill: corrupt frame' (a refused range's trailer at offset 0) by @sagiba in #551
- go: instrument the spill-cap stall (wait sites, frontier split eligibility, refusal time, writer during listing) (#535) by @sagiba in #549
- test: TestPhaseProfilesRun no longer fails when a short listing ends before the profile's start is stamped by @sagiba in #552
- go: say so when the spill cap is too small (error event after sustained refusal; --spill-wait-timeout exits resumably) (#535) by @sagiba in #553
- go: durable spill certificate limits scale with the run (#513) by @sagiba in #554
- deps: seekparquet to 8ceebf4 (read-ahead and forward routing fixes) by @sagiba in #550
- go: a diag knob for the segment tail's worker count (--segment-workers) by @sagiba in #556
- deps: seekparquet to 41e4319 (listing read-ahead sized by the entries its pages yield) by @sagiba in #559
- go: durable spill certificate slots are bounded by their metadata, not by spill size (#513) by @sagiba in #560
- go: a resume that reuses certificates writes its tail in parallel segments (#288) by @sagiba in #561
- go: a seeded run lists front to back inside a window at the output frontier (#535) by @sagiba in #555
- docs: one pass over the guide and internals after the spill, seeded-window and resume changes by @sagiba in #562
- go: every sorted dataset carries its cut points for the next listing (a side file beside the manifest) by @sagiba in #557
- test: TestSegmentsReplayDurableFiles accepts a certificate the drainer reaches before the cut rule by @sagiba in #564
- deps: bump the go group across 3 directories with 1 update by @dependabot[bot] in #548
- go: format versions start at 1 (report, journal, durable spill); a report schema change no longer needs a bump by @sagiba in #565
- go: untrusted names are escaped on every human-readable stderr line (#521) by @sagiba in #572
- docs: first-run commands, the benchmark sentence and copy fixes from the site review by @sagiba in #570
- docs: command options render as grouped, wrapping lists generated from the flag definitions by @sagiba in #574
- go: the frontier window shrinks to what the spill cap can serve instead of crawling on forced admissions (#535) by @sagiba in #563
- go: a regular expression given to a glob filter is refused; a 0.3 run directory is recognised (#526, #527) by @sagiba in #571
- objfile/remote: one Options for opening a location, with OpenWith (closes #384; step 1 of #567) by @sagiba in #569
- go: the default spill cap follows the disk, so a disk grown under a live run is used by @sagiba in #566
- go/s3: credentials refresh before they expire, and an expired-token answer refreshes and retries once (#523) by @sagiba in #573
- site: the home page leads with a real run on a public bucket and its replay by @sagiba in #575
- test: TestPlanStaleManyWorkersSmallCap allows a slow runner's refusals by @sagiba in #581
- test: the fake credential source's expiry advances with each fetch by @sagiba in #583
- deps: seekparquet at its release go/v0.2.0 by @sagiba in #580
- go: swath du rows carry both sizes under the dataset's names; the list table names its zone by @sagiba in #582
- go: the swath du table shows each prefix's newest time by @sagiba in #591
- go: main green again: an abort race in part publication, five flaky tests by @sagiba in #590
- go: list and du command-line grammar: -h, -o names, filters, SOURCE forms, one error shape by @sagiba in #585
- report, progress, log: one vocabulary for the machine surfaces by @sagiba in #587
- release: tag reader and objfile with every version by @sagiba in #592
- go: complete means nothing is missing; one verdict; --overwrite keeps the previous dataset until commit by @sagiba in #589
- du --from-dataset DIR [SOURCE]: the positional is the dataset's source or a path below it by @sagiba in #593
- go: bare swath du on a terminal draws the largest prefixes at any depth by @sagiba in #588
- go: what a person sees: the closing lines state the result, swath show reads in order, one table grammar by @sagiba in #595
- site: a field guide to the Go engine, and the home page's run report by @sagiba in #597
- du --from-dataset on a file, quoted-dot globs, cuts.txt after an overwrite resume; drop --segment-workers by @sagiba in #598
- spill_pressure: with the default cap, say to grow the disk by @sagiba in #600
- site: the field guide's visuals, rebuilt for the Go engine by @sagiba in #599
- go: the du view fills the terminal, reads size, count, share, bar, date, path, and marks what it misses by @sagiba in #596
- docs: split the user guide by the question asked by @sagiba in #601
- go: the du view spends no line under a floor and opens the largest prefixes by @sagiba in #603
- localfs: list directories deeper than PATH_MAX by @sagiba in #602
- site: home page leads with the swath du view by @sagiba in #594
- test: the CI flakes for 0.4.0 made deterministic; Azure admission waits out a busy account by @sagiba in #607
- Windows: journaled runs and swath resume by @sagiba in #605
- test: keyrange's stale-prior and first-page tests gate on the engine's decisions, not wall time by @sagiba in #609
- ci: Windows go-xos in three legs along the race shards by @sagiba in #608
- test: Windows flakes from journaled runs; GCS retries a connection reset on Windows by @sagiba in #611
- dataset: a refused --overwrite never restores the old manifest without its data/ by @sagiba in #610
- go: an --overwrite installation syncs the commit record apart from the data; du rows give an unreadable directory null; NOTICE files; documentation corrections by @sagiba in #612
- test: journaled tests that must complete get a long watchdog; the nightly Windows job runs in legs by @sagiba in #613
- release: the 0.4.0 copy (release notes, changelog, no release-candidate wording) by @sagiba in #604
New Contributors
- @varve-steno made their first contribution in #489
Full Changelog: v0.3.2...v0.4.0