ObjectFS v0.13.0

Two things that existed and did nothing now do something.

cluster.enabled: true started a gossip ring and stopped there: the adapter never constructed a
ClusterManager, FileSystem had no coordinator, and AnnounceKey/QueryKeyOwnership returned
ErrNotSupported from stubs. A cluster's nodes could see each other and could not use each other. Now a
write invalidates what peers have cached at the ETag it wrote, a cache miss learns from peers which keys
are hot and reads further because of it, and a joining node is told what to warm before it serves its
first request. The bytes still come from S3 — peer data fetch is not in this release, and [#399] is why:
a sealed gossip datagram tops out at 5802 bytes of payload, so the datagram carries the metadata and
S3 carries the object. That is the honest version of the feature rather than a 128 KiB read that fails as
a 30-second timeout on a different host.

The observability half is the same shape. internal/awsrates held request prices verified against the
live AWS Pricing API and every path from a rate to a user was severed — a cost package with no
importer, a RecordCost with no caller, a report behind a config key no mount could set. Latency
percentiles were declared and never assigned, computed from a histogram that was ms % 100 rather than
a bucketing, so 50 ms and 250 ms shared a bucket. A Transfer Acceleration fallback was permanent for the
life of the mount, invisibly. Compression ran zstd over BAM, tar.zst and JPEG and then discarded the
result. Each of those was a feature the documentation described, and none of them could be observed or
switched back, which is the difference this tag is about.

Milestone v0.13.0 — Cache Warming & Operational Excellence closed at zero open, 20 issues.

Cost accounting is worth one caution: the request counts come from a middleware in the AWS SDK's
Deserialize step, not from the wrapper layer, because those two disagree — a 5 GB PutObject is one
wrapper call and 641 requests to S3. The figures are what S3 was asked for by this mount, per attempt,
excluding 5xx. They are not a substitute for Cost Explorer, and the metric's help text says so.

Added

• Data that arrived compressed is no longer compressed again ([#184]). With compression enabled, a
  write above min_size whose leading bytes name a compressed format — gzip, zstd, bzip2, lz4, xz, zip,
  PNG, JPEG, GIF, WebP, MP4, Matroska, Ogg, MP3, PDF — is stored as-is and the codec is never entered.
  These are the formats a research bucket is mostly made of, and a BAM counts: BGZF is gzip per block, so
  its first bytes are gzip's.

  Measured on an Apple M4 Max at zstd level 3, six runs per case compared with benchstat: an 8 MiB zstd
  frame goes from 1.85 ms and 33 MiB of allocation to 3 ns and none, 1 MiB from 237 µs to 3 ns,
  64 KiB from 7.4 µs to 3 ns. Data the skip does not apply to is unchanged — 1 MiB of compressible text is
  91.5 µs against 90.4 µs, which is the row that says the check pays for itself, since a gate that taxes
  the compressible path is not a win. Reproduce with BenchmarkCompressAlreadyCompressed,
  BenchmarkCompressCompressibleText and BenchmarkGateVersusAnalyze in internal/compression.

  The gate is a magic-byte comparison at 17 ns, not the full analyzer. Analyze also runs a Shannon
  entropy pass over its 4 KiB sample, which costs 3.8 µs and was +53% on a 64 KiB text write for a
  figure the decision does not use — so the write path calls a new AlreadyCompressed, and both it and
  Analyze classify through the same classifyByMagic, leaving one authority on what "already
  compressed" means. The check sits after the min_size floor, so an object below the floor is not
  sampled, and after the enabled check, so a mount with compression off samples nothing at all.

  Not done, deliberately: nothing is skipped on entropy. High entropy is evidence that compression
  will not help, not proof, and no threshold for it has been measured here — declining to compress on a
  guess is worse than spending the pass. Formats whose compression is internal to the container (CRAM,
  Parquet, ORC, HDF5, Zarr) are also not skipped, since the container's magic bytes say nothing about its
  contents; docs/features/compression.md now sorts the formats into the two lists explicitly, and
  TestAlreadyCompressedMatchesTheDocumentedFormatLists fails if that page and the table disagree.

• A mount now publishes what it is spending at AWS ([#226]). objectfs_s3_cost is a new gauge family
  carrying billable request counts by pricing group, bytes retrieved, three dollar figures, and — beside
  each — the rate it was computed at, labelled with the region the rates were resolved for and the storage
  tier they belong to. Before this, internal/awsrates held a rate table verified against the live AWS
  Pricing API by an integration test, and every path from a rate to a user was severed: the
  access-pattern report was gated behind a config key no mount could set, internal/cost had zero
  importers, and metrics.RecordCost had no caller anywhere.

  The requests are counted in the AWS SDK's own response path, not at the wrapper layer, and that is
  the substance of it rather than a detail. The eight places this backend records metrics are not
  one-to-one with billable calls: a 5 GB PutObject is one wrapper call and 641 requests to S3 — one
  CreateMultipartUpload, 639 UploadParts, one CompleteMultipartUpload — and a large CopyObject,
  a parallel ranged read and the capability probes fan out the same way, one of them counted nowhere at
  all. Every one of those errors is in the direction that flatters. A middleware in the SDK's Deserialize
  step, installed on every client this package builds including the connection pool's, cannot disagree
  with what S3 received because it is what S3 received. It counts per attempt, since the retryer
  re-enters that step; it requires a response, since a connect or DNS failure never reached S3 and is not
  billed; and it counts only statuses below 500, because AWS does not bill for its own server errors
  and does bill 4xx — a HeadObject returning 404 is a charged read, and 5xx during an S3 event would
  otherwise inflate the figure at the moment an operator is looking at it hardest.

  Four pricing groups, not AWS's two request tiers: writes, lists, reads, and the free operations. Lists
  are separate from writes even though the published price list calls both "Tier1", because on
  DEEP_ARCHIVE a PUT is $0.05 per 1,000 and a LIST is $0.005 per 1,000 — a factor of ten, and pricing
  a directory traversal at the write rate would overstate it worst on the class where the error is largest.
  Classification is by operation-name prefix and an unrecognized operation counts as a write, so an
  operation this code has never heard of makes the reported cost too high rather than too low.

  What the figures are is stated narrowly, because a cost number that implies more than it knows is worse
  than none. They are this process's spend since it started, at list prices for the first volume band:
  not a bill, and not a reconciliation of one. bytes_stored is what this mount has uploaded, not the
  bucket's size — nothing lists the bucket, since that would be a billed request per scrape to publish a
  metric. Every figure is monotonic and there is no reset, because rate-of-change is the form a useful
  alert on cost takes and a counter something can zero is one that will be zeroed. A fresh mount reports
  read_requests of 1 before any filesystem work: NewBackend's health check is a HeadBucket, AWS bills
  it, and the tests assert that 1 rather than asserting zero.

  Verified against a recording proxy rather than against the counter's own arithmetic: the tests compare
  what the tally counted with what the S3 endpoint actually received, including a multipart write where
  both sides must agree and the count must exceed the number of parts, and an injected 500 where the
  counted total must equal the observed total minus the server errors. The dollar assertions state AWS's
  published rates as literals and divide in the test, per the lesson from [#220]: the old tests passed
  1024*1024*1024 bytes, called it one GB, and asserted the per-GB rate came back — an expectation that
  holds under both the right divisor and the wrong one. Byte-to-GB conversion goes through
  awsrates.GBFromBytes throughout, and the mutation to a binary divisor is caught with a ratio of
  0.9313, which is exactly the 7.4% gap.

• A cache miss now reads further when a peer holds more of the object ([#142]). A read that misses
  asks the cluster who holds the key and, if some peer holds a range reaching past what was just read,
  reads ahead to the end of that range — capped at 4 MiB per miss. The bytes come from S3, not from the
  peer: a gossip datagram carries 5802 bytes of object at the default limit, so a 128 KiB read is 21×
  over and fails as a 30-second timeout on the requesting host ([#399]). What a peer's claim is evidence
  of is that the key is hot in this cluster, which is the one thing a cold node cannot learn on its own,
  and it is worth acting on even when the bytes still cost an S3 read.

  A claim is acted on only at the version this node is reading. A holder's range describes the object
  it cached; against a different version the object may have a different length entirely, so warming on it
  would fetch bytes chosen by a claim about something else. A version mismatch — or no known version on
  either side — warms nothing, which costs exactly the read the application asked for. Every claim at the
  matching version is considered and the furthest-reaching one wins, since holders arrive freshest-first
  and freshest is not furthest.

  Warming goes through the existing read-ahead queue, so it inherits clamping to EOF, skipping what is
  already cached, standing off reads in flight, and sharing a fetch with a covering request. It does not
  touch the read pattern: another node's reads must not decide whether this reader looks sequential. It
  is triggered by the application's read alone and never by a warm's own fetch, so a warm cannot feed
  itself and walk the whole object. cache.read_ahead.enabled: false suppresses it entirely, and
  suppresses the ownership query too rather than asking and discarding the answer — an operator who
  turned read-ahead off has said not to read bytes nobody asked for.

• A joining node is told which keys are hot ([#143]). A node answering a join follows the membership
  sync with a cache_warmup message listing what it holds, freshest first, and the joiner records those
  as ownership claims — so it starts knowing where the cluster's working set lives instead of discovering
  it one miss at a time. Metadata only, like [#140]: the bytes are still fetched from S3 when a read wants
  them.

  The message is bounded by measured sealed bytes, not by an entry count, reusing the mechanism the
  membership sync already uses. The specification's "max 256 entries" would not have worked: at 52-character
  keys, 256 announcements seal to 65631 bytes against the default 8192-byte limit, 8× over, so every warmup
  datagram would be refused at the socket and a joining node would warm nothing at all. 31 fit at that key
  length — and 31 is not a constant to hardcode either, since it moves with key length, which is why the
  chunk is grown one entry at a time and sealed to check. A join sends at most four datagrams, ~124 keys at
  that density, and logs what it held back; the rest reaches the joiner through ordinary announcements.

  A batch is recorded through the same path as a single announcement, so the self-claim refusal, the
  per-node replacement, the map bound, the local timestamp and the empty-ETag refusal all apply to it
  unchanged — batching is not a way around any of them. Each entry is credited to the peer the datagram
  came from rather than to the node named inside it, so one member cannot populate a joiner's whole
  ownership map with fabricated holders.

• A write on one node now evicts what its peers have cached ([#141]). [#140] gave the cluster the
  vocabulary; this is the read and write paths using it. A read that misses and fetches from S3 announces
  the range it cached, so a peer that wants those bytes can weigh asking against reading S3 itself. A
  flush, an unlink, an rmdir, a mkdir, a create and both halves of a rename invalidate on every peer as
  well as locally. Until this, a two-node deployment kept serving a file's previous contents for up to
  five minutes after another node overwrote it — every part of the invalidation machinery existed and
  nothing called it.

  The invalidation carries the ETag the write itself reported, which is why Flush now asks the write
  path for it rather than discarding it: a receiver's replay ledger is keyed on (key, version), so a
  version fetched from a later HeadObject could name a third node's subsequent write and suppress an
  invalidation that was never applied. A delete has no version to name and sends an empty one, which is
  legal and means "evict whatever you hold". Local and remote eviction go through a single call, so a
  future mutation path cannot add the local half and forget the remote one — the half that is invisible on
  a single-node mount.

  Both directions are fire-and-forget and never fail a syscall, but they are not logged alike. A lost
  announcement costs a peer an S3 read, which is slower and nothing more, so it is Debug. A lost
  invalidation means peers serve bytes this node replaced until their cache TTL expires, and nothing else
  in the system reports it, so it is Warn: a mount whose invalidations are all failing is serving stale
  reads cluster-wide.

  An announcement is only sent when this node already knows the object's version, from the stat the kernel
  issues before any read. With no version known, nothing is announced rather than a version being invented
  — a peer that fetches bytes it cannot place against an object version hands them to a reading process as
  file content. On a single-node mount the coordinator is nil and none of this runs: measured at 4.8 µs and
  2 allocations for a 128 KiB cached read either way, since a cache hit never reaches the announce call.

• Peers now learn which keys are cached elsewhere in the cluster ([#140]). AnnounceKey and
  QueryKeyOwnership were stubs returning ErrNotSupported; both are real. A node broadcasts a
  cache_announce gossip message naming the key, its ETag, its size and the byte range it holds;
  receivers record the claim against the peer that sent it, expire it after cluster.announcement_ttl
  (default five minutes), and answer a local lookup with the holders freshest-first. It is metadata
  only — no object bytes cross gossip, because a datagram cannot carry a 128 KiB read ([#399]) — so what
  this buys is knowing which keys are worth warming, and the bytes still come from S3.

  Three details are worth an operator's attention because each departs from what a reader might assume.
  A node does not record its own announcements: the query exists to answer a miss on this node's own
  cache, so a self-entry would be a holder guaranteed not to hold, returned in place of a peer that
  might. Expiry uses the moment this node received the claim, never the sender's cached_at, whose own
  contract forbids comparison against a local deadline — otherwise a peer whose clock runs an hour behind
  would have every announcement expire on arrival, and one running fast would sort itself to the front of
  every key in the cluster. And the announcement is credited to the peer the message came from rather
  than to the node named in the payload, so a member cannot send the cluster to fetch bytes some third
  node never cached.

  An announcement missing its ETag is refused rather than sent, and refused with the field named. This is
  the integrity boundary: an invalidation with no version is still safe, since "evict whatever you hold"
  cannot serve wrong bytes, while a peer that fetches bytes it cannot place against an object version
  hands them to a reading process as file content. Announcing with gossip not running returns
  ErrNotSupported rather than nil — a caller told nil believes the cluster knows something it was never
  sent, which is precisely the defect [#284] deleted a CacheReplicator for. announced_keys is now in
  the coordinator's statistics, counting keys retained, so the gap between it and what a query returns
  is how far behind the expiry sweep is running.

• cluster.enabled: true now actually starts cluster coordination ([#139]). Every part of
  internal/distributed was built, tested and reachable by nothing: no code path anywhere constructed
  a ClusterManager, so a two-node deployment whose configuration said it was clustered got no
  membership, no cache invalidation and no warming — the cluster: block's only live effect was
  selecting a Redis cache. internal/adapter now builds one, injects the backend and cache, starts it
  before the mount and stops it during teardown, and passes its coordinator down through
  MountConfig to the FileSystem.

  Two things it deliberately does not do. It does not start Raft: ClusterConfig.EnableConsensus is
  new, off by default, and a mount leaves it off, because coordination here is compare-and-swap
  against S3 — the store evaluates a conditional write, which needs no quorum and keeps working with
  one node reachable — so an election would decide nothing a filesystem read asks about while making a
  cluster below quorum degrade a mount that never needed one. And it does not degrade: a cluster that
  cannot start fails the mount, with the reason, rather than continuing single-node. Coherence is a
  correctness capability, and a node that believes it is clustered and is not serves cached bytes a
  peer has already overwritten with nothing in its logs to say why.

  A single-node mount gets a nil coordinator, which is what every path added here is guarded by,
  and that nil is asserted rather than assumed: GetCoordinator returns a wrapper that is a non-nil
  interface value whatever it holds, so the adapter checks before calling it.

• cluster.secret_file, the key LoadClusterSecret's error message had been naming before it
  existed ([#139]). A cluster refuses to start without a shared gossip secret ([#206]), the error
  told operators to set OBJECTFS_CLUSTER_SECRET or cluster.secret_file, and only the first of
  those was real. The path — never the secret itself — is now a key in the cluster: block, and the
  file it points at must be mode 0600 or startup refuses it. The environment variable still takes
  precedence, which is what a container orchestrator injects.

• Wasabi probed and added to the conditional-write compatibility matrix. It is the first endpoint
  in the matrix to answer success to every cell: If-None-Match: * over an existing key replaces
  it, If-Match with an ETag that cannot possibly match performs the write, and If-Match against an
  absent key creates it. Conditional headers are accepted and never evaluated. The capability probe
  reports ConditionalWrite=false, PutObjectIf returns ErrNotSupported, and coordination features
  decline to start — the fail-closed direction, and the reason this is less dangerous than Ceph RGW,
  which enforces some preconditions and so passes a probe that only asks whether one was ever
  evaluated. Plain filesystem use is unaffected; this is a coordination limitation, not a storage one.

  "Every request succeeds" is also what a client dropping the header would look like, so the SDK was
  ruled out at the wire before the row was written — the same standard the RGW quoted-ETag row was held
  to. With a request dumper in place, If-Match is present as a header and inside
  SignedHeaders=…;if-match;…, Wasabi answers 200, and a following GET returns the contender's
  bytes. The row is dated rather than versioned because Wasabi returns no Server header and publishes
  no build identifier, so re-run the suite rather than reading the date as a guarantee.

• RustFS 1.0.0-beta.12 probed and added to the conditional-write compatibility matrix. Run, not
  read: the same s3compat suite that produced the AWS, MinIO and RGW rows, pointed at a local
  container, with each of the four cells the suite does not print measured on its own key. It is the
  first non-AWS endpoint to match AWS on every cell, including the two RGW gets wrong — If-Match
  against an absent key is 404 NoSuchKey rather than 412 (the distinction every CAS loop is built
  on), and a precondition on CompleteMultipartUpload is evaluated, so a conditional write above the
  multipart threshold stays conditional. It also honors a conditional DeleteObject, where MinIO and
  RGW both accept the header and delete anyway, and it accepts the quoted ETag it returned, so
  PutObjectIf works with the value the store gave it. The capability probe reports
  ConditionalWrite=true, and TestCompatCapabilityProbeMatchesObservedBehavior confirms the probe
  agrees with the endpoint. Recorded with the image digest and revision because beta.12 is a
  pre-release: the row says what that build did on 2026-08-08, and the mount-time probe is what
  protects a deployment if a later beta regresses.

• objectfs_s3_acceleration, so an operator can see whether Transfer Acceleration is in effect
  ([#204]). One gauge family with a statistic label, matching objectfs_predictive_cache, carrying
  configured, active, requests, bytes, fallbacks, avg_latency_seconds and
  retry_period_seconds. configured and active are separate series and the difference between them
  is the point: 1 and 0 is a mount that was asked to accelerate and is not, and neither series alone
  can say that. Before this, BackendMetrics.AccelerationEnabled — whose only writer was NewBackend,
  passing the config flag, behind a GetMetrics with no caller outside its own package — reported
  acceleration enabled on a mount that had been serving every byte over the standard endpoint since its
  first request.

  Registered whether or not acceleration is configured, unlike objectfs_predictive_cache, whose absence
  is meaningful. configured 0 says the operator asked for the standard endpoint; an absent family says
  this build does not report acceleration — and which of those they are looking at is the first question
  an operator investigating slow reads has to answer. sdks/testdata/metrics-scrape.txt carries the
  family with configured 1, active 0, so both SDK suites parse the state worth alerting on rather than a
  healthy one.

• storage.s3.acceleration_retry ([#204]). How long a Transfer Acceleration fallback lasts before one
  request is allowed to try the accelerate endpoint again; default 5 minutes, ignored unless
  use_acceleration is true. It must carry a unit — the loader rejects a bare number, because yaml.v2
  reads 300 into a time.Duration as 300 nanoseconds with no error, which would put one request per
  300 ns against the accelerate endpoint.

Changed

• BREAKING: cluster.enabled: true now requires a gossip secret, including for a Redis-only
  deployment ([#139]). cluster.enabled is what selects the shared Redis cache and it is also what
  starts the gossip layer, so a configuration that set it purely to get Redis will now fail at startup
  with no cluster secret configured until cluster.secret_file or OBJECTFS_CLUSTER_SECRET is set.

  The coupling is deliberate rather than an oversight of the wiring. A Redis cache shared by several
  mounts with no invalidation between them is precisely the incoherence the cluster: block exists to
  prevent: one node overwrites an object, the others keep serving what they cached, and nothing in any
  log says why they disagree. Before this release the invalidation half did not exist at all, so the
  shared-cache half was the only thing on offer; now that both do, having the cache without the
  coherence is not a configuration worth supporting. Generate the secret with
  openssl rand -hex 32 > /etc/objectfs/cluster.secret && chmod 600 /etc/objectfs/cluster.secret.

• The support posture is now stated as a thesis rather than left implicit: AWS S3 is the primary
  backend and ObjectFS uses every S3 capability that benefits it; S3-compatible endpoints are
  best-effort and get a fallback or a lesser capability. This was already how the code behaved —
  the capability probe, ErrNotSupported on an endpoint that fails it, and Transfer Acceleration's
  silent fallback are all v0.12.0 and earlier — but nothing said it, so the next capability had no
  rule to follow and the docs drifted the other way. README.md and CLAUDE.md now name the two
  degradation rules that the existing code already distinguishes: a performance capability falls
  back silently, because slower is a correct outcome, and a correctness capability fails closed,
  because a precondition an endpoint silently drops tells every contender for a lease that it won.
  Capabilities are established by probing the endpoint, never from a config flag, an endpoint-URL
  heuristic, or a version string.

• docs/index.md no longer claims "Universal compatibility: works with AWS S3, MinIO, Ceph, and
  all S3-compatible storage." It was the exact claim the thesis rejects, and it was also false in a
  way this repository had already measured: Ceph RGW 19.2.0 fails the conditional-write probe, so
  coordination declines to start there. The page's title and overview said "S3-Compatible Object
  Storage" where the project targets AWS S3. Replaced with what varies, which is coordination, and a
  link to the probed matrix — plain filesystem use is unaffected on any S3-compatible endpoint.

• scripts/pjdfstest.sh now says it runs on demand only, and why ([#352]). The script works —
  real mount subcommand, prerequisite checks, an EXIT/INT/TERM trap, ${PIPESTATUS[0]} propagated
  past the tee — and nothing runs it. That is a statement about infrastructure, not about the
  script: it needs /dev/fuse, real credentials and a real bucket, and this repository has no
  scheduled real-AWS job at all, so wiring it in means adding one with a bucket and a role. Written
  into the script header and beside make test-posix, matching what make test-fuse-mount already
  does, because the failure mode of an unrun conformance suite is that it reads as a passing one.

  Kept rather than deleted: it is the only third-party POSIX conformance suite this project has, and
  internal/difftest — which does run in CI — makes a weaker claim, comparing against the local OS
  filesystem over an operation sequence this repository chose rather than one nobody here wrote and
  cannot have tuned to what already works. Both notes point at README.md's supported-operations
  table for reading the output: ObjectFS is not POSIX-compliant, so a clean run is not the goal, and
  the useful question is whether the failing set grew.

• docs/index.md pointed at a deployments/ directory that does not exist, and never has —
  there is no deployments/ and no deploy/ in the tree. The real artifact is a single templated
  unit, configs/systemd/objectfs@.service, and the line now names it along with the part that is
  not guessable from the filename: systemctl start objectfs@research-data reads
  /etc/objectfs/research-data.yaml, which must set mount.uri, because one unit file serves every
  instance and the instance name is the only thing systemd passes it. Same class of defect as the
  GetPredictiveCache symbol in [#223] — a path in prose with nothing checking it. README.md
  already linked the correct file, so this was the only stale copy.

• Four duplicate link reference definitions removed from this file — #179 and #373 defined
  twice, #240 and #245 likewise. Reference definitions are document-scoped, so the later copies
  resolved nothing the first had not already resolved; markdownlint flagged all four as MD053 on every
  run. The first occurrence is kept in each case, which is the one in release-section order, and
  MD052 stays clean because every use still finds a definition. They survived this long because
  CHANGELOG.md is excluded from the markdownlint pre-commit hook — so the findings were real and
  blocked nothing, which is the condition under which lint output stops being read. One stray double
  blank line (MD012) in the v0.12.0 section went with them, for the same reason: it was in the way of
  reading the output that matters.

Fixed

• A Transfer Acceleration fallback was permanent for the life of the mount ([#204]).
  DisableAcceleration fired on the first acceleration error and EnableAcceleration had no caller
  anywhere in the tree, so every later request took the standard endpoint until ObjectFS restarted. A
  thirty-second DNS failure reaching the accelerate endpoint therefore cost a long-lived mount its
  acceleration for weeks, and nothing reported that it had happened.

  One request may now try the accelerate endpoint again after storage.s3.acceleration_retry. The
  mechanism is internal/circuit's breaker rather than a second bespoke recovery: withdrawn is its open
  state, probing is half-open with MaxRequests: 1, so exactly one probe is in flight at a time and a
  permanently broken endpoint costs one failed request per period rather than one per read — which is the
  cost that justified making the fallback one-way in the first place. The bound is what a mutex around two
  fields could not have expressed, and it is verified by mutation: raising MaxRequests to 64 fails
  TestOnlyOneProbeIsInFlightAtATime and nothing else.

• use_acceleration: true together with any endpoint: failed every read and write, permanently.
  The AWS SDK refuses the combination before a request leaves the process — A custom endpoint cannot be combined with S3 Accelerate — and that refusal is not a smithy.APIError, so it was not classified as
  an acceleration error and never triggered the fallback. Every GetObject and PutObject returned
  STORAGE_READ/STORAGE_WRITE for the life of the mount on every MinIO, Ceph, RustFS or Wasabi
  deployment that copied the acceleration example. It now falls back and keeps serving, silently, because
  acceleration is a performance capability and slower is a correct outcome.

  The first fix for this did not work, and the end-to-end test is what caught it: isAccelerationError
  matched against err.Error(), but the backend's translateError wraps the SDK error in an
  *errors.ObjectFSError whose Error() renders its own code and message and deliberately omits its
  cause — so what the classifier saw was [s3-backend:GetObject] STORAGE_READ: GetObject operation failed, with no trace of the ruleset message anywhere in it. It now walks the whole Unwrap chain.
  Two wrapping styles in this codebase behave differently under Error(), and a matcher over service
  prose silently stops matching the moment its input is wrapped by the second kind — which is every error
  the S3 backend returns.

  testaws.Fault grew a Message field for this. A fault could produce an S3 error code and not a
  message, so no injected fault could express a condition S3 reports only in prose, and the fallback
  branch was reachable only by calling the classifier directly — which proves the classifier and not the
  behavior.

• The predictive cache's statistics were computed on every read of every mount and discarded at
  unmount ([#223]). GetPredictiveStats existed and nothing could reach it: the mount holds its
  PredictiveCache as an opaque types.Cache — six methods about bytes, with GetLevelStats returning a
  types.CacheStats — so there was no accessor at any layer above it. MultiLevelCache now has
  GetPredictiveCache and PredictiveStats, the names docs/features/read-ahead.md had already
  described, and the mount publishes them to /metrics as objectfs_predictive_cache, one series per
  statistic under a statistic label. Both SDKs see them: sdks/testdata/metrics-scrape.txt carries the
  family, so a renamed statistic fails the Python and TypeScript suites in the same commit.

  The accessor alone would have been half the fix, because 14 of the 17 fields it returned were assigned
  nowhere. PredictionsTotal, PredictionsCorrect, PrefetchRequests, PrefetchWaste, the two
  eviction counters and both ratios were declared with JSON tags and never written, and PrefetchHits's
  guard was event.Hit && event.Prefetch where nothing in the tree ever set Prefetch — so it was
  unreachable rather than merely unwritten. A zero that reads as a measurement is worse than an absent
  one, which is the same defect as [#222] in a different subsystem, so the statistics are implemented
  along with the way to read them. Seven fields whose inputs the cache cannot observe were removed rather
  than left as zeros — LatencyReduction would need the latency of the read that did not happen.

  Attribution rests on a new bounded range ledger, because it cannot be recovered after the fact: a cache
  hit looks identical whether the bytes came from the application's own earlier read or from a prefetch. A
  read is credited to a recorded range only if the range contains it, and the range is consumed on
  the claim — a half-covered read was half-served, and a prefetch fetched its bytes once so it can be
  right once. Both rules make the counters undercount rather than overcount, which is the direction that
  does not flatter the prefetcher, and consuming is what keeps prefetch_efficiency from exceeding 1.
  Waste is counted at eviction, since that is the moment "nothing will ever read this" becomes knowable.

  Two honest zeros to expect. prefetch_* reads zero on a mount today while the prediction and eviction
  statistics populate, because initializeLevels builds the predictive cache with no backend to fetch
  through — the workers dequeue jobs and store nothing. And the family is absent, not zero, when there
  is no predictive layer to ask, as on a Redis-backed cache: zeros there would claim a predictor that
  never fires, which is a different statement from having no predictor.

  A ratio could also momentarily exceed 1, which is a value neither statistic can take. Both recording
  functions published their range into a ledger and then took the stats lock to count it, and a range is
  claimable the instant it is there — so under several readers against one prefetch worker, hits were
  credited against a denominator that had not been incremented yet. CI observed prefetch_efficiency at
  4, and forty local runs of the same test did not. Both now count before they publish. Neither lock can
  be held across both halves — the ledger is written by prefetch workers while a read holds the stats
  lock, and the reverse order exists too — so ordering the two uncontended sections is the fix, and it is
  the right direction: the remaining window makes a ratio briefly low rather than impossible, and a
  cumulative counter that has run past its bound stays wrong for the life of the mount.

• Latency percentiles were published as zeros, and the histogram they were meant to come from was a
  modulo rather than a bucketing ([#222]). DetailedOperationMetrics declared P50Latency,
  P95Latency and P99Latency with JSON tags and never assigned any of them, so anything serializing
  the struct reported a filesystem with no tail latency at all — the most flattering possible wrong
  answer, and one that reads as a measurement rather than as an unimplemented field. They could not have
  been computed from LatencyHistogram in any case: it was indexed by int(latency.Milliseconds()) % 100, which is a hash of the latency into 100 slots with no ordering, so 50 ms and 250 ms shared a
  bucket, 1 ms and 1001 ms shared a bucket, and every operation under a millisecond — the expected case
  for a cache hit — landed in bucket 0 along with everything at exactly 100 ms. An array indexed that
  way cannot answer "what fraction of operations were faster than N" for any N.

  The histogram is now 24 exponential buckets from 25 µs plus an overflow bucket, published by
  metrics.LatencyBucketBounds since counts without their intervals are not interpretable. That range is
  set by what has to fit in it: an L1 hit is tens of microseconds and a cold multipart GET is seconds, so
  no linear-in-milliseconds scheme resolves both ends. The three percentiles are estimated from it by
  interpolating within the covering bucket, as Prometheus's histogram_quantile does, and a rank landing
  in the overflow bucket saturates at the top bound — "at least this" rather than wrapping to a fast
  bucket, which is the failure direction that makes a slow filesystem look fast. Per-file metrics
  allocate no histogram and are documented as leaving the percentiles zero.

  GetOperationMetrics also stopped handing callers the live histogram. Its comment said it returned a
  copy to avoid races, but a struct copy copies a slice header, so the caller walked the array the
  recorder increments — harmless while nothing read the field, and a data race now that the percentiles
  give something a reason to.

• A read-ahead prefetch that fell behind the reader re-fetched bytes already cached. A prefetch is
  queued and the reader does not wait for it, so under load the reader can consume the front of the very
  range predicted for it before a worker picks the request up. Those reads are then finished — removed
  from the in-flight set that the existing trim consults — and the prefetch's full range is not a cache
  hit either, because its tail was never cached and a partial hit is a miss. Neither guard saw them, and
  the prefetch re-read the overlap: measured on CI, a 16 KiB file read in 1 KiB steps transferred 18432
  bytes, the last GET re-reading 2048 bytes two earlier reads had already fetched. The prefetch now
  advances past what the reader has consumed, and trims rather than skipping — dropping it outright also
  stops the byte count exceeding the file, by turning read-ahead off.

• A remote operation whose response was too large for a gossip datagram presented as a 30-second
  timeout instead of a size error ([#399]). Gossip is UDP, NodeResult.Data is a []byte that
  encoding/json base64-encodes, and the envelope and MAC are on top, so at the default 8192-byte
  max_gossip_packet a response carries at most 5802 bytes of object — measured against the real
  seal path, and an inflation of 38.9% rather than base64's 33%. The kernel's MaxRead is 128 KiB, so
  essentially every read is over the limit. The responder logged the refusal at Warn and returned,
  having sent nothing, so the requester learned only that nothing arrived and reported operation timed out waiting for remote response after its full timeout — a size failure, on a different host, at a
  level most deployments do not collect, which sends an operator to look at the network. The verdict
  fits in a datagram even when the bytes do not, so the failure is now sent in place of the data,
  naming the payload size, the limit and the setting that governs it. ErrMessageOversize is the
  sentinel a caller matches on.

  Raising max_gossip_packet is not the fix and was not applied: past the ~1500-byte path MTU a
  datagram is IP-fragmented and one lost fragment discards all of it, so a larger limit trades a clean
  error for intermittent loss that scales with size. Fitting object bytes through this transport is not
  the direction — [#142] warms from S3 and puts only metadata on the wire.

• SetBackend raced every operation a peer had asked for. It assigned
  cm.coordinator.backend while holding the cluster's mutex, and executeLocally read the field
  under no lock at all — from the gossip receive goroutine, which is where a peer's operation runs. Two
  different locks and an unsynchronized read of an interface value. The backend is now set and read
  through the coordinator's own accessors, and read once per operation rather than at each switch arm,
  so a put and its ETag read cannot land on two different backends.

  It stayed latent because every existing test injected its backend before Start, when the
  goroutine that reads the field does not yet exist — an ordering no caller is required to observe, and
  one that [#139] makes routinely false. Found by -race only once a test made an injection and a peer
  operation overlap, which is now TestClusterManager_SetBackend_IsSafeWhileAPeerOperationRuns.

• The Python SDK wrote configuration files the daemon refused to start on — every one of them.
  Configuration.to_yaml() emitted sixteen keys internal/config does not define, across seven
  sections, and LoadFromFile decodes strictly, so save_to_file produced a document that failed at
  startup naming the first key it hit. This affected the default Configuration() and all five presets,
  which means the SDK's documented path — build a config, save it, mount with it — could not work at
  all. Removed rather than added to the Go schema, because in each case the setting either had a real
  home under another name or had nothing to reach:

  • global.pid_file, global.daemon — ObjectFS does not fork, so there is no background mode to
    select and no forked child's pid to record.
  • storage.s3.timeout — an int of unstated unit; the real settings are
    network.timeouts.connect/read/write, as durations.
  • performance.read_ahead_size, performance.max_write_buffer — the first was removed from the Go
    side in v0.11.0 for naming a second read-ahead size beside performance.read_ahead.window_size
    ([#176]); the second is write_buffer.max_memory.
  • cluster.election_timeout, cluster.heartbeat_interval, cluster.join_timeout — these exist on
    internal/distributed.ClusterConfig, a disjoint type from internal/config.ClusterConfig with no
    conversion between them ([#139]). Nothing in sdks/ consumed any of the three.
  • security.tls_ca_path — no CA path in the schema's security block; trust configuration is the
    security.tls block.
  • monitoring.opentelemetry.headers — where an OTLP bearer token would go, in a document the loader
    rejected. An unloadable place to put a credential is worse than no key.
  • the entire fuse block — allow_other, allow_root, default_permissions, uid, gid, umask,
    replaced by the three keys the Go schema has: direct_io, keep_cache, sync_read. The removed
    six were doubly inert: discarded by the loader, and three of them are ones cmd/objectfs/doc.go
    already records as not settable because nothing on the adapter's mount path reads them.

  [#385] named three of the sixteen, and the other thirteen were found by the test that closes it —
  which is the argument for its shape. Each preset's to_yaml() output is committed under
  sdks/testdata/presets/, and internal/config's TestSDKPresetsLoadUnderTheGoLoader globs that
  directory and runs every file through LoadFromFile and Validate. It asserts the property — the
  Go loader accepts what the SDK writes — rather than comparing emitted keys against a list, because a
  list is a second copy of the schema and would have had to be right about all sixteen. The Python half
  compares rather than writes (OBJECTFS_UPDATE_FIXTURES=1 to regenerate), because a Go test reading
  committed files would otherwise stay green on stale ones. Verified by mutation in both directions:
  reintroducing one key fails the Python comparison on all six documents, and regenerating with it
  fails the Go loader gate naming the key and line.

• --config ./objectfs..staging.yaml failed with "path contains directory traversal."
  ValidatePath tested strings.Contains(cleanPath, "..") after filepath.Clean, which conflates
  "contains two dots in a row" with "escapes the working directory." Clean has already resolved every
  resolvable .. by then, so a surviving one can only be leading — but Contains also matches an
  adjacent pair inside a component, and those are ordinary names. A config file with a dotted
  environment suffix, a log file named run..1.log, a directory v1..2/ were all refused, with an
  error naming a cause that was not the reason. The check is now a leading-.. test, so ..foo and
  ... are not caught either — they are names too ([#384]).

  The table-driven test's only "dots in filename" case was config/app.config.yaml, whose dots are
  not adjacent, so it passed under the broken check and the correct one alike — a case that cannot
  tell the fix from the defect. Thirteen cases now cover both directions, and mutation-checking
  confirms they are independent: reverting to Contains fails six subtests, all false positives;
  deleting the check entirely fails six different subtests, all real traversal. A new
  FuzzValidatePath states the property instead of a case list — it walks the cleaned path's
  components tracking depth and asks whether resolving it leaves the starting directory — and finds
  the original defect in under a second. It also caught the same Contains mistake in the first draft
  of its own oracle, on /..0.

  The doc comment now says what the function refuses, which was the issue's second question and is
  narrower than the name suggests. Clean treats the root as its own parent, so /../etc/passwd and
  /var/../etc/passwd both become /etc/passwd: an absolute path never reaches the traversal check
  with a .. in it. With allowAbsolute: true — which all three callers pass, for an operator's own
  config, discount-file, and log paths — the only thing refused beyond an empty string is a relative
  path that climbs out. That is a typo check, not a security boundary, and it is written down in those
  words with test cases pinning Clean's behavior so the claim fails rather than quietly rots.
  SecureJoin got a note for a related reason: it joins an absolute element rather than refusing it
  (SecureJoin("/var/cache", "/etc/passwd") → /var/cache/etc/passwd), which is filepath.Join's
  documented behavior and still contained — the surprise is in the return value, not the safety.

• A compatibility probe reported a finding by failing the run. conditional_compat_test.go
  called t.Errorf when an endpoint accepted If-None-Match: * over an existing key and replaced its
  contents, which contradicts the suite's own contract: record what an endpoint does and fail only on
  what would be unsafe. That cell is safe, because it is exactly what the capability probe detects and
  refuses. The branch had never fired — AWS, MinIO, RGW and RustFS all enforce absence on PutObject
  — and Wasabi is the first endpoint to reach it, turning a correctly-refused endpoint into a red run.
  It is now a t.Logf("FINDING: ..."), and the assertion that matters is the one below it: the probe
  must report the capability unsupported.

• A flaky test that failed a CI run, in a helper written to make tests reliable.
  testhttp.FreeAddr binds 127.0.0.1:0, records the address and closes the listener — which
  returns the port to the ephemeral pool, so the kernel is free to hand it to something else before
  the caller binds it. In CI it did: TestStartMetricsBindsTheEndpoint failed with
  bind: address already in use on a port the miniredis in internal/adapter's own
  cache_selection_test.go had been given in the interval. Nothing in the test was wrong; the
  address was stale by the time it was used.

  Every caller that binds now configures port 0 and reads back where the kernel put it, closing the
  window rather than narrowing it. That means the bound port is not known in advance, so the
  where-it-bound assertion is now testhttp.SameHost — which is the assertion [#211] always turned
  on: fmt.Sprintf(":%d", Port) produced a wildcard host, publishing an unauthenticated
  /metrics on every routable interface, and a wildcard bind reads back as 0.0.0.0 or [::]
  rather than the loopback address configured. Verified by mutation: reintroducing the host-stripped
  bind fails both wiring tests naming the wildcard, and deleting the Serve goroutine fails the
  scrape. FreeAddr survives for the one caller that needs an address in advance and never binds it
  — the test asserting a disabled endpoint listens nowhere, which a competing bind cannot make pass.

  internal/health had the same reserve-then-release pattern in the #211 regression test, not yet
  triggered, and no way to fix it: nothing reported where the health listener bound. So
  health.Checker.Addr() now exists, matching metrics.Collector.Addr() — which also means an
  operator running the health endpoint on port 0 can find out where it went.

Removed

• internal/cost — a second cost-calculation package with no importer ([#226]). 710 lines and 540
  lines of tests: Calculator, Reporter, AlertManager, PriceTable, per-tenant accumulation, ROI
  reporting against a Standard baseline, and budget-threshold alerting with soft and hard limits. Zero
  importers outside itself, and no configuration path ever existed for the tenants or the budgets it was
  built around — a single-mount filesystem has no source of tenant identity, and being under internal/
  it cannot be consumed by another module either. With objectfs_s3_cost now publishing from the backend,
  the only caller it could ever have has a reachable path that does not go through it.

  Deleted rather than kept as the calculation layer, because two cost-calculation packages where one is
  unreachable is precisely the arrangement that produced [#209]: the same rate written in five places, two
  of them disagreeing by a factor of ten, so what a write cost depended on which package a caller reached
  for. Its pricing_drift_test.go was the guard against that shape and the half worth keeping moved to
  internal/storage/s3, where it now asserts that every rate a caller can reach through PricingManager
  is internal/awsrates' own value, exactly, for every storage class the config loader accepts —
  storage, PUT, GET, LIST and retrieval, not storage alone, since a partial regression that leaves requests
  on constants passes a storage-only check. Verified by mutation: a private PUT rate in the manager fails it
  with the ratio named, and a round factor of ten in that ratio is reported as what it is.

  One rate now has no consumer as a result: awsrates.EgressPerGB, whose only reader was the deleted
  calculator. It stays in the generated table, since it comes from AWS's price list and a mount that ever
  reports egress will want it, and the relocated guard says in a comment that there is no plumbing to check.

• metrics.RecordCost and CostMetrics ([#226]). Ten fields of per-operation cost on the detailed
  collector, populated by a method with no caller outside this package's tests. The issue asked to wire it
  or delete it but not to leave a third unreachable path; it is deleted, because the shape could not be
  usefully wired: it took request, storage and transfer costs as float64 dollars, so every price would
  have been decided by the call site rather than by internal/awsrates — the exact arrangement above. Two
  of its calculations were also wrong in ways no amount of wiring would have corrected. Cost per GB divided
  by 1 << 30 where AWS bills decimal GB, understating by 7.4%. And EstimatedMonthlyCost extrapolated
  from process uptime, so a mount thirty seconds old reported its first half-minute as the whole month's
  rate — a figure that is most wrong exactly when someone is most likely to read it.

  The rest of DetailedPerformanceMetrics stays. Its latency percentiles were assigned and its histogram
  bucketing fixed this release ([#222]), so only the cost half was dead. internal/metrics/doc.go and the
  docs/index.md row say so, including that per-operation-type cost is the one thing objectfs_s3_cost
  does not carry and that it belongs as a label on the tally rather than as a second collector.

• pkg/api — 12 declared HTTP routes nothing ever served ([#367]). 559 lines of handlers and 999
  lines of tests for /health, /health/components, /health/live, /health/ready, /status,
  /status/operations, /status/history, /api/v1/mounts, /info and the rest, with zero
  importers outside the package. A running mount's real HTTP surface is two endpoints: /metrics
  from internal/metrics and /health from internal/health. Nothing users could reach is gone,
  because nothing users could reach was ever there.

  Deleted rather than left waiting for a caller, because a declared-but-unserved surface is worse than
  an absent one: it produces documentation that cannot be checked against behavior. It already did —
  the six fabricated endpoints [#336] had to correct in the docs playground looked plausible because
  a package in this tree declared their shapes, and a reviewer comparing docs to code would have found
  the routes and stopped there. The docs/index.md "not yet wired up" row is replaced with a note
  saying it was deleted and why; a dead /api/rest sidebar entry in docs-platform (a link to a page
  that never existed) went with it; and the "REST API" box in
  docs/ARCHITECTURE_EVOLUTION.md's Phase 2 diagram is annotated rather than redrawn, since that file
  is explicitly a proposal and its header already lists where it diverges from the code.

  The pkg/api 73 floor is out of .coverage-floors. Its 999 lines of tests were the reason the floor
  was as high as it was, which is worth stating: a package can be well tested and still not be part of
  the product, and coverage cannot tell the difference.

---

Verify a download:

sha256sum -c objectfs-<platform>.tar.gz.sha256

Container image: ghcr.io/scttfrdmn/objectfs:0.13.0

Full changelog: https://github.com/scttfrdmn/objectfs/blob/v0.13.0/CHANGELOG.md