Skip to content

5.1

Latest
Latest

Choose a tag to compare

View all tags
@alex-aizman alex-aizman released this 28 Sep 18:22
· 147 commits to main since this release
v1.5.1
6eb578a

AIStore 5.1 is a short-cycle reliability and scalability release following the major 5.0 transition. It closes the two-release security rollout opened by 5.0: the bridge gate is removed, per-node Ed25519 request signing and startup node-join authentication become active when configured, and pre-5.0 nodes are refused.

The main themes are:

  1. Intra-cluster security. The cryptographic mechanisms staged in 5.0 can now be enabled. Nodes publish verifying keys, protected requests and transport-stream setup are signed, configured startup join credentials are enforced, and the 5.0 version boundary is checked in both join directions.

  2. List-objects scale-out. Listing very large buckets in large clusters no longer falls on the primary gateway. Listings are distributed across all gateways - consistent with the way the AIS datapath has operated since inception.

  3. Shard index v2 and on-demand cache. A new ordered, packed representation removes the decoded per-entry map, substantially reducing allocation and load cost. A new memory-aware cache is aimed primarily at Get-Batch over large TAR datasets, where many member reads repeatedly revisit the same shards. End-to-end cache performance characterization is still pending.

  4. Blob download and prefetch. Large remote-object downloads now have explicit size and memory admission rules, parent-to-child cancellation, and cleanup of buffers, manifests, and object locks on failure.

  5. Reliability under failure. Erasure coding completes every destination when one of them fails and rebuilds peer streams after node restarts; global rebalance continues after per-object failures.

  6. kTLS transmit offload (experimental). An opt-in path, for controlled evaluation, that lets HTTPS GET transfers (i.e., AIS transmit) use sendfile(2) while the kernel - or a capable NIC - encrypts TLS records. Large-scale benchmarks are planned.

  7. Python SDK 2.0, released alongside 5.1. Interrupted ObjectFile reads now resume without gaps or duplicate bytes even when the next response starts at byte zero rather than the requested offset. The SDK also improves parallel reads, batch extraction, ETL, and PyTorch integration. The new major version reflects its compatibility changes: the minimum Python version is now 3.10, and several previously announced API removals take effect.

Separately, 5.1 sets the objective for chunked storage - a third storage layout that, like n-way mirroring and erasure coding, is governed by bucket configuration - and deprecates the per-job overrides that contradict it. Implementation is planned for v5.2.

Upgrade boundary: v5.0 remains the mandatory bridge from every 4.x release. Upgrade your 4.x cluster to 5.0 before starting 5.1; direct 4.x-to-5.1 upgrades are rejected. See Upgrade Notes below for other compatibility changes.

Table of Contents

  1. Security and Upgrade Completion
  2. List Objects
  3. S3 Compatibility
  4. Object Metadata and Response Headers
  5. Shard Index v2 and On-Demand Cache
  6. Erasure Coding
  7. Global Rebalance
  8. Chunked Storage
  9. Blob Downloader and Prefetch
  10. Internet Downloader
  11. Jobs
  12. Memory Pressure
  13. kTLS Transmit Offload (experimental)
  14. Socket Buffers
  15. Native Go API
  16. Python SDK, ETL, and ML
  17. Miscellaneous Improvements and Fixes
  18. Documentation
  19. Removed and Deprecated APIs
  20. Upgrade Notes
  21. Acknowledgments

Security and Upgrade Completion

v5.0 established the new network and metadata boundaries but deliberately suppressed the cryptographic mechanisms while 4.x nodes could still be present in the cluster. v5.1 removes that bridge behavior and completes the transition.

Per-node request authentication

Every 5.1 node generates an Ed25519 keypair at startup and publishes its verifying key through the cluster map. When auth.intra_cluster.request_auth is enabled, AIS signs and verifies protected proxy-to-target redirects, node-to-node control requests, and transport-stream connection setup.

The signing key remains in memory only. Restarting a node rotates its key; cluster-map propagation distributes the replacement. Transport bundles treat a changed verifying key, network endpoint, or membership state as a new peer incarnation and reopen the affected streams.

The setting remains opt-in. With request_auth disabled, the arrival-network checks and proxy-mediation rules introduced in 5.0 still apply: when auth.client_auth_required is enabled, targets reject unmarked requests sent directly to their public endpoints. An unsigned redirect marker can, however, be supplied by a client and is not authenticated at the receiving target. auth.client_auth_required protects requests at proxies; it does not independently authenticate redirected requests at targets.

Network path and identity: AIS records each request's arrival network and enforces the corresponding routing checks even when signing is disabled. This establishes where a request entered, but cannot prove that a public redirect marker came from a gateway. To keep client authentication effective across redirects, enable both auth.client_auth_required and auth.intra_cluster.request_auth after completing the upgrade.

Enabling or disabling request_auth requires no cluster or node restart, and no downtime of any kind. Nodes sign immediately when it is enabled and accept unsigned traffic for a bounded convergence window before becoming strict. The same mechanism provides a short startup window while verifying keys converge after a full-cluster restart; it is not intended to make a long-running 5.0/5.1 mixture secure.

Node-join authentication

When auth.intra_cluster.node_join_secret_path is configured, 5.1 reads the locally provisioned secret and uses a timestamped HMAC proof to authenticate startup self-join in both directions:

  • the primary verifies the joining node before admitting its identity and public key;
  • the joining node verifies the primary before accepting the returned cluster metadata.

The secret itself is never transmitted or persisted in AIS metadata. The configured pathname is cluster-wide, while the file must be provisioned locally on every proxy and target with restrictive permissions. Slow keepalive paths that would admit an absent or restarted node use the same proof.

The secret is read once during node startup and is not reloaded by a cluster-configuration update. Setting or changing the path persists the new configuration but does not affect running processes until they restart.

Administrative-join and force-join authentication remain outside the implemented boundary. See Node-Join Authentication for the exact status, provisioning order, credential requirements, and recovery procedures.

Version boundary

Joining nodes and primaries exchange their respective AIS software versions. A 5.1 node refuses a pre-5.0 primary, and a 5.1 primary refuses a pre-5.0 node through both self-join and administrative join. Missing version headers are treated as pre-5.0.

This closes the two-phase sequence introduced by 5.0: the bridge release accepted 4.x peers long enough to cross the metadata boundary; 5.1 depends on that boundary and fails a skipped-bridge upgrade with a specific compatibility error.

Additional enforcement

Proxy mediation now covers node-control queries as well. Previously, a target would answer requests for its configuration, status, and runtime information sent directly to its public endpoint - even on clusters that require client authentication, which only gateways enforce. Such requests are now rejected, the same way as direct bucket and object requests. Node control goes through a gateway, which authorizes the request before relaying it to the target over the intra-cluster network.

Two related changes concern the control plane itself. A control operation now uses one cluster-map snapshot from sender validation through downstream dispatch, so identity decisions within one request cannot straddle map versions. And relayed requests clear system-controlled intra-cluster headers before AIS constructs the next hop.

For the complete request matrix and the distinction between proxy mediation and data-plane authentication, see AuthN: Authentication Boundaries.

Object names and append handles

Both fixes below originated in reports from outside the project (see Acknowledgments).

Object names are now validated wherever AIS turns them into on-disk paths, not only at the gateway: copy and transform destinations (single- and multi-object), rename, promote, archive destination shards, list- and range-based operations such as prefetch, evict, and delete, names returned by a remote listing, and blob download. Names containing path-traversal components are rejected before they can reach the filesystem.

Append handles are bound to the object that created them. A handle carries only a validated tie-breaker for the object's work file; the target derives the work file from the current request's bucket and object, instead of accepting a path encoded for another object.

Commits

  • 9e4617f5f: cross the v5.0 bridge and enable intra-cluster security
  • bae0c2a19: rebuild EC peer streams after restart and membership changes
  • 2dc9c94b8: propagate cluster-map snapshots across control paths
  • 3e2b18579: require proxy mediation for public target node-control requests
  • 5b7b51267: cover redirect-marker verification
  • 5059be970: remove pre-5.0 compatibility leftovers; reserve feature bit 0
  • 96b111f54: validate object names wherever they become on-disk paths
  • ae2a2716f: bind append handles to their object

List Objects

Scale-out across gateways

Until now, every paged listing - native or S3 - was forwarded to the primary gateway. Listing very large buckets in large clusters therefore fell squarely on a single node, however many gateways the cluster had.

Starting with 5.1, the work of listing large buckets is spread across all gateways (proxies) instead of being concentrated on the primary, consistent with the way the AIS datapath has operated since inception. Each listing is identified by a UUID assigned with its first page; the UUID determines which gateway owns the listing. Any gateway accepts any page and hands it to the owner, so a client - or a load balancer in front of the cluster - can use whichever gateway it likes, page after page.

S3 clients need no changes: the listing UUID travels inside the S3 continuation token, which remains opaque (see S3 Compatibility).

The primary is still involved in one case: listing a remote bucket that is not yet present in the cluster's bucket metadata.

Improvements and fixes

Targets now report the end of a listing explicitly, rather than leaving the gateway to infer it from page length. A walk error on a target is sticky: it remains visible after traversal finishes instead of disappearing with it. Non-recursive listings, which merge and deduplicate virtual directories across targets, now detect page overflow correctly.

Prefix validation and request-error accounting are the same for native and S3 requests. An aborted remote-bucket listing startup, and listing a remote bucket without adding it to bucket metadata, no longer leave ambiguous routing state behind.

S3-specific changes - max-keys, delimiters, and continuation tokens - are described in S3 Compatibility.

Commits

  • 79e040dd2: distributed proxy ownership and the new execution flow
  • a00f71bec: contextualize proxy-side listing execution
  • 5dc1c0e79: report end-of-listing from targets
  • 6c5826460: merge/dedup overflow and sticky target errors

S3 Compatibility

S3 object keys containing ? now work by default. AIS previously reinterpreted any ? in the decoded request path as a query delimiter to accommodate TensorFlow clients that embed parameters in encoded object paths. This could address a different key, including on PUT and DELETE. The legacy behavior is available via the opt-in cluster feature S3-TensorFlow-Query; enable it only for workloads that require that TensorFlow query format. See S3 Compatibility: Question marks in object names.

ListObjectsV2

The S3 API now returns exactly one page per request, and max-keys is the effective page size. When omitted, it defaults to 1000 or the bucket's maximum page size, whichever is smaller; larger values are capped; max-keys=0 returns an empty page without listing anything; negative and non-numeric values return InvalidArgument. The only supported (nonempty) delimiter is /; other values are rejected.

Continuation tokens returned by the S3 compatibility API remain opaque. In 5.1, such a token carries the listing UUID (see List Objects) together with the underlying marker, which may be an AIS object name or a remote backend's own token.

Request bodies and errors

XML request bodies are now bounded: multi-object delete at 8MiB and bucket versioning at 4KiB, for both declared and streamed lengths. Oversized requests fail with HTTP 413 instead of growing memory without bound.

When a gateway fails to relay an S3 object request, the client receives an S3 XML error with status 502 rather than a bare response.

Commits

  • 99b821149: preserve ? in S3 object keys; opt-in S3-TensorFlow-Query
  • 7782ceec8: honor S3 max-keys and paginate list-objects
  • 7453623b0: compound S3 continuation tokens
  • 087592f05: bound S3 XML request bodies
  • 17ded3a05: XML-encode S3 relay failures

Object Metadata and Response Headers

Target response headers for GET, PUT, and S3 HEAD are now produced in one place for both the native and S3 APIs, which fixes several differences that S3 clients could see: a streaming cold GET now returns a quoted ETag, as S3 requires; a cold HEAD returns the backend's checksum; and a GET served by the blob downloader carries the remote object's Last-Modified.

Starting with 5.1, AIS stores a remote S3 object's Content-Type with the other supported remote attributes, and returns it as an HTTP header on GET. A cold GET captures the type reported by S3; an S3 PUT into an ais:// or s3:// bucket also stores a non-default type. S3 HEAD returns it as well, and so does HEAD(object) when the object is present in the cluster. For an object not yet in the cluster, HEAD(object) does not return Content-Type. The defaults (application/octet-stream and binary/octet-stream) are not stored. S3 multipart uploads and the GCP, Azure, and OCI backends are to follow.

A HEAD(object) that checks the latest version with a cloud backend can also refresh remote-owned attributes - Content-Type, Last-Modified, ETag, version, and MD5/CRC32C checksums - when the in-cluster object still matches the remote one but its copy of those attributes is missing or stale. Objects cached before 5.1 can thus pick up their Content-Type without being fetched again. The update is best-effort: it is skipped if the object is busy or changes before the write lock is taken. When the in-cluster object does not match the remote, HEAD returns a conflict (HTTP 409) and updates nothing.

Local regular and multipart S3 PUTs preserve x-amz-meta-* user metadata, and metadata returned by remote S3-compatible backends is propagated as well; AIS-internal keys are excluded from user metadata.

Commits

Shard Index v2 and On-Demand Cache

Shard indexes accelerate reading archived files - individual files packed inside TAR shards - by mapping file names directly to their offsets and sizes within the shard. v5.1 changes both the persisted representation and the repeated-read behavior while keeping the external GET, Get-Batch, and CLI APIs unchanged. The main expected beneficiary is Get-Batch over large sharded datasets, where a request commonly selects only a few members from each TAR while many requests revisit the same shards.

Version 2 representation

Version 1 kept the packed index and also decoded every entry into a map[string]Entry. That front-loaded proportional allocation even when a request needed only one member from a large shard.

Version 2 stores entries in strict name order and keeps the packed payload plus a []uint32 offset table in memory. Lookup binary-searches names in place and decodes offset and size only for the selected entry. Eligible packed and offset buffers use size-classed pools above the page-slab ceiling.

The v2 payload fields and uvarint encoding remain the same, but sorted order is now an invariant and the checksum covers the preamble kind as well as the payload.
Unit-level measurements documented in Shard Index show roughly 2x faster 10K-entry loads and substantially lower steady-state allocation traffic; the primary benefit is avoiding construction of the decoded map.

On-demand cache

The cache is new in 5.1. It is enabled automatically and needs no tuning: it is governed entirely by available memory, growing and shrinking on a best-effort basis and gradually evicting indexes that have not been used for a while.

On the first lookup, AIS loads and validates the compact v2 index; simultaneous cold readers share that work. Subsequent reads reuse the loaded index, binary-search the requested file, and read its byte range directly from the TAR. A regular GET of an archived file uses the same path, but Get-Batch is the primary intended consumer.

The cache has not yet been benchmarked end to end or run in production. Its benefit depends on shard size, member count, working-set locality, and how often requests revisit the same shards within the retention window.

Loaded indexes remain in a per-mountpath cache keyed by object identity and bucket generation:

  • admission, idle aging, and eviction respond to memory grade;
  • high pressure stops new admissions and shortens retention; critical pressure clears the cache;
  • cached references are dropped before returning memory to the operating system;
  • replacing, removing, rebuilding, or invalidating the source shard removes its cached index.

When a target is at high memory pressure, a shard whose index is not already cached is read by sequential archive traversal - the same path as an unindexed shard - rather than by loading the index uncached. This deliberately favors memory stability over lookup latency and can make indexed reads slower than in 5.0 under sustained pressure.

Cache activity (misses, loads, denied admissions, evictions, clear-all passes) is logged at verbosity level 4 or with the cluster log module enabled (ais config cluster log.modules cluster). A growing deny count indicates reads falling back to traversal.

Index creation is treated as a transaction. AIS revalidates the source before committing the index, and never marks a shard as indexed when its checksum or size no longer matches. Loading distinguishes absent, stale, corrupt, and transient failures; malformed or truncated encodings fall back without panicking. Shard-index metadata follows source-object replacement, local copy, and chunk relocation, without inheriting flags that belong to the old object instance.

Existing indexes

Indexes written by v4.5 through v5.0 have metadata version 1. A 5.1 target classifies an intact v1 index as stale, clears the source marker, and falls back to sequential archive traversal. Object reads continue to work but are not index-accelerated until rebuilt.

Rebuild existing indexes without --skip-verify:

$ ais bucket shard-index build ais://dataset --prefix shards/ --wait

See Shard Index: Upgrading for summary output and rebuild details.

We are actively working on on-demand (re)indexing - indexing new writes (PUT, cold GET, copy) into an indexed bucket, and rebuilding v1 indexes, automatically - and plan to include it in the very next release.

Commits

  • 33219cda9: loading, recovery, persistence, and malformed-input handling
  • 8c336d790: v2 on-disk and in-memory representation
  • 36206099b: ownership checks, pooling boundaries, and v1 compatibility
  • a659515aa: on-demand in-memory cache
  • e46d9a80b: cache eviction and transactional indexing
  • fea74ec9b: shard-index metadata across chunk relocation

Erasure Coding

EC operates over intra-cluster peer-to-peer streams: to protect an object, a target sends its replicas or slices to multiple peer targets at once. Previously, a failure at one destination terminated the entire send. The remaining destinations were never completed, the error could get lost, and a reader could be accessed after it had been recycled. EC now attempts every destination, reports the first error, and lets go of each reader only after the transport is done with it.

EC also keeps its peer streams current. Streams are now rebuilt when a peer target restarts, when cluster membership changes, and when a peer's network endpoint changes - previously, a stale stream could outlive the peer incarnation it was opened for.

Encoding is faster and stricter. Data-slice checksums are computed within the same Reed-Solomon encode pass, removing a full scan of the source; for a 512MiB object with 4 data and 2 parity slices, checksum plus encode runs roughly 2x faster. A short data-slice read now fails with io.ErrUnexpectedEOF even when checksums are disabled.

Commits

Global Rebalance

A read failure on a single object could previously cut short the traversal of the remaining mountpath, or skip later buckets altogether, while the job still reported completion with zero errors. Rebalance now records the error, counts it (err-read), and continues - stopping only when cluster-map placement itself cannot be computed.

Rebalance cleanup now verifies every misplaced source copy through one consolidated path. By default, AIS removes the source only after confirming that the copy on its owner represents the same object. If the peer copy is absent or differs, AIS retains the source; an explicitly forced cleanup may remove a divergent source.

Commits

  • e87fd62ef: continue rebalance traversal after per-object failures
  • 1c2dedcbe: consolidate per-object cleanup verification

Chunked Storage

The objective: chunked objects become the third storage layout, in addition to n-way mirroring and erasure coding, implemented consistently across the entire functionality and configuration - and, like the other two, governed by bucket configuration. A bucket's chunks properties are then the single, authoritative statement of how that bucket's objects are stored. Cluster-administrative jobs that operate on many objects converge the data to that configuration, the same way re-mirroring and re-encoding converge it to mirror and ec, rather than override it.

That is the plan for v5.2 (see Upcoming changes below). v5.1 articulates the objective and identifies where current behavior departs from it - most notably rechunk, which can impose its own chunk sizing on a bucket configured otherwise.

Single-object operations are a different matter. A client-side multipart PUT, an S3 multipart upload, or a single-object blob download (started explicitly or selected via api.GetArgs.BlobThreshold) can choose its own chunk size for that one object. This is a feature - deliberate, advanced usage scoped to a single object. The per-object chunk sizes - including apc.BlobMsg.ChunkSize, --chunk-size with ais blob-download or ais get --blob-download, and the blob-downloader default when omitted - remain supported and are not affected by the changes below.

Apart from the copy and size-validation fixes below, v5.1 changes no storage behavior. It deprecates the per-job rechunk overrides that contradict the objective, and emits warnings where a request depends on them.

Copy and chunked writes

chunks.max_monolithic_size is the bucket's hard limit: PUT stores any larger object as chunks, regardless of the rest of the chunks configuration. Object copy now follows the same rule. Previously, copying into a bucket with a lower limit - including a copy within the same bucket after the limit was lowered - could leave an oversized object monolithic. Such a copy is now stored as chunks of the destination's chunks.chunk_size, including a copy that reads a remote source through the same target. The same change fixes the previous cross-bucket path, which left the source object read-locked after copying it.

ETL transformations remain an exception when the output size is unknown before the write: AIS cannot use the destination's hard limit to choose a layout in advance. If the transformed size is known, the copy follows the same hard-limit rule.

For remote copies, AIS now uses the source object's reported size to decide whether to store the destination as chunks. Multipart-upload parts check the bytes received against their declared size; a copy stored as chunks fails if its source ends before the reported size. AIS closes the source before completing the chunked object and releases it on no-op and error paths as well. A copy that switches from monolithic storage to chunks keeps the source metadata and version; an already-chunked source is not re-chunked solely because it exceeds the hard limit.

Prefetch: blob-downloader chunking

apc.PrefetchMsg.BlobThreshold and apc.PrefetchMsg.BlobChunkSize are not deprecated, but their semantics change in v5.2. The threshold selects how an object is fetched - a regular cold GET or the parallel blob downloader - and does not itself prescribe how it is stored. With auto-chunking enabled, storage layout follows the bucket's chunks configuration regardless of fetch method; the auto-chunking-disabled case below is a deliberate exception.

In v5.2:

  • BlobThreshold == 0 disables blob-downloader selection (unchanged; the previous API comment incorrectly described zero as "server default").
  • With BlobThreshold > 0 and bucket auto-chunking enabled (chunks.objsize_limit > 0), blob download applies only at or above max(BlobThreshold, chunks.objsize_limit) and uses chunks.chunk_size; BlobChunkSize is ignored, with a log message.
  • With auto-chunking disabled, blob download applies at or above BlobThreshold and uses BlobChunkSize (or the blob-downloader default). This is a deliberate exception: chunks.objsize_limit == 0 means automatic writes remain monolithic, but reassembling parallel-fetched ranges into a monolithic file degrades performance. Prefetch keeps the chunked result, warns once per job, and suggests ais bucket rechunk.

For the complete rules - storage layout by object size, and the prefetch mechanism - see Chunking.

Upcoming changes

Planned for v5.2, implementing the model above:

  • Rechunk converges only. Objects whose layout already conforms to the bucket's chunks configuration are skipped rather than rewritten. This includes enforcing chunks.max_monolithic_size for existing monolithic objects, which rechunk currently leaves as is.
  • Changing chunks properties starts rechunk automatically, as changing mirror or ec properties already starts re-mirroring or re-encoding. A change to chunks.objsize_limit, chunks.chunk_size, or chunks.max_monolithic_size aborts any running rechunk on the bucket and converges it to the new configuration; ais bucket rechunk remains for incremental (--prefix) conversion and --sync-remote.
  • PUT, cold GET, and copy honor chunks.objsize_limit, storing objects at or above it chunked - currently only rechunk does.
  • The deprecated apc.RechunkMsg.ChunkSize and ObjSizeLimit are removed, and prefetch adopts the blob-downloader semantics described above.

Commits

Blob Downloader and Prefetch

The blob downloader fetches a single large remote object using concurrent range reads (ais blob-download, ais get --blob-download, or prefetch above a size threshold). Not to be confused with the Internet downloader, which runs multi-object download jobs.

Memory and lifecycle

Blob download now accounts explicitly for in-flight streaming memory and uses the same effective-free-memory view as the memory manager. Critical pressure rejects all blob downloads; high pressure rejects streaming downloads, while eligible prefetch requests can fall back to ordinary cold GET.

The job holds the object's write lock through manifest commit or abort cleanup, stops a streaming download when its client disconnects, propagates parent cancellation into range requests, bounds chunk-read timeouts by timeout.send_file_time, honors latest-version policy, and returns synchronous runtime failures when the response has not already been committed. Remote-AIS range requests now retain their contexts across the backend hop. Chunk-size bounds are aligned with bucket chunks configuration: a shared 32KiB minimum, up to 5GiB for cache-only downloads, and 64MiB for streaming.

Teardown - aborting an uncommitted manifest, releasing the memory claim, unlocking the object, and finishing the job - now runs exactly once on every exit path, including a panic, so a failed download can no longer leave its object write-locked. Buffer ownership between out-of-order chunks and final cleanup is fixed as well, removing a double free.

Prefetch

Prefetch is the blob downloader's main background caller: objects at or above --blob-threshold are fetched by parallel range reads, smaller ones by regular cold GET. Much of the blob-downloader work in this release is about making that pairing predictable.

With --blob-threshold set, prefetch no longer checks objects already in-cluster with the backend. Previously, it issued a remote HEAD for every such object - a shortcut to find object sizes - which also implicitly applied the latest-version check. Now the backend is consulted only for objects not in-cluster, and only when their size decides between blob download and cold GET; checking in-cluster objects remains an explicit option (--latest). The choice is re-checked once the object is write-locked: if a concurrent GET has brought the object in meanwhile, there is nothing left to download.

Prefetch tracks its blob-downloader children from the moment they are scheduled, accounts for each by the object's actual size, and admits them according to current CPU and memory load. Aborting prefetch cancels its in-flight cold GETs and aborts pending blob-downloader children within a bounded time. Once its own work is done, prefetch waits up to 32 minutes for its children, then aborts those still running.

Job status now shows how many large objects went through cold GET instead of blob download, and the log says why. A size mismatch between HEAD and GET is logged as a warning.

Minimum sizes are now explicit per interface: any non-zero size for the single-object job, max(--blob-threshold, 1MiB) for prefetch, and max(threshold, 128KiB) for streaming GET. Zero-size objects never go to the blob downloader, and a negative prefetch threshold is rejected. See Blob Downloader: Size limits.

How prefetch selects chunk sizes in v5.2 is described under Chunked Storage.

Commits

Internet Downloader

The Internet downloader (ais download) runs multi-object jobs that fetch datasets from HTTP(S) sources and remote buckets.

Not to be confused with the blob downloader. The Internet downloader brings in many objects - from arbitrary HTTP(S) sources as well as remote buckets - fetching each one whole. The blob downloader fetches a single large object from a remote bucket, splitting it into concurrent range reads.

When the downloader cannot accept another job, the request now fails with HTTP 429, and the job that was never accepted is removed instead of being left behind. Housekeeping retains running and aborting jobs, while rejected and partially constructed jobs release their rate limiter without going through the full cleanup of an accepted job. Removing a job is now serialized with concurrent removals, status, and list requests.

Commits

Jobs

In AIS, a job (also called an xaction) is an asynchronous operation. Some jobs run across the cluster; others, such as a single-object blob download, run on one target. The changes below concern how jobs are identified and tracked.

For multi-object archive, copy, and transform jobs, targets independently derive the same job (xaction) ID without an extra round of coordination. They now use the gateway's two-phase begin timestamp rather than their own clocks, which removes one way they could disagree on the ID: targets processing the same begin a few milliseconds apart, failing the job with ErrBusy ("xid mismatch"). The job registry again reports live jobs after its history exceeds the cap; it also gives back excess capacity after bursts.

Get-Batch counts successful zero-size objects and archived files while continuing to exclude per-entry failures, and its phase-two GET response carries the job ID header, as other job endpoints do.

Commits

Memory Pressure

Working well under extreme load - and never breaking - has been an AIS requirement since v1.0, and memory-pressure handling has evolved with it over many releases. This release continues with consolidation: metadata caching, erasure coding, mountpath walks, space cleanup, LRU, and list-objects all consult the same load advice, and housekeeping intervals scale with memory pressure. Code that holds large allocations can release them before AIS returns memory to the operating system, and oversized pooled buffers and reset slices release their large backing allocations instead of keeping them alive behind a shortened slice header.

Commits

kTLS Transmit Offload (experimental)

AIS targets can now use kernel TLS transmit (kTLS_TX) on their public HTTPS listeners, on a best-effort basis. AIS continues to use Go's crypto/tls for the handshake and receive path, then installs supported transmit keys into the socket. Eligible warm, monolithic object GETs can use sendfile(2) while the kernel emits encrypted TLS records.

kTLS supports TLS 1.2 and TLS 1.3 AES-GCM cipher suites. When kTLS cannot be armed because of the kernel, protocol version, cipher suite, or connection state, the connection remains on userspace TLS. AIS serializes the transition and subsequent transmit operations and wipes derived key material after every handshake outcome.

"Experimental" refers to mileage, not to the state of the code: every limited benchmark so far has run without failures. What is missing is a large, well-organized benchmark of the right workload on a correctly tuned setup - and possibly Go 1.27, whose net/http drives the handshake and application-protocol negotiation (ALPN) on wrapped TLS connections. Both are coming.

The capability is off by default and temporarily uses the restart-required System-Reserved-KTLS feature bit:

$ ais config cluster features System-Reserved-KTLS

Why this name: the bit is deliberately marked system-reserved - a temporary switch for evaluation rather than a stable feature. Once kTLS has enough mileage, we expect to replace it with a regular setting or to enable kTLS by default. Hence, the name, meaning, and polarity may change; do not build deployment automation around it yet.

Feature configuration is a complete list: include any other enabled feature names in the same command.

It currently applies only to target public listeners. While configured, those listeners use HTTP/1.1 and disable TLS session tickets even for connections that fall back to userspace TLS.

We intend to run large-scale kTLS benchmarks soon and publish the results. Throughput and CPU results depend on object size, concurrency, kernel, cipher suite, and whether the kernel performs encryption in software or offloads it to a capable NIC. Early internal measurements point in one direction: the main effect is lower target CPU per transferred byte, and most of it comes from NIC TLS transmit offload; kernel software encryption alone saves comparatively little.

When evaluating, measure target CPU per GiB, not throughput alone: once the network is saturated, the gains appear only as lower CPU. Report software kTLS and NIC-offloaded kTLS as distinct configurations, and confirm the NIC path with the NIC's TLS transmit counters.

TLS version matters for NIC offload. AIS deliberately has no server-side TLS version setting: targets accept both TLS 1.2 and 1.3 and honor the client's choice, and most clients negotiate 1.3. On many current kernels and NIC drivers, TLS transmit offload is available for TLS 1.2 only; a TLS 1.3 connection armed with kTLS is then encrypted by the kernel in software.

See HTTPS: kTLS transmit offload for host prerequisites, supported paths, fallback behavior, and diagnostics.

Commits

  • 9fc6462f2: initial kTLS TX and HTTPS sendfile(2) path
  • 770c4303e: stage-aware unsupported-kernel classification
  • a7efcc29c: key wiping, shutdown serialization, and connection retirement
  • 7bf39e9a7: cache kTLS state in the GET fast path

Socket Buffers

AIS storage nodes can now explicitly leave TCP socket-buffer sizing to the operating system. The cluster setting net.l4.sndrcv_buf_size sizes the kernel send and receive buffers (SO_SNDBUF, SO_RCVBUF) of target data connections: the public listener, the intra-cluster data listener, and intra-cluster transport streams. Proxies always keep the operating-system defaults.

A positive value sets the buffer size explicitly, and zero means the 128KiB default, as before. The new value -1 selects kernel autotuning, which grows buffers as needed - typically beneficial on high-bandwidth, high-latency paths. Previously read-only, the setting is now part of the cluster configuration that can be updated; the change takes effect after a cluster restart:

$ ais config cluster net.l4.sndrcv_buf_size=-1

Commits

Native Go API

AIS supports two APIs: its own native API and S3 compatibility. Native API clients are available in Go and Python; this section covers Go, and the Python SDK follows below.

In the Go API, api.GetArgs adds Context, so ordinary and remote-AIS GETs carry cancellation and deadlines through backend reads. The xaction wait APIs aggregate the instances that actually reported, preserve terminal status, distinguish idle from finished, and return on abort; an abort return does not mean cleanup has finished, and targets that do not report the job ID are ignored. Checksums survive a JSON round trip; previously a decoded checksum appeared absent, and callers silently skipped checksum comparison. Blob-download and list-objects errors now reach the caller instead of hiding behind partial success or follow-up polling.

Commits

Python SDK, ETL, and ML

Python SDK 2.0.0 brings both compatibility changes and a broad correctness pass; the major version reflects the breaking changes below. See the Python changelog for the complete API-level list.

Compatibility changes

The minimum supported Python version is now 3.10; Python 3.8 and 3.9 have reached end-of-life and are no longer tested, and 3.10 through 3.14 are supported. Object.head() now uses the selective object HEAD API and returns ObjectAttributes rather than a header mapping; the legacy HEAD behavior, the experimental Object.head_v2(), and ObjectAttributesV2 are removed.

Dependency minimums move up for upstream security fixes (requests>=2.33.0, urllib3>=2.8.0) and for Python 3.14 wheels (xxhash>=3.6.0,<5). The unused torchdata dependency is dropped from the PyTorch extra and pyaisloader; applications that import it directly must install it themselves.

ObjectFile read resumption

An interrupted ObjectFile read may request the remainder with a byte range, yet receive a complete response starting at byte zero. This occurs for archive members and ETL output, for some range forms, and when a target ignores the range. The reader now tracks the byte position delivered to the application separately from the position consumed in each new response. It detects whether the response actually starts at the requested offset and discards any replayed prefix, so a retry does not duplicate bytes. An uncached remote object also restarts from the beginning rather than attempting a range resume while the object is still being cached. A clean but short response is retried when its expected end is known.

Parallel reads, retries, and batch results

Parallel range GETs validate every response length and stop cleanly after a failure, never releasing shared memory while dispatched work is still in flight. A retried streamed upload rewinds to its starting position, or fails explicitly when it cannot. TAR and ZIP batch extraction reports failed entries as failures rather than as valid empty objects, and preserves both duplicate ZIP names and opaque tracking data. Listing no longer re-copies accumulated entries with every page, and can list buckets across all providers.

ETL and PyTorch

Kubernetes Pod-spec ETL initialization is removed on both server and Python sides. Use runtime specifications through Etl.init() or pure-Python Etl.init_class(); persisted legacy ETLs are discarded during upgrade. ETL pipelines also preserve encoded object paths between stages, apply configured connection limits to the active HTTP pool, and reject unsupported chunked PUT framing before transformation.

AISBatchIterDataset now raises when a batch member reports an error; cont_on_err=True logs and skips that member. The dynamic sampler reduces conversion and repeated size lookup overhead, and pyaisloader corrects PUT/MIXED accounting, dataset construction, and latency aggregation.

Commits

Miscellaneous Improvements and Fixes

  • Cold GET resets reused object metadata before issuing the backend request, so stale chunk flags or a shortened local name can no longer change that request.
  • Object placement: runtime-only HRW placement state survives metadata reloads; it is derived from object naming and is not persisted with the object.
  • Intra-cluster streams: concurrent reopens of streams to the same peer are serialized. Previously, two racing reopens could lose an update to the set of open streams.
  • Reverse proxy: gateways no longer parse the destination node URL on every reverse-proxied request; parsing happens once, when a reverse-proxy instance is created and cached.
  • Internet downloader: removing a job is serialized with concurrent removals, status, and list requests (see Internet Downloader).
  • Python AuthN client: repeated or overlapping permission flags in role creation and updates no longer grant unintended permissions. Previously, for example, requesting object GET twice produced object HEAD permission instead.
  • Python SDK: dSort requests preserve file-specification settings, empty object selections are rejected, a zero job end time is treated as unfinished, the SDK works under python -O, and SDK logging no longer alters shared loggers. See the Python changelog.
  • Production builds avoid nontrivial debug-assert work; a repository check identifies assertion arguments that would leave function calls in optimized binaries.
  • Dependencies: golang.org/x/crypto v0.57.0, gRPC v1.83.2, and OpenTelemetry v1.45.0. AuthN uses the Go standard library's PBKDF2 instead of x/crypto.

Commits

Documentation

Two technical articles were published during this cycle:

Reference documentation updated for this release:

Removed and Deprecated APIs

Removed in v5.1

  • The Enforce-IntraCluster-Access feature flag (Go: feat.EnforceIntraClusterAccess), a no-op since 5.0, is removed. Feature bit 0 is reserved starting in v5.1: a persisted configuration that still sets it loads normally and the bit is cleared.
  • The deprecated reb.Status.Targets response field is removed.
  • Kubernetes Pod-spec ETL initialization (InitSpecMsg, CLI Pod-spec parsing, Python Etl.init_spec(), and bundled Pod templates) is removed. Use runtime specifications.
  • Python's legacy object HEAD behavior, Object.head_v2(), and ObjectAttributesV2 are removed; use Object.head() and ObjectAttributes.

Existing deprecations

  • HEAD(object) v1 was deprecated in v4.2 (Jan 2026). Its removal has been deferred to give deployed clients more time to migrate. Go callers should use api.HeadObjectV2, which returns a minimal default set of properties (name and size) and any others on request. In Python, Object.head() provides the same starting with SDK 2.0.0; versions through 1.26.0 still use v1. HTTP clients should specify the props query parameter (see HTTP API or LsoMsg and HEAD(object) enum).

  • keepalivetracker.retry_factor, deprecated in 5.0, remains accepted with its existing behavior.

  • The pre-5.0 configuration names auth.enabled and auth.cluster_key.* are still translated on load and accepted by the set-config APIs, so that Kubernetes Operator-driven upgrades keep working. Support is planned for removal in a later release; use auth.client_auth_required and auth.intra_cluster.*.

Rechunk: per-job chunking overrides (deprecated; removal planned for v5.2)

Deprecated:

  • apc.RechunkMsg.ChunkSize ("chunk-size")
  • apc.RechunkMsg.ObjSizeLimit ("objsize-limit")
  • CLI: ais bucket rechunk --chunk-size and --objsize-limit

Both fields override the bucket's chunks.chunk_size and chunks.objsize_limit for a single job without recording the change anywhere. The result is a layout that the bucket configuration no longer explains, and that the next rechunk silently undoes. See Chunked Storage for the model these overrides contradict.

In v5.1 both fields are still honored. The CLI warns when either flag is given; the proxy logs a warning when a request carries values that differ from the bucket's configuration.

Removal of both fields is planned for v5.2; rechunk will then always converge to the bucket's chunks configuration. To change the layout, update the bucket first:

$ ais bucket props set ais://mybucket chunks.chunk_size=16MiB chunks.objsize_limit=50MiB
$ ais bucket rechunk ais://mybucket

--prefix remains, as a way to convert incrementally rather than to apply a different policy to part of the bucket.

Until v5.2, API callers should pass values equal to the bucket's chunks configuration - which is what the CLI does when the flags are omitted. Note that an omitted objsize-limit decodes as 0, which restores chunked objects to monolithic.

Upgrade Notes

Rebuild v1 shard indexes

v5.1 treats shard-index metadata version 1 as stale. Reads safely fall back to sequential TAR traversal, but indexed performance returns only after rebuild. After every target reaches 5.1, run the standard build without --skip-verify for indexed buckets:

$ ais bucket shard-index build ais://dataset --wait

Large datasets can be rebuilt by prefix. Use ais bucket shard-index summary to monitor valid, stale, invalid, and unindexed shards.

On-demand (re)indexing, which will rebuild v1 indexes automatically, is planned for the next release.

Re-initialize Pod-spec ETLs

Pod-spec ETL initialization is removed, and 5.1 discards persisted legacy ETLs. Initialize them again through the runtime-spec API - Etl.init() or Etl.init_class() in Python.

The Python SDK's breaking changes (Python 3.10 minimum, Object.head() returning ObjectAttributes, removed names) take effect when you upgrade to SDK 2.0.0; see Python SDK.

Rechunk overrides are deprecated

The rechunk overrides are deprecated in 5.1, with removal planned for 5.2. Move storage policy into bucket chunks properties now; see Removed and Deprecated APIs.

Experimental kTLS is opt-in

No action is required for existing HTTPS deployments. System-Reserved-KTLS is disabled by default, requires target restart when changed, and should be enabled only after confirming kernel, TLS-version, cipher, HTTP/1.1, and client reconnect compatibility as described in HTTPS.
Where NIC offload is the goal, also confirm which TLS versions the NIC driver can offload.

Acknowledgments

Several of the security fixes in this release originated in reports from outside the project. We are grateful to the people who took the time to investigate AIStore and report what they found:

  • YJK (@YJK0805) - object names were not validated everywhere AIS turns them into on-disk paths, allowing traversal through copy, transform, promote, multi-object, and archive paths (96b111f54); and an append handle issued for one object could be presented for another (ae2a2716f).
  • Ali Sherif - targets answered node-control queries sent directly to their public endpoints, bypassing the gateway authorization required on protected clusters (3e2b18579).

Reports are handled per SECURITY.md.

Assets 10
Loading