Skip to content

5.2

Latest
Latest

Choose a tag to compare

View all tags
@alex-aizman alex-aizman released this 01 Nov 19:19
· 32 commits to main since this release
v1.5.2
38dfeb3

AIStore 5.2 is a short-cycle reliability and scalability release following 5.1. About half of it is fixes (including critical) - in long object names, chunked writes and copies, erasure-coded rebalance, list-objects, and cluster startup. The other half adds capabilities: per-target placement weights, batched verification for cleanup at scale, bucket-governed chunked storage, and protection from transfers that stop making progress.

The main themes are:

  1. Placement weights (experimental). Storage nodes (aka, targets) can store unequal shares of each bucket - for instance, in proportion to their usable capacity. Placement remains the traditional HRW when weights are not set or are equal; a bucket can opt out.

  2. kTLS transmit offload is on by default for HTTPS targets on Linux, after a cycle as an opt-in preview.

  3. Chunked storage governed by bucket configuration. v5.2 implements the model announced in 5.1: changing a bucket's chunks properties starts rechunk automatically; PUT, cold GET, copy, prefetch, and rebalance store objects according to those properties; the per-job overrides are removed. Separately, fixes address failure paths that could leave partial chunked objects or lose previous content.

  4. Security. Fixes for access checks, path handling, and outbound connections - most of them reported by external researchers, who are credited in Acknowledgments.

  5. Protection from stalled GET transfers. A client that stops reading, or a cloud backend that stops sending, no longer holds object locks, open files, and goroutines indefinitely. Covered GET paths enforce a minimum transfer rate in both directions.

  6. Head-batch verification. Rebalance cleanup and space cleanup confirm misplaced copies with the owning target in batches rather than one request per object. Batching amortizes the cost of round trips between targets, especially on higher-latency networks.

  7. Python SDK 2.1.0 and aisloader. The SDK adds response streaming and size-based blob-download selection, and fixes object-file, shared-memory, and PyTorch behavior. The benchmark tool adds YAML workloads, object lists read from AIS, and results saved back to the cluster.

AIStore now builds with Go 1.27.

Upgrade: see Upgrade Notes below - in particular, kTLS on by default (with the renamed feature flag), the minimum transfer rate on GET, stricter authorization, and the constraints on placement metadata during a rolling upgrade. The notes also cover the native Go multipart API and the Python SDK's compatibility changes.

Table of Contents

  1. Fixes
  2. Placement Weights (experimental)
  3. Bucket Summary
  4. Rebalance and Space Cleanup: Head-Batch Verification
  5. Chunked Storage
  6. Blob Downloader and Prefetch
  7. Protection from Stalled GET Transfers
  8. kTLS Transmit Offload
  9. Object Metadata
  10. Erasure Coding and Global Rebalance
  11. Cluster Membership and Startup
  12. Networking and TLS
  13. List Objects
  14. Space, LRU, and Capacity
  15. Cloud Backends and S3 Compatibility
  16. Shard Index
  17. Jobs
  18. Security
  19. CLI
  20. Native Go API
  21. Python SDK
  22. aisloader
  23. Build, Tools, and Deployment
  24. Miscellaneous Improvements and Fixes
  25. Documentation
  26. Removed and Deprecated APIs
  27. Upgrade Notes
  28. Acknowledgments

Fixes

v5.2 includes important reliability and security fixes. We describe the affected behavior here for transparency and to help users plan their upgrades; the linked sections provide details and commits.

We recommend upgrading to v5.2 to benefit from these corrections. Review the Upgrade Notes before starting; for Kubernetes deployments, follow the AIS version update guidance in ais-k8s.

Data integrity

  • Object metadata decoding. Fixed a rare decoding error that could report intact objects as corrupted. Affected objects load correctly after the upgrade, without an on-disk format change. See Object Metadata.
  • Long object names. Fixed placement and cleanup handling for names that exceed the local filesystem limit. Rebalance could select the wrong target, and space cleanup could remove chunks and manifests. Cleanup now retains these objects pending full reclamation support; the upgrade does not restore previously deleted content. See Object Metadata.
  • Rename and promote. AIS now retains the source when a transfer to another target fails, including promote --delete-src. Previously, AIS could remove the source even after the destination rejected the write. See Object Metadata.
  • Chunked writes and copies. Fixed manifest-completion failures that could leave a manifest inconsistent with its chunks or lose the previous object, along with duplicate first-chunk writes and shared buffers during concurrent copies. See Chunked Storage.
  • Erasure-coded rebalance. Rebalance now records an EC metadata broadcast failure and continues with other objects. Slice relocation preserves main-replica ownership and sends the slice's own checksum. See Erasure Coding and Global Rebalance.

Listings and copies

  • Remote bucket copies. Empty remote listing pages with continuation tokens no longer end a copy prematurely or prevent ais cp --all from starting. If a previous remote-bucket copy encountered such pages, verify it against the source and run the copy again where needed. See List Objects.

Cluster availability

  • Primary recovery. A former primary restarting after an election confirms the current primary and continues as a secondary, rather than taking over the running cluster when some nodes contact it. See Cluster Membership and Startup.
  • Remote listing during a membership change. A remote list-objects request that aborted mid-page because cluster membership changed could wait until client.list_timeout; it now returns the abort error. This is an interim fix for one abort scenario. See List Objects.
  • Target stability. Fixed panics during EC rebalance after a failed bucket initialization, and during automatic chunked upload to GCP. See Erasure Coding and Global Rebalance and Cloud Backends and S3 Compatibility.

Security

Placement Weights (experimental)

See Placement weights

Until v5.2, AIStore always distributed objects uniformly across targets, regardless of their usable storage capacity. That works well for homogeneous clusters, but can leave capacity unused when targets have different amounts of usable storage. v5.2 adds relative target weights: with weights 1, 3, 6, three targets are expected to own 10%, 30%, and 60% of a bucket's objects. Weights are positive 64-bit integers and have no prescribed unit; usable capacity in bytes is one practical choice.

The shares describe expected object counts; byte totals depend on object sizes, and small buckets can deviate visibly from the configured shares. AIS does not measure capacity and adjust weights automatically; the operator assigns them. See Bucket Summary to compare configured shares with the content actually stored on each target.

Placement and redistribution

With no weights, or with equal weights, placement remains the traditional HRW. Unequal weights use weighted HRW; changing one target's weight moves objects to or from that target rather than reshuffling the other targets' ownership. Changing relative shares starts global rebalance; setting the same weights is a no-op, and multiplying every weight by the same factor changes metadata without moving objects.

A bucket can retain uniform placement with placement.weighting=uniform. Changing its effective placement starts a bucket-scoped rebalance and returns that job's ID, including when resetting the override or resetting bucket properties. A change that leaves effective placement unchanged does not redistribute data. As with other layout changes, a single bucket-properties update cannot combine placement redistribution with re-mirroring, re-encoding, or rechunking.

Weights apply wherever AIS determines an object's owner: request routing, rebalance, cleanup, and jobs that locate or write objects. Selecting a designated target for a job remains uniform (for example, the target that lists a remote bucket for list-objects or bucket summary). Weights also apply to EC: the full replica and slices follow the weighted order, with at most one part per target. Moving the owner of an erasure-coded object requires a spare active target beyond the full replica and slices. Updates that need that migration are rejected before changing metadata if the spare is unavailable.

$ ais cluster placement set-weights 't[KxHgNnqB]=1' 't[mRqzPWbe]=3' 't[TdVnLcsJ]=6'
$ ais storage summary ais://abc --per-target
$ ais bucket props set ais://abc placement.weighting=uniform

Membership and limits

An existing target keeps its assigned weight across restart and re-registration. A new target can request a weight through its local placement.weight or an administrative join's --weight; otherwise, it receives the mean of active targets' weights. An existing cluster-map assignment takes precedence over a changed local preference. During a weighted cluster restart, readiness waits for the primary to reconcile weights and distribute the resulting cluster map.

Weights require at least two targets. AIS warns at a max/min ratio of 10 or greater and rejects 100 or greater. It also rejects clearing target weights while any bucket retains a placement override. Complete the upgrade before enabling either target weights or bucket overrides; see Upgrade Notes.

Placement weights are experimental in 5.2. Python SDK direct-to-target access (get_target_for_object) still computes uniform placement; use proxy-mediated requests for weighted buckets.

Commits

Bucket Summary

In AIStore, a summary is a set of total statistics collected across the cluster. A bucket summary reports, for each bucket, the number of objects, their sizes (min, average, max), total size, and the percentage of total disk capacity occupied by the bucket - for remote buckets, separately for content stored in the cluster and in the backend. It runs as an asynchronous job on all targets.

v5.2 adds a per-target breakdown: ais storage summary --per-target (also ais bucket summary and ais ls --summary) shows how a bucket's in-cluster content is distributed across targets - object counts, object bytes, and their shares of the bucket totals. --per-target implies --cached. With --no-headers, the command prints per-target rows only; with --json, each bucket's result carries a per_target map keyed by target ID. Per-target numbers are observed storage statistics, including misplaced objects awaiting cleanup after rebalance; compare them with the configured shares shown by ais cluster placement. See Per-target content distribution.

In the API, set apc.BsummCtrlMsg.PerTarget (per_target) to receive per-target results in cmn.BsummResult.Targets. A pre-5.2 proxy rejects the field.

Also fixed:

  • bucket-summary results (api.GetBucketSummary) now report is_present for buckets present in the cluster; previously, only bucket info (api.GetBucketInfo) set it
  • a bucket query (e.g., ais:) that matches a single bucket now reports that bucket's name and provider
  • bucket-summary polling validates the job ID and requires --dont-wait

Commits

Rebalance and Space Cleanup: Head-Batch Verification

Rebalance cleanup (ais start rebalance --cleanup) and space cleanup (ais storage cleanup) remove a misplaced copy only after its owner confirms that it holds the same object. In v5.1, each confirmation was a separate HEAD request, so cleanup time grew with the number of copies and with the latency between targets. v5.2 verifies copies in batches (64 by default) with a single request to the owner.

  • The verification policy is unchanged: missing, busy, divergent, and unanswered copies stay on disk; --force retains its existing meaning. Space cleanup now also honors --keep-misplaced for confirmed copies.
  • Cleanup keeps a copy that changed while its owner was being queried.
  • An owner running v5.1 or earlier rejects the batch, and the sender falls back to one HEAD request per object.
  • An owner at critical memory or CPU load answers HTTP 429; the unverified copies stay for a later run. A sender under critical load throttles or defers batches.
  • ais show job rebalance --all and ais show job cleanup-store --all report the new hdb-* counters.

Limitation: head-batch verification runs only in a separate pass: rebalance cleanup or space cleanup, after rebalance has finished migrating objects. A regular, data-moving rebalance still does not remove source copies as it goes, so receiving targets need free space for everything that moves. A cluster that is already highly utilized (for example, above 80% capacity) may be unable to rebalance. A future release will verify and remove source copies on the fly, while rebalance runs.

Commits

  • 921c0c5f4: the intra-cluster contract for batch HEAD requests
  • 23bf88e2b: the receiver and the sender
  • 10513da1b: requests for many buckets, routed by path
  • 020f1742c: the batcher that groups objects by owner
  • a68eae9a8: the receiver answers HTTP 429 at critical load
  • 924322570: one batcher for each cleanup run, and its use in rebalance cleanup
  • c57f03c0b: space cleanup verifies misplaced copies with head-batch
  • c35fb8a93, 042bf2bca: release the object lock during peer checks and revalidate before removal

Chunked Storage

In v5.1, we set the objective: chunked objects must become a true third storage layout, alongside n-way mirroring and erasure coding - and, like the other two, follow bucket configuration. v5.2 implements it, as announced. A bucket's chunks properties are now the single statement of how its objects are stored; jobs converge the data to that statement instead of overriding it.

Bucket configuration is authoritative

  • Changing chunks properties starts rechunk, as changing mirror or ec already starts re-mirroring or re-encoding. The bucket update returns the job ID. AIS rejects a single properties update that would start more than one such job - for instance, changing chunks and ec together; make the changes one at a time.
  • Rechunk converges only. Rechunk skips objects that already match the bucket's configured layout rather than rewriting them, and now converts existing monolithic objects above chunks.max_monolithic_size. Rechunk started by a properties change continues past per-object failures; an explicit ais bucket rechunk stops at the first failure unless run with --cont-on-err.
  • PUT, cold GET, and copy store an object as chunks when it exceeds chunks.max_monolithic_size or, with auto-chunking enabled, reaches chunks.objsize_limit. Previously, only rechunk applied the latter.
  • Prefetch follows the blob-downloader rules announced in 5.1: with auto-chunking enabled, the bucket's chunk size wins. See Prefetch: blob-downloader chunking.
  • Rebalance stores each received object according to the chunking policy of its bucket.
  • The deprecated per-job overrides apc.RechunkMsg.ChunkSize and ObjSizeLimit, and the matching CLI flags, are removed.

Single-object operations - client-side multipart PUT, S3 multipart upload, and single-object blob download - still choose their own chunk size, as described in 5.1.

A failed chunked PUT (including multipart upload) or copy no longer leaves a partial object behind or loses the previous one; space cleanup (ais storage cleanup) reclaims chunks that remain orphaned. See Chunked writes and cleanup on failure.

Commits

Blob Downloader and Prefetch

The blob downloader is the only AIStore job that operates on a single object: it fetches one large remote object using concurrent backend range reads, either as a standalone job or while serving a GET. Prefetch is the multi-object job that applies it en masse, using the blob downloader for objects at or above its --blob-threshold. v5.2 aligns prefetch with bucket storage policy and adds size-based selection in Python.

Fetch method and storage layout

Prefetch's --blob-threshold selects how an object is fetched; the bucket's chunks properties determine how it is stored. With auto-chunking enabled, prefetch uses the bucket's chunk size; with it disabled, prefetched blob downloads remain chunked, and the job warns and suggests rechunking. A single-object blob-download job or GET keeps its own tunables and does not change the bucket's storage policy.

Selecting blob download on GET

Python SDK 2.1 adds BlobDownloadConfig.threshold: an omitted threshold preserves explicit blob download, a positive value selects it by remote object size, and zero uses regular GET. read_timeout sets the timeout for each backend range-read attempt. Go callers can use the existing api.GetArgs.BlobThreshold; HTTP clients can send Ais-Blob-Threshold.

Range reads and archive-member extraction use regular GET when only a threshold selects blob download. Explicitly forcing blob download for either operation returns HTTP 400. Downloading the entire archive object remains supported.

The new Reading large objects from remote buckets guide compares the available approaches.

Commits

  • 96c2e6efc: prefetch follows bucket chunking policy
  • 8efd74b94: Python blob threshold and per-attempt read timeout
  • 9e35ef9fb: large-object guide; preserve range and archive GET semantics
  • d8f06fed8: cancel and clean up OCI multipart downloads

Protection from Stalled GET Transfers

A client that stops reading, or a cloud backend that stops sending, could previously hold a GET open indefinitely, together with the object's lock, an open file, and a goroutine.

v5.2 aborts only such stalled transfers: the deadline is generous - minutes, not seconds - and a transfer that keeps making any reasonable progress is never affected. Specifically, covered GET paths enforce a minimum transfer rate in both directions: by default, 18.75 MiB per 5-minute window (timeout.send_file_time), or 64 KiB/s on average.

Direction Covered Not covered
AIS => client whole-object GET, through sendfile or buffered transmit range reads, archive-member reads, streaming cold GET
Cloud backend => AIS AWS, GCP, Azure, and OCI object readers - cold GET, prefetch, blob download, and copying or transforming remote buckets remote AIS, PUT, listing, and standalone HEAD requests

When the deadline fires, the client sees a truncated response; a stalled backend read fails with ErrRemoteGetTimeout, and a cold GET that has not yet responded returns 504 Gateway Timeout. AIS counts both. To tolerate longer pauses, raise timeout.send_file_time; to disable both deadlines, set the Disable-GET-Deadline feature flag. The same commits close a GCP reader that was left open after a range-validation error.

See Minimum transfer rate.

Commits

kTLS Transmit Offload

With regular, user-space TLS, an HTTPS target reads object data from a local file into a user-space buffer, encrypts it with Go's crypto/tls, and writes the encrypted data back to the kernel for transmission. Both the copying and the encryption cost CPU time.

With kernel TLS (kTLS), AIStore still establishes the TLS session in user space, then passes the transmit keys to the kernel. The kernel encrypts outgoing data on the host CPU (software kTLS) or hands encryption to a NIC with TLS offload support (hardware kTLS). Either way, object GETs can use sendfile(2) and skip the copy through user space.

v5.1 introduced kTLS as an opt-in preview. In v5.2 it is the default for HTTPS targets on Linux. The Upgrade Notes describe what that means for clients, the renamed feature flag, and socket buffers.

Changes since 5.1:

  • Lazy arming. AIS arms kernel TLS when a connection's first response is a GET of at least 1 MiB served from a monolithic object; otherwise, it stays in user-space TLS.
  • Key-usage limit. A response of unknown size, such as streaming cold GET, retires the armed connection after the response.
  • Observability. Per-target counters for offload attempts, outcomes, and retirements.
  • Go 1.27. The Go 1.26 handshake workaround is removed.
  • Host check. scripts/check-ktls.sh verifies that a Linux host or container can use kTLS and, with --armed, that connections actually did.

kTLS is best effort: unsupported kernels, ciphers, and connection states retain user-space TLS. See kTLS transmit offload for prerequisites and diagnostics.

Hardware kTLS depends on the TLS version. Mainline Linux can offload TLS 1.2 to capable NICs (ethtool -K <dev> tls-hw-tx-offload on). TLS 1.3, which most clients negotiate, requires TLS 1.3 device offload support - in development upstream, and available today only with a patched kernel. TLS 1.3 still benefits from kernel software encryption and sendfile on standard kernels. If you run AIStore on such hardware, we would like to hear about your results.

For initial measurements - user-space TLS vs software and hardware kTLS - see Hardware-Accelerated HTTPS: Doubling GET Throughput in AIStore.

Commits

Object Metadata

AIS stores each object's metadata - size, checksum, version, and custom key-value pairs - alongside the object on its target. v5.2 fixes several cases where this metadata could be misread, misapplied, or exceed its limits:

  • Decoding. Fixed a rare metadata decoding error that could report an intact object as corrupted. The on-disk format does not change; affected objects load correctly after the upgrade.
  • Long object names. When an object name is too long for the local filesystem, AIS stores the object under a shortened name and keeps the original name in the object's metadata. Rebalance, cleanup, and listing used the shortened name for placement, so rebalance could send such objects to the wrong target and space cleanup could remove their chunks and manifests. These jobs now use the original name; if it cannot be resolved, rebalance skips the object, cleanup keeps it, and listing skips it with a warning. Until reclamation of long-name objects is fully supported, space cleanup keeps them, which can retain reclaimable data.
  • Custom metadata. The 2 KiB limit now applies to the merged result of existing and incoming keys. Previously, a PATCH or PUT validated only the incoming keys and could exceed the limit. A rejected merge leaves the existing metadata unchanged; internal keys do not consume the user metadata budget. Clients can no longer set keys reserved for internal use, and AIS does not return them in responses or listings.
  • Copy during concurrent eviction. Fixed a race between object copy and concurrent eviction that could cache stale object metadata.
  • Rename and promote. When rename or promote --delete-src moved an object to another target and that target rejected the write, the source could be deleted anyway. The source now stays.

Commits

Erasure Coding and Global Rebalance

EC relocation and failures

  • A failure to broadcast EC metadata for one object no longer aborts global rebalance; rebalance records the error (err-ecmd in the job status) and continues with other objects.
  • Relocating a misplaced slice no longer changes main-replica ownership, and the slice carries its own checksum instead of the full object's.
  • Bucket-scoped EC rebalance, including redistribution after a bucket placement change, traverses only the selected bucket.
  • Fixed a panic after a failed bucket initialization; EC discards late messages for a deleted bucket.

Maintenance completion

If a target's maintenance or shutdown rebalance was skipped or interrupted, the target could remain in an unfinished transition. The next successful full global rebalance now completes pending transitions. Rebalance also no longer waits for targets already in maintenance-post-rebalance, which may be shutting down. See Incomplete Transitions.

For verification and safe removal of misplaced copies, see Head-Batch Verification.

Commits

Cluster Membership and Startup

This release fixes startup and membership paths in which a recovered view of the cluster could override its current state:

  • A former primary no longer hijacks a running cluster. When some of the starting nodes contacted the former primary while the rest of the cluster had already elected a new one, the former primary could take over. Unless the primary is selected explicitly, a node that recovers leadership at startup now confirms with the current primary and, if confirmed, continues as a secondary.

  • False duplicate-instance detection. Behind a Kubernetes load balancer, a restarting target could conclude that another instance of itself was already running. The check now binds the local listener endpoints instead of dialing them.

  • Duplicate endpoints among nodes joining or restarting produce warnings in the log; admission still proceeds. See Duplicate Runs and Duplicate Addresses.

  • Changed network addresses. A slow or unreachable node reporting new addresses no longer holds up other membership changes. Failed probes and duplicate node IDs are rejected explicitly.

  • DNS. A target with a public hostname that does not resolve yet keeps retrying the join for up to timeout.join_startup_time, instead of failing on the first rejection.

  • Fixed an unlikely deadlock in metadata synchronization when the primary steps down, and a race at cluster startup in which a node join could be lost.

  • Information Center (IC). The group of proxies that tracks asynchronous jobs:

    • excludes proxies in maintenance or being decommissioned, and uses non-electable proxies only when needed to fill the group
    • forwards job-status queries that reach a former member to an active one, avoiding a request-handler panic
    • receives job ownership tables after maintenance, decommission, and keepalive-driven membership changes
    • is reconciled by the primary at startup

    See IC.

Commits

  • a3769a15d: do not hijack a running cluster at former-primary startup
  • bfad0497b, 8f5c6b15f: duplicate-instance detection; duplicate endpoints
  • fa0881397: network-address confirmation outside the lock
  • 5ebcbd215: retry joins on unresolved hostnames
  • 470b168af, c51aa8e71: metasync deadlock; lost node joins at startup
  • 508cdd6ef: IC staffing, routing, and ownership synchronization

Networking and TLS

Node-to-node TLS with a public certificate

When intra-cluster control traffic runs over the public network with certificates from a public issuer, the name a node dials - often an internal address - does not match the name in the certificate. The new net.http.alt_tls_dns_name gives nodes the DNS name to verify instead. It applies to control requests and to relays within the cluster; data transfers between targets and requests to external services are not affected. See Node-to-Node TLS.

Clients behind a load balancer

The CLI keeps using its configured endpoint after it fetches the cluster map, instead of switching to the primary's URL for shutdown and decommission; gateways forward requests that require the primary. Network errors keep their original cause: the CLI distinguishes connection failures from timeouts, and names the unreachable endpoint when it differs from the configured one. Certificate verification errors include hostname and CA guidance.

Keeping the gateway endpoint does not change data redirects: clients still need to reach the target addresses advertised by the cluster, with certificates valid for those addresses. The alternative TLS DNS name above controls node-to-node verification; it does not change how external clients verify targets.

Commits

  • 551071d1c, 9df8e6a0a: alternative TLS DNS name for intra-cluster requests
  • 3bf5df9a5: preserve the configured endpoint behind load balancers
  • bab474d19: keep original network errors; report unreachable endpoints

List Objects

  • Membership change during a remote listing. When cluster membership changed while a remote listing was serving a page, the job aborted but the request did not return until client.list_timeout, and each aborted job left a housekeeping callback behind. The caller now receives the abort error. Propagating the abort to the other targets is an interim solution; a dedicated control path for aborting jobs across targets is planned.
  • Empty pages are not the end. An empty remote page with a continuation token (for example, a page of filtered-out S3 directory markers) ended remote-bucket copies, CLI listings, and the Python ObjectIterator; ais cp --all saw an empty bucket and did not start. All three now continue while the token advances.
  • S3 ListMultipartUploads now paginates across targets, applying prefix and delimiter filters, key and upload-ID markers, and the requested page limit.
  • Bucket inventory (NBI): fixed decoding when inventory chunks differ in size. AIS rejects creating an inventory for an ais:// bucket.

Commits

Space, LRU, and Capacity

Cleanup watermarks and PUT admission

Previously, every PUT(object) above space.cleanupwm invoked the out-of-space handler - a capacity refresh and, possibly, full storage cleanup. Now targets above space.cleanupwm remove deleted buckets and objects in the background, one pass at a time, skipping disks under critical load. Full storage cleanup, followed by LRU if pressure persists, is reserved for usage above space.highwm.

PUT invokes the out-of-space handler only above space.highwm, and still fails with HTTP 507 only when the target is out of space. The target now determines the error after the capacity refresh; previously, a PUT could report the pre-refresh error. LRU and Space describes the roles of the cleanup, high, low, and out-of-space watermarks.

LRU policy

LRU (Least Recently Used) eviction is one of AIStore's 35 asynchronous job kinds, or xactions. It is disabled by default.

LRU now honors each bucket's lru.dont_evict_time; previously, it always used the cluster-wide value. A forced LRU run keeps its force policy when periodically refreshing eligibility, and enabling LRU on a bucket while above the high watermark no longer fails. These fixes do not enable LRU on buckets that have it disabled.

Capacity checks

Capacity checks for PUT requests now reuse cached filesystem statistics, reducing system calls.

Mountpaths

Mountpath directories cannot contain one another. AIS now enforces this for both enabled and disabled mountpaths.

Commits

Cloud Backends and S3 Compatibility

AIStore can operate as standalone storage or as a fast tier in front of remote object stores, called backends. These include Amazon S3, Google Cloud Storage (GCP), Azure Blob Storage, and Oracle Cloud Infrastructure (OCI) Object Storage. Applications access local and remote data through the same bucket abstraction. See At a glance.

Metadata round trip

  • GCP preserves Content-Type and S3 user metadata across PUT, HEAD(object), and cold GET.
  • OCI preserves Content-Type and user metadata across regular and multipart PUT, HEAD(object), cold GET, and copy.

Multipart

  • GCP implements multipart upload abort.
  • OCI multipart completion keeps the provider's part ETags (previously, local MD5s replaced them and completion failed).
  • Fixed a panic in automatic chunked upload to GCP.
  • Multipart start, completion, and abort honor the caller's cancellation and deadlines for all cloud backends and remote AIS clusters; once the backend commits, AIS finishes local completion even if the client disconnects.

Other fixes

  • HEAD(bucket) on an s3:// bucket succeeds when the provider does not allow querying bucket versioning; AIS then omits the versioning header.
  • Fixed alias substitution for remote AIS clusters, which could break GET and prefetch through the alias after a primary restart.
  • AIS rejects S3 requests with consecutive slashes in the path (InvalidArgument). Previously, it silently dropped empty path components, so that a//b and a/b named the same object.

Commits

Shard Index

Shard index lets AIS locate and read archived files in TAR objects by offset, avoiding a sequential archive scan.

Indexes currently require an explicit build. Automatic index creation and rebuilding when a GET requests an archived file remain in development.

Warm the in-memory cache

v5.1 introduced the memory-aware cache for packed TAR indexes. v5.2 adds an explicit way to warm it before a workload starts:

$ ais bucket shard-index build ais://dataset --prefix shards/ --cache --wait

The build reuses cached indexes or loads valid persisted indexes into memory. Missing, stale, or corrupt indexes follow the normal build path. --cache and --skip-verify are now independent; previously, --cache silently overrode --skip-verify. Caching shard indexes in memory remains best effort under memory pressure.

ais start shard-index and ais job start shard-index start the same build.

For v1 indexes, omit --skip-verify, as described in Shard Index: Upgrading.

See Warm the in-memory cache.

Commits

Jobs

AIStore jobs, or extended actions (xactions), are asynchronous operations such as rebalance, prefetch, cleanup, and dataset copy. They report progress and statistics and can be monitored, waited for, or aborted through the CLI and APIs. See Jobs and extended actions.

  • Job history. Running jobs always appear in the full job list. List-objects jobs, which are numerous and short, age out after a minute (ten minutes at verbosity 4); other finished jobs after an hour, keeping at least the most recent 1,024. Get-Batch jobs follow the ordinary retention.
  • Job abort now reaches every running job, including ones the history limit kept out of the listing.
  • Job IDs. The previous generator could produce colliding IDs; generated IDs are now random alphanumeric strings that cannot be mistaken for numeric rebalance IDs.
  • Fixed a potential deadlock in multi-object (list, range, and prefix) jobs when a worker aborts.
  • Fixed prefix filtering in jobs that process multiple buckets.
  • Querying a job on a given target goes to that target only.

Commits

Security

AIStore's two security perimeters separate user authority from cluster-member authority. The outer perimeter authenticates users and checks cluster and bucket permissions; the inner perimeter authenticates current cluster members and their signed requests, including gateway-signed handoffs carried by redirected clients.

Security fixes in this release address authorization, path handling, multipart upload ownership, and outbound connections. Acknowledgments lists the externally reported issues, reporter credits, and fixing commits. See Upgrade Notes for changes to access requirements, signed copy compatibility, and endpoint restrictions.

Additional authorization fixes:

  • S3 GetBucketVersioning checks access to the bucket.
  • AIS authorizes the first access to a remote bucket that is not yet in the cluster's metadata before adding the bucket - for object operations and for bucket jobs such as prefetch, copy, transform, rechunk, and inventory.
  • Showing inventories across multiple buckets requires permission to list buckets.

Commits

CLI

The AIS CLI manages and monitors AIStore clusters, with commands for initial setup, daily operations, object and dataset management, and distributed ML workflows.

Name/value arguments

All commands that take name/value pairs - cluster, node, and CLI configuration, bucket properties, object custom metadata, mountpaths, remote cluster aliases, and placement weights - now share one parser with one set of rules:

  • name=value and name value are both accepted, with or without spaces around =.
  • A pair splits at the first unescaped =; \=, \ (escaped space), and \\ are escapes. JSON values pass through as is.
  • List values - such as features, log.modules, and access - accept commas or spaces, and end at the next property name.
  • show and set are optional in bucket-properties commands. The CLI applies JSON configuration sections at their paths in one request, rejects unknown fields, and warns when a change requires a restart.

Object custom metadata now accepts = in values, and the CLI decodes backslash escapes. See Name/value arguments.

New and changed commands

  • ais cluster placement (show, set-weights, clear-weights), a WEIGHT column in ais show cluster, and --weight when adding a target. See Placement Weights.
  • --per-target in ais storage summary, ais bucket summary, and ais ls --summary. See Bucket Summary.
  • ais bucket shard-index build --cache and ais start shard-index.

Fixes

  • ais cp --all progress totals were wrong for partially cached remote buckets.
  • ais bucket rm checks emptiness only when confirmation is needed and, when asked to destroy a cloud bucket, suggests evicting it instead.
  • The --regex column filter is now case-insensitive.

Commits

Native Go API

The native Go client provides access to AIS over HTTP(S). Existing v5.1 clients remain compatible with v5.2, subject to the removals and behavior changes in Removed and Deprecated APIs and Upgrade Notes. New APIs and request options are opt-in; unset options are omitted from requests.

The multipart function signatures below change Go source compatibility, not the HTTP request format. Existing binaries can still perform multipart uploads; applications upgrading the Go client package must adapt those calls to the new argument structures.

Multipart contexts (API change)

api.CreateMultipartUpload, api.CompleteMultipartUpload, and api.AbortMultipartUpload now take argument structures - MptArgs, CompleteMptArgs, and AbortMptArgs - with an optional Context. An omitted context defaults to context.Background(); callers that need cancellation or a deadline can carry it through multipart start, completion, and abort, including requests to remote AIS clusters. Once the backend completes, AIS finishes local finalization even if the client disconnects.

Placement, summaries, and errors

api.SetPlacementWeights accepts an edited cluster map and returns a rebalance ID when changed relative shares require redistribution. apc.BsummCtrlMsg.PerTarget requests per-target summary results in cmn.BsummResult.Targets; aggregate totals remain available.

api.QueryXactionSnaps now honors the selected target when querying job statistics. Client-side cmn.ErrHTTP retains its original network cause for errors.Is and errors.As.

Commits

Python SDK

Python SDK 2.1.0 was released on October 28, 2026. It adds response streaming and blob-download controls and fixes object-file handling, shared-memory cleanup, listing, ETL URLs, and PyTorch integration. We recommend upgrading to benefit from these fixes; review the following compatibility changes first.

DynamicBatchSampler.__len__() is removed because PyTorch interprets the result as a number of batches, while the sampler returned samples. Use num_samples() for the sample count.

ObjectReader.raw() is deprecated, with removal planned for SDK 3.0. Use ObjectReader.stream() and the response's raw field; close the response when finished.

ObjectFileWriter now implements io.BufferedIOBase. Code that explicitly checks for BufferedWriter should use BufferedIOBase.

The Python multipart upload API is unchanged; the Go function-signature changes described above do not apply to Python callers.

Fixes

  • Object files: entering an open writer's context no longer truncates data twice, .closed reports the actual state, and read resumption counts only interruptions without progress.
  • ParallelBuffer.close() unlinks shared memory and closes its file descriptor even while zero-copy views are alive. The mapping is released after the last view is gone.
  • Listing: empty remote pages no longer stop ObjectIterator; exist_ok and missing_ok work outside the global namespace; retries again handle ErrGETConflict and ErrObjNotFound.
  • Get-Batch: multipart responses are no longer split incorrectly when the body contains a header terminator.
  • Object URLs preserve ETL names, pipeline stages, and transform arguments.
  • PyTorch: AISShardReader streams shards when counting samples, and members such as sample.left.jpg and sample.right.jpg no longer overwrite one another.

SDK 2.1.0 does not yet provide dedicated helpers for assigning target placement weights or requesting per-target bucket summaries; use the CLI or Go API for these operations. For weighted buckets, keep direct=False (the default) when reading objects: the SDK's direct-to-target routing still computes uniform placement. See Python SDK direct reads.

The complete API-level changes are in the Python changelog.

Commits

aisloader

The AIS load generator adds reproducible workload configuration and the ability to keep both inputs and benchmark results in the cluster.

Workloads and object lists

-config reads a YAML workload file using the existing flag parsers and validation. Explicit command-line flags override the file, so you can reuse the same workload while changing one parameter. See the sample configuration.

For permutation-based reads over a fixed object list, -seed controls the initial read order; it does not guarantee reproducible ordering across every epoch.

-filelist-obj reads a newline-separated object-name list from an AIS-accessible object. The list can live in a different bucket or provider from the workload data.

$ aisloader -config=workload.yaml -numworkers=64
$ aisloader -bucket=ais://images -filelist-obj=ais://lists/names.txt -pctput=0 -duration=10m -cleanup=false

Results and connection reuse

-stats-output-obj uploads the final statistics to an object in a local AIS bucket. Combine it with -stats-output to also save the results locally. JSON output also reports multipart PUT statistics separately from regular PUTs.

$ ais bucket create ais://results
$ aisloader -bucket=ais://images -pctput=0 -duration=10m -cleanup=false -json -stats-output-obj=ais://results/run-1.json

The idle HTTP connection pool now follows the concurrent workload, including multipart part uploads, with a minimum of 32 connections per host; -idle-conns-per-host overrides it. On exit, aisloader deletes the ETLs it created for the run, using their actual names.

Helm/aisloader integration and coordinated multi-instance runs (single shared config, centralized start/stop, combined total statistics) are planned.

Commits

Build, Tools, and Deployment

  • Go 1.27 for all modules, CI, and container images; golangci-lint 2.14 with the modernize linter. The node's version log line includes the Go version used to build it.
  • OSS dependencies upgraded; scripts/upgrade-oss.sh automates the procedure.
  • Deployment: scripts set kernel auto-tuned socket buffers for new clusters (see Upgrade Notes); aisinit adds AIS_PUBLIC_HOSTNAME_OVERRIDE; a Kubernetes sample deploys the ETL-enabled image (make etl).
  • aisloader testing: a Go end-to-end suite replaces the shell CI test, covering PUT, multipart, mixed workloads, Get-Batch, cleanup, and optional ETL. See aisloader for the user-facing additions.
  • CI: HTTPS test stages run with kTLS and fail if no connection used it.
  • Python formatting: Ruff replaces Black; make fmt-check installs and runs it.

Commits

Miscellaneous Improvements and Fixes

  • Metadata updates: bucket-metadata synchronization uses direct lookups instead of repeated full scans, and EC processes bucket-property transitions only when relevant settings change. This reduces control-plane work as the bucket count grows.
  • Request paths: fewer allocations in request and metadata handling.
  • Monitoring: certificate-invalid and election-in-progress alerts use the intended red and yellow colors. Documentation now uses the exported Prometheus names and distinguishes backend reads from user cold-GET counts.

Commits

Documentation

We published a technical article during this cycle:

New:

Revised:

  • Configuration: initial configuration vs runtime updates vs environment; restart requirements; minimum transfer rate
  • Node lifecycle: skipped and incomplete rebalance transitions
  • Networking: node-to-node TLS; duplicate runs and addresses
  • Storage services: LRU, space watermarks, and chunked writes on failure
  • Monitoring: exported Prometheus names, alert flags, and PromQL examples
  • Authentication, HTTPS, HTTP API, and distributed tracing
  • CONTRIBUTING.md: coding style and an AI-assisted contribution policy

Removed and Deprecated APIs

Removed in v5.2

  • Rechunk per-job overrides, deprecated in 5.1: apc.RechunkMsg.ChunkSize ("chunk-size"), apc.RechunkMsg.ObjSizeLimit ("objsize-limit"), and the CLI flags ais bucket rechunk --chunk-size and --objsize-limit. Rechunk always converges to the bucket's chunks configuration; to change the layout, update the bucket properties, which now starts rechunk automatically.
  • Python: DynamicBatchSampler.__len__() (SDK 2.1.0).

Deprecated in v5.2

  • Python: ObjectReader.raw(), planned for removal in SDK 3.0. Use ObjectReader.stream().

Existing deprecations

Unchanged from 5.1: HEAD(object) v1, keepalivetracker.retry_factor, and the pre-5.0 configuration names auth.enabled and auth.cluster_key.*. See Existing deprecations.

Upgrade Notes

kTLS transmit offload is on by default

On Linux, HTTPS targets now offload TLS transmit to the kernel (kTLS) on their public listeners and serve eligible object GETs with sendfile(2). In v5.1, this was an opt-in preview. For HTTPS clusters, this means:

  • Public target connections use HTTP/1.1 and have TLS session tickets disabled.
  • Targets retire long-lived connections near the per-key transmit limit; clients must tolerate connection closure and reconnection.
  • Fixed-size socket buffers limit transmit throughput; consider net.l4.sndrcv_buf_size=-1 (see Socket buffers below).

The v5.1 System-Reserved-KTLS feature flag is renamed Disable-KTLS and now disables kTLS - same bit, reversed polarity. A cluster that set the flag in v5.1 to enable kTLS runs with kTLS disabled after the upgrade; to re-enable, remove Disable-KTLS from the feature list. Scripts that use the old name fail with "unknown feature flag". Changing the bit requires restarting the targets. Disable-KTLS is reserved for internal use and may be redefined or removed in a later release. Plain-HTTP clusters are not affected. See kTLS transmit offload.

Socket buffers: kernel auto-tuning for new clusters

The deployment scripts now set net.l4.sndrcv_buf_size to -1 (kernel auto-tuning) for new clusters. The AIS default (0) still means a fixed 128 KiB buffer size. This applies to Kubernetes deployments (aisinit) and to deployments that use aisnode_config.sh, such as the local playground and the Docker images. Existing clusters keep their current value. For the supported values, see Socket Buffers in the 5.1 release notes.

With kTLS, a fixed buffer size (0 or any positive sndrcv_buf_size value) makes transmit less efficient. On an existing HTTPS cluster, set -1 and restart the targets:

$ ais config cluster net.l4.sndrcv_buf_size=-1

On Kubernetes, set spec.configToUpdate.net.l4.sndrcv_buf_size in the AIStore resource instead, so that new and replacement target pods use the same value.

Placement metadata requires completing the upgrade

Complete the upgrade on every proxy and target before assigning target placement weights or setting bucket placement overrides. Older nodes cannot decode placement metadata in cluster maps (Smap), bucket metadata (BMD), or the rebalance metadata used for bucket redistribution.

During a rolling upgrade, leave target weights unset and bucket placement overrides omitted. Unweighted Smaps retain the pre-5.2 wire format and uniform HRW placement; default bucket properties also omit the new placement section.

Assigning target weights requires all nodes to run the same AIS software version as the primary. The weight-update API returns HTTP 409 while the primary has recorded a software-version mismatch. Clearing all target weights remains allowed unless a bucket has a placement override. A pre-5.2 node cannot participate in a cluster if the cluster's Smap carries weights (including equal weights that produce uniform placement) or its BMD carries any bucket placement override.

The primary learns node versions on join and from keepalives. Right after a primary change, the weight-update API may return ErrBusy (HTTP 409) until every active member has reported its version; retry. See Placement weights for compatibility constraints.

Rebalance requires free capacity

A data-moving rebalance keeps source copies until a separate cleanup pass, so receiving targets need free space for everything that moves. On highly utilized clusters, plan membership and placement-weight changes accordingly, and reclaim the source copies afterward with ais start rebalance --cleanup or ais storage cleanup. A future release will remove source copies while rebalance runs. See Head-Batch Verification.

GET enforces a minimum transfer rate

Covered GET paths now abort a transfer that falls below the minimum rate - by default, 18.75 MiB per 5-minute window (64 KiB/s on average) - in both directions: a client that stops reading receives a truncated response, and a stalled cloud backend read fails (cold GET: 504). Clients that intentionally pause long between reads, or that read through very slow links, should raise timeout.send_file_time. See Protection from Stalled GET Transfers.

Stricter authorization

With authentication enabled, some requests that previously succeeded now return 403. Review roles for callers that:

  • manage download jobs (list, query, abort, remove) without admin access;
  • start rechunk without read-write access to the bucket;
  • reset bucket properties without permission to set the bucket's access attributes;
  • link an AIS bucket to a remote one (backend_bck) without read-write access to the remote bucket;
  • use S3 CopyObject without PUT permission on the destination bucket;
  • copy or transform a single object with the native API without read access to the source bucket or PUT permission on the destination bucket (previously, AIS required PUT on the source and did not check the destination);
  • access a remote bucket for the first time - with an object request or a bucket job - without the permission that request requires.

See Security.

Signed copy redirects

With auth.intra_cluster.request_auth enabled, S3 CopyObject and native single-object copy (or transform) use an updated signature payload: S3 CopyObject binds the source bucket, and native copy binds the destination. During a rolling upgrade, these copies fail verification when the proxy and target are on opposite sides of this change; retry after both have been upgraded. This protection requires request signing to be enabled; unsigned redirects bind neither.

S3 endpoints and request paths

  • AIS refuses, at connection time, requests to a bucket with a per-bucket S3 endpoint that resolves to loopback, link-local, multicast, CGNAT, or another blocked address range. Private address ranges remain allowed for S3-compatible storage. For a local S3-compatible endpoint, use the S3_ENDPOINT environment variable instead.
  • AIS rejects S3 requests with consecutive slashes in the path (InvalidArgument).

Promote

Promote rejects symbolic links in any component of a source path, and any error while walking a source directory fails the job (previously, promote skipped it). With --delete-src, promote always copies sources and then removes them; it never renames them.

Blob download and range reads

AIS rejects (400) a range read or archive read that explicitly requests blob download (Ais-Blob-Download). When only the size threshold selects blob download, such reads fall through to a regular GET. See Reading large objects from remote buckets.

Chunked storage

Changing a bucket's chunks properties now starts rechunk - plan such changes on large buckets as you would mirror or ec changes. AIS rejects a single properties update that would start more than one such job - for instance, changing chunks together with ec. API callers of rechunk must stop sending chunk-size and objsize-limit; see Removed and Deprecated APIs.

Native Go API: multipart upload

api.CreateMultipartUpload, api.CompleteMultipartUpload, and api.AbortMultipartUpload take argument structures. See Native Go API.

Python SDK 2.1: compatibility and blob download

BlobDownloadConfig adds threshold for size-based blob-download selection and read_timeout for backend range reads. Existing configurations remain compatible; omitting threshold preserves forced blob download, while zero disables it. ObjectReader.raw() is deprecated; use stream() and the response's raw field. Replace calls to DynamicBatchSampler.__len__() with num_samples() where a sample count is needed, and update explicit writer type checks to io.BufferedIOBase. See Blob Downloader: Streaming GET.

Acknowledgments

Several security fixes in this release address issues reported by external researchers.
We are grateful to the people who took the time to investigate AIStore and report what they found:

  • Habibullo Izzatilloyev (TrendAI Zero Day Initiative):

    • a per-bucket S3 endpoint could resolve to loopback, link-local, and other blocked address ranges, letting targets send backend requests to internal services (9e3a5a4c1)

  • Santhosh Kumar (TraceX Labs):

    • listing, querying, aborting, and removing download jobs did not require admin access (0ed8d1ab3)

  • zx (Jace) (@manus-use):

    • object POST actions, including blob download, did not validate object names before dispatch (84f0057c1)

  • NguyenNguyen (@nguyennguyen0206):

    • destroy-inventory joined an unvalidated inventory name into an on-disk path (f9c0bd0d3)
    • S3 CopyObject checked the destination bucket's ACL but not the caller's PUT permission on the destination bucket (2cf759234)
    • creating an AIS bucket with backend_bck, or setting it on an existing one, did not verify the caller's read-write access to the referenced remote bucket, letting a caller with PATCH (or create-bucket) permission read, list, overwrite, and delete that bucket's objects with the cluster's backend credentials (74e437f85)
    • promote followed symlinks under the promote root, letting a caller with promote access read, and with DeleteSrc delete, files outside the root (0b83e6c18)

  • Muted Cybersecurity (@MutedCybersecurity):

    • multipart completion did not verify that the request's bucket and object matched the upload, so pending parts could be published under another bucket or object (48cba6bb6, ef29f641a)
    • starting rechunk required no access permission, so anonymous and unprivileged callers could start it with AuthN enabled (ccae2abab)
    • resetting bucket properties restored the default access attributes while requiring only PATCH, letting a user without SET-BUCKET-ACL lift a bucket's access restrictions (821ac9712)

  • btncwn (@btncwn):

    • with intra-cluster request signing enabled, the signed S3 CopyObject redirect did not cover the copy source, letting a client replace it when following the redirect and copy from a bucket it could not read (391b6f4d0)

  • seslo82 (@seslo82):

    • native single-object copy (and transform) checked PUT permission on the source bucket instead of read access, and did not authorize the destination bucket (01974de15a)

Reports are handled per SECURITY.md.

Assets 10
Loading