3.21

Highlights

• cold GET: extract and micro-optimize the flow
• sync Cloud bucket
  • leverage validate-warm-GET bucket config, and
  • extend it to support non-versioned Cloud buckets, and
  • optionally, delete (remotely deleted) objects
• bucket sizing and counting:
  • support very large buckets that are not necessarily present in the cluster;
  • unify ais ls --summary and ais storage summary to utilize the same control message and flags
• list, summarize, and lookup the properties of remote buckets without adding them to cluster's BMD
• HTTPS:
  • support TLS configuration to authenticate clients
  • switch cluster from HTTP to HTTPS, and vice versa
• optimize metadata cache
• optimize capacity management
• bug fixes, performance improvements

Core

• set prime-time to amend local generation of globally unique IDs !6325
• multi-object (archive, copy, transform) jobs: transport endpoint !6326
• core: (maintenance, decommission, shutdown) transition w/ rebalancing !6327
• core: (maintenance, decommission, shutdown) transition w/ rebalancing !6328
• intra-cluster transport: make receive-side stats optional !6329
• intra-cluster transport: reduce receive side contention !6330
• fix channel full condition; rebalance-cluster; transport !6331
• feature flags: add limited-coexistence; transport: track closed endpoints !6334
• fix prime-time: add caller-is-primary; up cli module !6335
• switch existing cluster between HTTPS and HTTP !6336
• Go 1.21: use built-in min and max functions !6337
• list-objects(remote-bucket-and-only-remote-props); Go 1.21 clear built-in !6339
• Go 1.20: use typed atomic pointer, remove unsafe !6343
• core: assorted micro-optimizations; remove read locks !6346
• tweak multi-error join-err, remove error channel (minor) !6347
• [API change] capacity management !6348
• xxhash; field-align vol package !6349
• bucket: new-query help; silent GET; test tools !6350
• etl: adding fqn param to spec templates !6351
• low-level control structs: bucket, namespace !6352
• etl: Keras template fix !6355
• etl: fix hello-world ais-etl tests !6356
• core: don't recompute uname hash !6359
• repackage HRW methods !6361
• core: lom cache v2 (major update) !6362
• refactor: downloader's diff resolver; control plane (receive BMD) !6363
• core: lom metadata cache (cont-ed) !6365
• dsort: error handling, assorted cleanups, more scripted tests !6366
• core transactions: concurrency !6368
• downloader: throttle; wait !6369
• optimize cold GET !6370
• global rebalance: log; minor edits !6373
• core: update backend 'get-reader' API (all supported backends) !6374
• core: validate-warm-get to support non-versioned buckets, and more !6375
• validate-warm-get to support non-versioned buckets !6376
• [API change] silent HEAD(object) request !6378
• core: add load-unsafe (the faster way to load local metadata) !6382
• total disk size: compute at startup, recompute on change !6383
• [API change] new bucket summary; unify list-objects and summary !6384, !6386, !6387
• add config.Rom to consolidate assorted "read-mostly" config values; refactor and unify !6388
• [API change] new bucket summary (major update) !6390
• mountpath jogger: support bucket query !6392
• backend providers: do not include (checksum, version) if not asked to !6394
• python: updated bucket info API !6395
• feature flags: don't-add-remote & don't-head-remote; log: add s3 module; verbosity; !6398
• support listing remote buckets without adding them to cluster's BMD !6399
• concurrent HEAD(object) vs evict/create bucket - fix the race !6400
• [API change] list and summarize remote buckets without adding remote buckets to cluster's BMD !6401
• datapath query (dpq) !6402
• Go-based API: response header to error message !6403
• [API change] new bucket summary !6405, !6406
• downloader: streamline and cleanup initialization sequence !6409
• HTTPS: support TLS configuration !6410, !6411, !6412, !6413, !6414, !6415, !6416
• assorted minor fixes !6417, !6418
• core: cold GET: fast path & slow path !6419
• cluster configuration: flip validate-cold-get !6420
• downloader (major update); [API change]: xaction registry !6422
• validate-warm-get: add scripted test utilizing remote ais cluster !6423
• core: cold GET: fast path & slow path !6424, !6427
• feature flags: add disable-fast-cold-get; show performance latency; up cli module !6425
• refactor ais/utils !6429

Bench: aisloader and aisloader-composer

• skip list objects for 100% put load !6332
• composer: add playbook and script for intial aisloader copy !6333
• composer: add support for aisloader --filelist option !6345
• default value for duration should be infinite if num-epochs value is defined !6353
• composer: add epochs option for GET workloads !6354
• composer: add cluster name prefix to netdata sources for easier filtering !6357
• new bucket not to be listed; usability !6358

CLI

• typed does-not-exist error; misc !6358
• always print dsort job description !6367
• show cluster to report total num disks !6371
• show performance: usability fixes, improvements !6379
• show performance not to filter regex-selected zero columns !6380
• attempt to copy/transform an empty remote bucket !6393
• new bucket summary; evict multiple buckets in one shot; pretty print !6396, !6397
• ais show bucket with an option to add remote bucket to cluster's BMD (effectively, create bucket) !6404
• ais search: CLI command search results to include idiomatic extensions !6428

Build, CI, Deployment

• tests: upon node shutdown: wait for the node to stop (tcp) listening !6338
• CI: add gather-logs template for K8s tests !6340
• deploy: ais with HTTPS in minikube !6364
• build: bump urllib version !6372
• tests: validate-warm-get (scripted) !6423
• K8s playbooks: update kill aisloader command !6385
• docs: validate-warm-get; assorted !6377
• docs: add performance.md; inline help; rm all-columns flag (redundant) !6381
• build: upgrade all minors !6389
• CI: add checkmarx scan !6391
• build: upgrade golangci-lint, add linters !6407, !6408

3.20

Core

• tweak stop-maintenance logic; rebalance: cleanup log messages; assorted minor fixes !6288
• do not timestamp err-aborted message !6290
• [API change] dsort: remove extended metrics; add new counters; revise and refactor !6297
• list-objects; house-keeper; aisloader, logger (assorted fixes) !6298
• core stats: remove mutex and work channel - speed up !6299
• slab allocator: remove stats mutex, do not sort !6300
• consolidate and revise OOM handling !6301
• ETL: require admin access to create & delete; add feature flag !6302
• remove unused heartbeat tracker w/ minor ref !6308
• reimplement keep-alive mechanism (major) !6309
• keep-alive v2 (major update) !6312
• keep-alive v2: remove timeout stats (control structure and code) !6317
• keep-alive v2: add fast path !6320
• micro-optimize get-all-running (jobs); atomic heard-from/timed-out !6321
• node-restarted: remove 'lsof', use net dialer; fix node-decommissioning tests !6322

Tools and tests

• CI: update fspath (aka mountpath) config for minikube-based aistore deployments !6289
• aisloader: list and read s3 buckets directly !6291
• aisloader: list, read, and write s3 buckets directly !6292
• tests: K8s long tests (EchoGolang) fix !6293
• aisloader: fix cleanup option for s3 bucket benchmarks !6294
• aisloader: reimplement direct get from s3 - use SDK !6295
• aisloader: show progress when listing s3 directly !6296
• CLI: add show details param to etl !6304
• tools: add check for ais etl deployment !6305
• tools: add ETL_NAME var for CLI tests !6310
• aisloader-composer: add playbook and script for clearing Linux Page Cache on all AIS targets !6311
• aisloader-composer: add playbook for copying aws credentials !6314
• tools: update check for aistore Kubernetes deployment !6315
• CI: update github action version (all modules) !6316
• CLI/ETL: support enumerated arg-type !6287, !6323

Build

• upgrade all OSS packages (minor versions) !6313
• transition to Go 1.21 !6318

3.19

Core

• [API change] archive and download logs (feature) !6172, !6175
• [API change] dsort: extend input format !6181
• [API change] dsort spec; CLI: print job spec !6204
• [API change] revise request spec (major upd) !6217
• [API change] dsort: is now 'xaction' as well !6253
• (downloader, dsort, ETL): disallow to run when out of space !6235
• handle "DNS lookup fail" as one of the unreachable err types; nlog flush-exit !6164
• when electing new primary; when joining nodes at startup !6165
• k8s: Change prod k8s and docker default to not log all to stderr !6166
• revise GFN !6167
• stats runner is now responsible to periodically flush logs !6170
• core: fail user attempt to abort global rebalance when !6184
• new Go API; assorted fixes !6189
• metasync BMD; up modules !6190
• downloader: return not-found when not found !6196
• start using scripted integration tests; CLI: 'dsort src dst spec' !6198
• support S3 AWS profiles with alternative creds (feature) !6214
• core: state transition => rebalance => (point of no return) !6216
• amend low-level Go API check-response routine; add error type-code !6228, !6229
• control plane: deserialize original error from call result !6230
• xactions: when checking inactivity ("is idle") !6242 !6243
• primary readiness vs cluster shutdown !6244
• Go API: wait for xaction-related conditions !6245
• assorted tuneups: space cleanup; housekeeping (HK) callback; log !6246
• access control: when copying/transforming/dsorting to non-existing 'ais://' destination !6255
• core: a call to update stats should never block !6257
• core stats: add fast counters !6258 !6259 !6261
• sparsify latency stats !6260
• ETL: refactor and cleanup construction !6267
• deploy/dev: updated minikube scripts !6272
• new option to add Cloud bucket to aistore without checking accessibility !6275, !6277
• un-throttle PUT mirroring; assorted changes !6278
• feature: local generation of global (job) IDs !6280 !6282

Performance

• Add distributed loader scripts and playbooks for using aisloader with multiple hosts !6156
• pyaisloader: usability improvements !6215
• Update Grafana dashboard to include latency statistics !6249
• Reorganize benchmarks and related tools !6254
• aisloader: no need to call rand for 100% or 50% read/write workloads !6256
• aisloader-composer: add dashboard for DC network and disk !6266
• aisloader: add an option to randomize gateways !6279
• aisloader-composer: fix output files for GET bench !6283

Python

• sdk: update ETL templates (docker migration) !6168
• sdk: Release version 1.4.1 !6169
• sdk: ETL templates (compress + ffmpeg decode) !6185
• sdk: ETL templates (imagepullpolicy as always) !6191
• sdk: adding keras_transform template !6200
• sdk: ETL templates fix !6201
• sdk: ETL templates (ffmpeg decode transformer) !6205
• sdk: compress ETL template (updated usage) !6211
• sdk: torchvision sample transformer ETL template !6221
• sdk: fix comments (minor) !6240
• sdk: update version !6248
• sdk: increase timeout for torchvision transformer template (large image) !6252
• sdk: updated torchvision transform ETL !6262
• sdk: update dsort job info query and related tests !6265
• sdk: switch ETL init code 'transform_url' boolean flag to 'arg_type' string !6269
• docs: update ETL dev deployment for macOS !6163
• ETL: keras template minor fix !6213
• ETL: remove incorrect reference !6268
• ETL: add 'arg-type=FQN' (new) !6271

Datasets (resize, resort, and shuffle)

• [API change] dsort: extend input format !6181
• dsort input format: iterate list, iterate range !6186 !6187
• start using scripted integration tests; CLI: 'dsort src dst spec' !6198
• add test scripts; memsys: init gmm only once !6192
• refactoring and renaming !6193
• move/consolidate error types; continued refactoring !6202
• Go API change; add dsort/api.go; CLI: print job spec !6203
• [API change]: dsort spec; CLI: print job spec !6204
• CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
• dsort: continued refactoring (major update) !6208, !6209, !6210
• free sgl on error; feature: any extension !6212
• [API change] revise request spec (major upd) !6217
• create destination on the fly !6218
• record content path to retain full shard name !6219
• output shard size estimation (rewrite) !6223
• add is-compressed; refactor dsort-mem !6227
• compressable shards (major) !6231
• output ext; rcb buffer; fixes !6232
• duplicated records (full coverage & stress); fixes !6233
• fix tests; add stress !6234
• rename subpackage, fix comments, refactor !6237
• remove dsort-context, rewrite initialization !6238
• static/stateless shard readers/writers; refactor and simplify !6239
• two goroutines per each shard-distributing request !6241
• [API change]: dsort: is now 'xaction' as well !6253
• dsort: support generic abort-xaction API !6264
• no need to block when sending shard records !6286

CLI

• archive and download logs (feature) !6180
• clarify "copying" vs "transforming" and "cached" vs "present" !6183
• start using scripted integration tests; CLI: 'dsort src dst spec' !6198
• dsort: extend inline help, pretty-print job spec; update docs !6206
• dsort: Go API change; add dsort/api.go; CLI: print job spec !6203
• 'archive get' is now a shortcut (an alias) !6222

Build, test, and tools

• add test scripts; memsys: init gmm only once !6192
• tests and tools: cleanup around stop-maintenance, wait-rebalance" !6194
• deployment: update local deployment script to allow target-only deployment with defined primary host !6195
• deployment: optionally, skip deploying primary proxy !6197
• start using scripted integration tests; CLI: 'dsort src dst spec' !6198
• tools/generate shards: optimize buffer allocation !6224
• deploy/dev: Add ansible deployment scripts for deploying locally on multiple nodes !6199
• aistorage/CI docker image (lzma libraries) !6220
• tests: init with cleanup and without !6226
• CI: Retry stuck Python ETL tests in GitLab CI pipeline !6270
• remove aisfs (FUSE) !6273
• dev tools: readers; handle read from corrupted arch or non-arch !6250

Documentation

• update getting started !6161
• updated python sdk readme !6162
• update ETL dev deployment for macOS !6163
• update documentation with recent ETL changes !6173
• CLI/dsort: extend inline help, pretty-print job spec; update docs !6206

3.18

Core

• add htext to track restarted state; target run and misc !5966
• cluster rebalance (scenarios) !5969, !5971, !5973, !5974, !5975, !5977, !5980, !5983, !5986, !5987, !5989, !5991, !5992, !5993, !5995, !6002
• add 'cluster-ready' helper; use it to reinforce !5976
• cleanup better when decommissioning; previous BMD at startup !5979
• fs: reliable remove-all !5981, !5982, !5984
• yet another buf pool !5985
• do not modify cluster map when starting up; always skip logging idle disks !5988
• rebalance (scenarios, major update) !5992
• [API change]: core: rebalance (scenarios) !5993
• rebalance (major update); when receiving new cluster map !5995
• up modules; handle housekeeper registration race !5994
• 'not present in the loaded cluster map' and similar startup validation !5996
• shutdown or decommission a node that's already in maintenance !5998
• transport: never establish a streaming connection to the peer that's in maintenance (or will be) !5999
• metasync just-in-time; assorted refactoring (minor) !6001
• maintenance mode: pre & post vs keepalive & metasync; CLI: more colored cues !6004
• shutdown is also 'maintenance'; docs: adding-removing intro !6005
• add meta package !6006, !6007
• ETL: add arg-type parameter when initializing with code !6008
• archive v2: support empty template (tar entire bucket); atime !6013
• keep poi.atime in nanoseconds !6015
• archive v2: append to arch; refactoring !6017
• archive v2: up modules !6018
• archive v2: part four (major) !6019
• archive v2: detect an empty tar when appending, and handle !6020
• archive v2: part six !6022
• archive v2: mime detection !6024
• archive v2: extend 'append-to-arch' to support tar.gz !6025, !6027
• archive v2: tar and tgz append; fixes !6028
• log filenames; overlapping run vs node-restarted !6029
• archive v2: multi-object append-to-arch !6030
• archive v2: multi-object append-to-arch !6033
• archive v2: multi-object append-to-arch !6034
• cleanup disk utils (minor) !6035
• ios startup: run the command only once !6036
• hide AuthN secret !6038
• archive v2: append to zip !6041
• archive v2: append to msgpack !6043
• add cmn/archive package !6044
• archive v2: write and copy via new 'cmn/archive' !6045
• archive v2: append via new 'cmn/archive' !6046
• [API change] archive v2: MIME vs file extensions !6047
• ios: cleanup lsblk cache; CLI: refactor get-node-arg; up modules !6048
• archive v2: remove msgpack; refactor !6051
• archive v2: add '.tar.lz4' serialization (new) !6053
• archive v2: tar.lz4 cont-d !6054
• archive v2: lz4 features; checksum !6055
• s3 compat: run E2E tests with correct HTTP/HTTPS mode !6057
• [API change]: append to arch if exists !6062
• [API change] append to arch if doesn't exist; CLI cont-d !6064
• checksumming and buffering vs reader-from !6074
• core: content-length universally; revise write-json and friends !6075, !6076
• archive v2: [API change] put (files, dirs) with an option to append !6081, !6082
• archive v2: quiesce faster, refine continue-on-error logic !6083
• core: double-check target-in-maintenance, quiesce faster !6084
• archive v2: finalize cmn/archive package !6085
• archive v2: finalize cmn/archive package !6086
• log verbosity: core and modules !6087
• http client: disable compression; core: undefer & micro-optimize !6066
• append to (non-existing) arch: an option to create !6068
• mem-pool alloc/free symmetry: copy/transform & archive !6069
• copy/transform, multi-archive: refactor Rx logic and error handling !6071
• log verbosity: core and modules; remote cluster !6089
• log verbosity: core and modules; remote cluster !6091
• ec: minor refactoring !6092
• archive v2: WD basename; get with extraction; Range !6093
• archive v2: WD basename; get with extraction; Range !6094
• archive v2: tools/archive utils !6095
• archive v2: tools/archive utils !6096
• compile-out asserts; super-verbose logging; log module 'mirror' (ref) !6097
• log verbosity at runtime; log modules; remove glog; unify (major update) !6099
• fields iterator; size converter; log rotation (fixes) !6101
• [API change] get-bucket-info to count remote objects !6102
• [API change] get-bucket-info to count remote objects !6103
• [API change] get-bucket-info (part three); docs and CLI !6106
• list-objects vs buckets: revise and refactor, add validation, clarify !6107
• list-objects: introduce optional args (ref, cleanup) !6108
• list-objects: mem-pool msgpack buffers !6109
• kvdb: remove redundant err-not-found; amend dsort, downloader, authn !6110
• x-lso must idle more time !6111
• log modules (part three) !6112
• add nlog (new logger) !6113, !6122, !6124
• do not log perf counters when there's no change; sort the names !6119
• fix disk usage call for clusters on mac OS !6121
• log etl events: spec parsed, pod ready, hpull/hpush !6125
• cleanup fs.PathError; add object name validation !6127
• extend dsort to support .tar.lz4 !6128
• aisloader: cleanup output when running with json option !6129
• xactions to return extended error info !6130
• xactions: add and return errors (major upd) !6131, !6132
• xactions: error handling cont-d; bucket-scope multi-object op-s !6133
• jobs error handling and reporting; tests !6134, !6135, !6136
• use Go 1.20 join-err; reduce default pre-election interval !6141
• http request multiplexer !6142
• dsort: use common cmn/archive pkg (major upd) !6143
• archive v2: tar formats (USTAR, etc.) !6153
• archive v2: extend list-objects and GET to operate inside archives !6155
• archive v2: extend list-objects and GET to operate inside archives !6157
• control plane: consistently propagate cluster map !6158
• old EC metadata shouldn't terminate cluster-wide rebalance !6159
• core: when primary goes down it notifies !6160

CLI

• performance tabs: always show 'cluster idle' if idle !5997
• stop-maintenance w/ no rebalance; stats idle-ness (minor) !6000
• add transform-url flag (used when initializing ETL with code) !6010
• misc. improvements !6011
• mountpath completions; disable/detach; minor ref !6056
• de-spaghettify put handler !6059
• archive multi-object (cont-d) !6060
• archive v2: CLI put, append, alias, docs !6072
• move 'gen-shards' and extend it to support all formats !6073
• archive v2: CLI: APPEND-to-arch is now on-par with multi-PUT !6077
• archive v2: CLI: dry-run option is back !6078
• archive v2: CLI: destination naming; dry-run; tips and examples !6079
• list-objects: extended help, template with no ranges !6104
• error and warning verbosity (major update) !6105
• log modules (part two); CLI archive: pre-parse and add tip, advice !6100
• archive help !6126
• archive v2: CLI and docs !6149
• archive v2: multiple CLI updates and improvements; is-archive bit !6150
• remove k8s apimachinery pkg (minor) !6151

Python

• sdk/python: Add option to pre-import modules when initializing ETL with code !5963
• sdk/python: Refactor ETL to provide name once on object creation !5968
• sdk/python: Release version 1.2.0 !5970
• sdk/python: Fix python package links !5978
• sdk/python: improve SDK testing, fix bucket eviction keep_md parameter !5990
• sdk/python: Release version 1.2.2 !6012
• sdk/python: Add dSort support in SDK !6032
• sdk/python: Add wait to dsort abort test !6037
• sdk/python: Fix dsort abort test on faster systems !6039
• sdk/python: Add get_url option to bucket, object group, and objects !6042
• sdk/python: Add interface for AIS sources that can be accessed via list of URLs !6052
• PyTorch: Add new PyTorch DataPipe to iterate over URLs from various AIS sources !6058
• update lint-tests to work with Python 3.11 (pytorch unsupported) !6061
• sdk/python: Include user agent in SDK requests !6067
• sdk/python Use msgpack content type when listing objects in bucket !6070
• sdk/python: Add msgspec to pyproject dependencies !6088
• sdk/python: Release SDK version 1.3.0 !6090
• sdk/python: bucket summary + bucket info !6098
• sdk/python: bck info integration test fix !6115
• sdk/python: bck summary integration test fix !6117
• sdk/python: bck summ + bck info fixes (revert and cleanup) !6120
• sdk/python: pyaisloader !6123
• pyaisloader: bucket utils (minor fixes) !6139
• sdk/python: bucket summary and info (minor fixes) !6140
• pyaisloader: total size/count fix !6146
• sdk/python: Release version 1.4.0 !6152

Docs

• add lifecycle.md !6009
• WebDataset Blog Post part 1: Storing WebDataset in AIS !6014
• WebDataset Blog Post part 2: AIS ETL on WebDataset shards !6016
• update docs/tools and cmd/README !6050
• WebDataset blog post pt 3 -- PyTorch Datatpipe with ETL !6065
• archive v2: CLI put, append, alias, docs !6072
• archive v2: CLI: destination naming; dry-run; tips and examples !6079
• archive v2: CLI and docs !6149

Build, CI

• add local build options to k8s scripts and fix local registry !6021
• build: upgrade all minors !6040
• add pyaisloader stage !6148
• k8s: fix deployment scripts for compatibility with the latest aisnode image !6154

3.17

Table of Contents

• CLI v1.2
• Python SDK v1.1.2
• S3 compatibility and Botocore
• API changes
• Tests and Documentation
• Core: bug fixes and improvements
• Build and Continuous Integration
• Extensions: Downloader, dSort, ETL

See also:

• Blog: Transforming non-existing datasets
• Blog: Python SDK: transform and load into PyTorch

CLI

• show all jobs !5645
• start/stop job/xaction !5660
• refresh rate and countdown; long-running 'show job' and friends !5651
• 'show log node-name' to mimic 'tail -f' !5652, !5654
• add custom duration flag and logic !5655
• 'ais config (cluster|node|cli)', 'ais config reset', and friends !5656
• bucket completions !5657
• set-config to show all updates; tweak iter-fields reflection !5658
• 'show job' to aggregate all categories and support all selections !5661
• transition to using job display names (major) !5663
• start, stop, show jobs and xactions (cont-d) !5665
• amend and restructure jobs !5666
• running xactions (completions) !5672
• tweak config json printout; get-config from memory !5673
• update backend config !5674
• update backend config (part two) !5677
• add footnote, marshal message only once !5678
• remove xaction term and subcommand (everything is job now) !5692
• suggest (targets, proxies, nodes) !5694, !5696
• revise bash completions script !5697
• remove 'xaction' (term and subcommand) !5698, !5699, !5700
• 'show cluster': separate cluster nodes from all other (tab-tab) completions !5701
• consolidate and refactor cluster map access !5704
• tweak ais create bucket --props & ais bucket props set !5706
• extend 'job start' to support (resilver, copy-bucket, rename-bucket) !5715
• tweak listed props !5719
• remove (cleanup) download and dsort jobs !5721
• extend 'ais stop' to support --all|--regex !5722
• 'show job' verbose option; unify usage args; ref PUT/APPEND !5723
• rewrite command-not-found logic; add similar commands !5724
• show jobs (major) !5726, !5727
• bash autocomplete ordering improvements !5728
• improvements (usability) !5729
• add bucket cp alias !5730
• flag printable name; split 'show job' in parts; usability !5736
• further unify stopping, waiting-for, and showing jobs !5744
• revise & amend 'show rebalance' - all permutations !5761
• universal start-end formatting; template refactoring !5762
• jobs grouping by name and, within name, by UUID !5764
• complete etl-name transition !5767
• ETL tools, UUID (part one) !5745, !5746, !5749, !5753, !5754, !5763
• fix download/dsort progress !5769
• new table to show target statistics !5788
• 'ais show performance' (new) !5791, !5793, !5800, !5802, !5803, !5809, !5810, !5811, !5812, !5816
• IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5820
• reduce code, simplify, cleanup !5821
• IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5823
• disk stats: add average read/write sizes !5824
• amend existing mountpath tab and add a new one !5833, !5834
• expect node unreachable when iterating '--refresh' !5837
• assorted usability; add 'no-color' config !5839
• 'ais show performance': average (GET, PUT, etc.) sizes on the fly !5840
• support new API to reset stats !5841, !5843
• 'ais show performance': refactor throughput, add latency !5844
• 'ais show performance': finalize latency tab !5847
• 'ais show performance' cont-d !5848
• 'ais show performance': finalize top-level tab !5850
• 'ais show performance': add cluster-level throughput, beautify !5852
• 'ais show performance': alias 'stats' and remove older code !5853
• 'ais show performance': disk table v2 !5855
• 'ais show performance': finalize disk table !5857
• 'ais show performance': new mountpaths/disks/capacity table !5858, !5859
• 'ais show performance': finalize capacity table !5861
• refactor and cleanup multi-object put !5862
• multi-object PUT: source dir, list/range; matching pattern !5865
• fix concatenation logic, refactor progress bar !5867
• copy bucket: support progress bars (copied objects and size) !5870
• consistent timeout management !5871
• copy/transform a list or range of objects: add progress bar !5873
• copy/transform with progress bar: style, reuse !5874
• multi-object PUT !5876
• rogress bar: all multi-object operations; universal 'wait-for' !5879
• PUT multi-object - all flavors !5880
• get multiple objects in one shot (""multi-object GET"") !5884
• GET destination & assorted fixes !5882
• copy-bucket: prepend prefix, command helps, examples !5889, !5891
• more inline help !5892
• assorted improvements (minor) !5900
• fix downloading with progress bar enabled !5903
• how-to text: how to reconfigure remote ais cluster !5932
• add CLI compatibility warning (new CLI vs old cluster and vice versa) !5952
• cluster membership-changing operations (shutdown, decommission, et al): improve usability !5956

Python: SDK library and ETL

• Add unit tests for cluster class !5675
• Add unit test for api client class !5676
• Fix python ETL test workflow !5680
• Add unit tests for Xaction class !5681
• Add unit test for object class, fix object put data from a filepath !5682
• Add unit tests for ETL class !5683
• Add unit tests for sdk utils !5685
• Restructure python subdirectories and containment !5686
• Run python unit tests as part of default python test make option, include python ETL tests in all python test runs !5688
• Only run python ETL tests when python labels are added !5702
• update test utils to support running Python tests on Windows !5716
• Add multi-object functionality !5720
• Add tests and validation to object ranges, support leading zeros !5731
• Add unit test for object group class !5732
• Update documentation for python multi-object ops !5734
• Use aws bucket for python sdk CI tests to test caching functions !5739
• Add string template support to object groups !5741
• Refactor all references from xaction to job !5742
• Update ETL runtime defaults and add new python 3.11 option !5743
• Fix remote bucket tests to avoid collisions !5748
• Increment cloudpickle version and update github action to use 3.11 for tests !5756
• Fix python test dependencies !5780
• Refactor to simplify typing and tests !5789
• Bump python SDK to 1.0.5 !5790
• Update python sdk version !5792
• Update pytorch integration README with compatibility issue !5794
• Patch sdk to support torchdata integration !5795
• Bump aistore package version !5798
• Fix pylint and formatting !5799
• Add ObjAttr type for returning additional object metadata !5805
• Address lint warnings, general improvements !5808
• Add writer option to object get !5813
• Split README for different python projects !5814
• Add PROMOTE functionality to objects !5818
• Standardize pylint version, fix lint errors !5822
• Improve object put behavior and add directory put options !5826
• Use Pathlib over os.path !5827
• Refactor multi-file put to bucket class !5830
• Improve job interface and update promote options !5832
• Update python package build tools, increase version to 1.0.9 !5835
• Improve example documentation !5842
• Improve input validation !5846
• Set up proper logging, update constants !5849
• Add job wait for idle status and fix job bucket filter !5863
• Add multi-object copy !5866
• Fix remote test fixture !5877
• Add multi-object ETL !5878
• Release 1.0.10 !5881
• Expand multi-object examples !5883
• Improve usability !5888
• Follow-up for copy/transform prepend !5890
• Improve object interface !5893
• Add prefix_filter option to bucket copy !5899
• Add flags and target options to bucket list_objects methods !5901
• Release 1.1.0 !5910
• Release 1.1.1 !5911
• Improve python sdk examples !5913
• Add cluster list_jobs, require id for individual job status query !5916
• Add support for multi-object archive !5920
• Update bucket params to use Bucket object, support namespaces !5925
• Add prefix filter to bucket transform !5949
• Release 1.1.2 !5950
• PyTorch: add support for etl-name in AISDataset and AISFileLoaderIterDataPipe !5957

S3 compatibility and Botocore

• Add botocore monkey patch alongside python SDK !5684
• Move botocore and pytorch packages to python top-level, separate from sdk !5691
• Add s3 compatibility testing with boto3 !5703
• list-objects vs HEAD !5705
• compute multipart md5 and set etag !5708
• Improve s3 compat test documentation and update validated tests !5747
• Pass S3 delete a list of objects !5854
• fix infinite loop when listing objects; add bucket name to list object response !5864
• return bucket creation date in UTC !5869
• new flag NoRecursion to support S3 delimiter feature !5930

API changes and new APIs

• add API to query multiple xactions via any IC proxy !5670, !5671
• yet another API to query xactions (new) !5687
• list-objects API: tweak listed props !5718
• [API change] flatten xaction-snap control structure (major upd) !5740
• [API change] ETL: tools, UUID (part nine) !5754
• [API change] ETL: new query parameter to specify transform name !5755, !5765
• [API change] init/start ETL to return xaction ID !5768
• [API change] GET(object) !5770, !5772, !5773, !5775
• [API change] remove xaction 'query-msg' (deprecated) !5783, !5784, !5785
• get-object API: amend comments !5769
• refactor api package (major) !5776
• [new API] query metric names and kinds ('counter', 'latency', 'throughput', et al.) !5796
• [API change] get node status !5811
• [API change] core stats: consistency between stats-querying APIs !5817
• [API change] PUT(object) !5828
• [API change] amend capacity-disks-filesystems control !5831
• [API change] add API to reset stats !...

3.16.rc2

v1.3.16

v3.16.rc2

3.12

Table of Contents

• Remote AIS clusters
• List objects
• List and summarize buckets
• S3 API
• CLI
• Documentation
• Tests
• Python SDK & ETL
• Core
• Refactoring & fixes
• Tools
• Build & CI

See also:

• Blog: AIStore 3.12 Release Notes

Remote AIS clusters

• list remote ais buckets; fix remote namespace containment !5567, !5621
• aliasing and bucket namespaces (major revision) !5622, !5623
• ais gateways to cache remote ais clusters (information) !5630
• extend remote ais errors - add cluster UUID !5631
• remote ais aliasing !5632
• [API change]: remote ais clusters (major) !5633, !5634
• [API change] add HealthUptime; use it to show remote clusters !5636
• remove support for multi-aliasing !5637
• alias vs bucket metadata !5638

List objects

• CLI to show CACHED (column) only when it's informative (and not when user asks ais ls ... --cached) !5554, !5555
• rename list-objects structures; regenerate message pack !5542, !5543
• amend and re-enforce single target rule for remote buckets !5556
• introduce location property, phase-out target URL (property) !5557, !5559
• add custom metadata, regenerate message pack; refactor !5564, !5565
• list-objects version 2 (major): !5583, !5584, !5586, !5589, !5590, !5591, !5601, !5602, !5605, !5607
• [backend API change] mem-pool 'list-objects' pages !5604
• list-objects flow with a single target listing remote props (name, size, version) !5608
• list remote pages v2: flow controls; ais gateways will now select single target to execute remote call !5609, !5610
• list-objects to include (in listing result) custom object properties !5614
• consistency in custom props across: get, put, head object, and list-objects !5615, !5616

List & summarize buckets

• easy URL to support listing buckets via /gs, /az, and /ais endpoints !5486
• list-buckets and get-bucket-info API; CLI summary table !5521, !5522
• list-buckets version 2 (major update) !5524
• unify 'list-buckets' and 'bucket-summary' !5530, !5531
• shorter and faster list-buckets version: w/ presence but wo/ summary !5535, !5536- [API change] add presence filter; eliminate bucket-info control structure (is redundant); refactor !5540
• list-buckets and bucket-summary cont-d !5541
• bucket sizing: always compute two sizes: on-disk and sum objs !5544
• bucket-summary on-disk (apparent) vs sum-all-objects, remote vs cached !5545
• bucket-summary and list-buckets cont-d unification !5546, !5547
• list-buckets by target: remote ais; hdfs and http; cloud; other !5567
• rename/move bucket summary; align control structures !5568
• list remote ais buckets; fix remote namespace containment !5621
• [API change] list-buckets correctness for named bucket queries !5626

S3 API

• multipart upload !5458, !5460, !5467, !5468
• extend s3 API to support remote buckets (major) !5462
• persistent multipart state (major) !5466
• multipart GET !5472
• multipart list active uploads and upload parts (major) !5474
• return xml-formatted errors (compliance) !5480
• multipart: refine and simplify !5482
• revise list-buckets to include all providers !5485
• always set header Server when redirecting !5495
• amend s3 compatibility readme; add multipart example !5496
• multipart: sha256 fix; md5 not checked but computed; aws s3api example !5500
• xml-format errors (parts two and three) !5501
• revise HEAD object implementation; multipart ETag; miscellaneous fixes !5504
• compatibility: configurable root '/' access to AIS cluster !5572, !5573, !5574
• compatibility: assorted error codes !5620

CLI

• 'ais etl' to replace extension in transformed objects !5418
• standardize error and warning coloring !5493
• don't log and don't show backend secret !5506
• improve unknown-command messaging, provide hints !5512
• add ais object ls command; add present flags !5514
• support updated HEAD(bucket) API; 'ais ls' and permutations to show obj props !5517
• static linkage with cgo disabled !5526
• rename aws, et al. backend provider constants !5528, !5529
• option to display sizes either in bytes or KiB, MiB etc. units !5532
• rename sub-packages; show-unmatched & hide-header !5534
• bucket display-name helper (and consistent usage) !5539
• flip bucket listing default to 'present' (was 'all accessible') !5550
• option --all to list both buckets and objects (in remote buckets) !5551
• multi-feature completions; action-warn and action-done !5575
• show config subsections in JSON; multi-feature completions !5576
• amend config JSON printout !5577
• extend and amplify: (config | section | json) !5578
• is-time-not-set, is-object-not-cached (for consistency) !5611
• atime & ctime !5612
• throughput must be averaged over an interval of time since the previous request !5643

Documentation

• amend main readme; add Reuse section !5375, !5376, !5378
• blog: Python SDK: Getting Started !5373
• add curl example to set backend bucket property !5380
• python/sdk: ETL !5397
• amend main readme: add PyTorch integration section !5406
• remote backend; native integration !5407
• revise ulimit, reference checked-in version of limits.conf !5429
• update CONTRIBUTING.md !5459
• blog: AIStore: Data Analysis w/ DataFrames demo !5469, !5473
• update s3cmd readme - add multipart example !5483
• restructure and extend getting-started; revise and amend Local Playground option !5484
• easy URL: add readme with extended examples and comments !5487
• REST API querying information !5641
• REST API listing buckets (remote namespace, presence and existence) !5644
• update ETL main readme, CLI readme, and examples !5471
• amend s3 compatibility readme, add multipart example !5496
• main readme: Installing from release binaries (add section) !5640

Tests

• python ETL !5410, !5461, !5476
• python IO communicator !5415
• tests: skipping short (minor) !5416
• pyhton/sdk: ETL stress test !5420
• add s3 multipart test !5482
• add ETL test for streaming with python runtime !5513
• re-enable and amend object-properties tests across all providers !5515
• amend list-buckets: add cases; up CLI !5569, !5570
• reinforce multi-object tests, fix bucket-exists helper !5606
• amend and fix multi-object range/list copy test !5619
• amend list-buckets test that checks expected result !5631

Python SDK & ETL

• ETL API part 1 !5370
• sdk version !5374
• code (re)formatting and misc !5381
• reorg Makefile dependencies !5390
• ETL APIs part 2 !5394
• bucket ETL transform API !5405
• bump sdk version to 1.0.3 !5408
• fix IO communicator fixes !5413
• fix transform bucket !5421, !5428
• add examples !5441
• add streaming runtime w/ refactoring and cleanup !5444
• streaming init-code with chunk-size param (major) !5446, !5447
• with temp debug !5448
• init_code changes for before and after functions !5449
• remove debug w/ minor ref !5450
• functions filter (new), transform, and before/after !5451
• runtime v2 fixes !5455
• bootstrapper; by reference !5457
• ETL API fixes !5464
• update readme (etl init_code) !5475
• extend hpush yaml with bitwise flags !5479
• remove unused ETL functions; add test for streaming with python runtime !5513
• update Jupyter notebook to reflect sdk changes !5548

Core

• global rebalance: initiating new vs receiving delayed ACKs !5426
• etl md: remove sgl, read-all !5470
• error handling: introduce type codes (major upgrade) !5477, !5481
• easy URL: support listing buckets via /gs, /az, and /ais endpoints !5486
• easy URL: add readme with extended examples and comments !5487
• user agent, internet browser: pretty-print JSON response !5492
• add User-Agent to http request headers !5494
• backward compatibility; remote backends !5498
• consolidate http headers; add type-cast fail !5503
• introduce error types unsupported and not-implemented-yet !5508
• refactor GET(object) !5507
• add lom.rename-file w/cleanups !5509
• simplify lom.persist & recache/refresh !5510
• update atime on PUT !5511
• HEAD(bucket): extend the API in re: cluster metadata !5516
• [API change] add api.GetBucketInfo !5519
• enumerate and unify bucket and object 'presence' vs 'existence' !5520
• fs: add size-bucket and refactor x-bucket-summary !5523
• backend providers as simple KVs (minor) !5525
• on the fly and (new) upon request remote bucket creation/addition !5538
• object props: remove get-node, add location and helpers !5561, !5562, !5563
• creating/adding buckets on the fly: further clarifications !5566
• debug pkg: assert-msg is redundant; minor ref !5579
• cos-assert; align (minor ref) !5587
• periodic stats log: tweak idleness logic !5588
• x-archive to always close data mover !5594
• key-value d...

3.11

Highlights

• Python SDK v1.0.1 with PyTorch integrations
• Authentication and Access Control
• GitHub Action to deploy and test local playground
• Bug fixes, documentation updates, performance improvements

Authentication and Access Control

• CI: fix AuthN to run !5264
• replace form3tech-oss/jwt and upgrade to jwt/v4 !5266
• prep for production !5267
• prep for production (major update) !5268
• prep for production (part three) !5269
• prep for production (part four) !5272
• prep for production (aisloader) !5275
• prep for production (clean_deploy) !5277
• extend error info, add usage examples, tweak CLI help !5279
• revamp ais auth CLI !5281
• rename APIs for consistency; add CLI to update existing role !5283
• E2E test: added basic AuthN test !5284
• make cluster optional for role add/set; randomize E2E tests; unify auth-enabled checks !5285
• rename AuthN entities for consistency !5288
• add api package, isolate server internals (major) !5289
• continued refactoring, assorted fixes !5291
• add 'tok' package to consolidate all jwt calls; continued refactoring !5292
• status 'forbidden' or 'unauthorized' to accompany all errors !5310
• allow 3rd party client lookup ht://bucket !5312
• separate enabling HTTPS for AuthN and AIS cluster when deploying !5313
• oad-token: if user-given location is empty then !5314
• secret handshake when adding/updating AIS clusters !5316
• revisit AIS calling and retrying; unify errors !5317
• login by non-superusers: require cluster (to be explicitly specified) !5326
• assorted fixes, improvements !5333
• introduce 'show-cluster' permission; reinforce admin access !5350

Python SDK

• add copy_bucket method !5225
• add method xact_status !5226
• move test to cluster ops file !5232, !5233
• add xaction start !5237
• restructuring (adding pytorch) !5238
• PyTorch plugin (dataset/aisio) implementation for aistore !5242
• fixed bug - list empty buckets !5251
• changed error handling !5258
• add iterator for bucket objects !5259
• improved pytorch plugin !5274
• release 0.9.0 fixes !5276
• cluster health api !5280
• health check for non existing urls !5282
• syncing aisio.py !5286
• rename_bucket() port !5304
• test suite changes !5305
• makefile dependency fix !5307
• cluster health api !5278
• added aisio example !5322
• restructure api (bucket) !5323
• fixing aisio example !5325
• aisio example fixes !5328
• restructure cluster !5335
• obj streamer fixes !5337
• blog post for ais file loader and lister !5338
• using session for requests !5341
• restructure object !5342
• object method name changes !5343, !5346
• restructure fixes !5348
• bucket api fixes !5351
• object get fix !5352
• restructure xaction operations !5353
• fixes and release prep !5355
• minor readme edits !5356
• readme changes wrt pytorch !5357
• api docs and make file fixes !5359
• removing aistore.pytorch imports from main init !5360
• makefile fix (pipx path) !5367

GitHub Action

• add github action for building ais !5265
• add github workflows for lint, python-tests and build !5270
• add test-short github workflow (Linux) !5271
• github/workflow: changed go lint version !5366

CLI

• confirm assorted operations: decommission, rm bucket, rm -rf !5224
• revamp ais show cluster; add K8s POD names !5243, !5244
• bucket extra-props-to-update; filter by provider !5256
• allow setting permission on per-bucket basis for a role !5309
• revise 'show config', add nested completions !5339
• revise 'set config', add local vs inherited !5344
• add --cluster filter flag to 'auth show role' command !5349

Bucket Summary

• revise bucket summary (major) !5230
• bucket summary: micro-optimize and simplify !5262
• bucket summary: multi-bucket; add min/max/avg counters; fixes !5299
• bucket summary: apparent vs on-disk sizing, formatting clarity !5300, !5302

Common environment and system filenames

• move and refactor common environment; AuthN version and build time !5318
• filename constants; environment variables !5324
• consolidate all configs under $HOME/.config/ais !5327

Misc. Features

• warn when decommissioning cluster, support '--rm-user-data' option !5221
• ability to decommission node and join it back with a new identity !5222
• anonymous public-access buckets: list-objects anonymously (w/ API change) !5330
• s3: the capability to set custom s3 endpoint !5257, !5260

Docs

• tweak ETL markdown !5227
• updated getting_started and etl markdown !5228
• changes to standalone deployment README !5253
• minor indentation fix to standalone readme !5254
• standalone ec2 deployment instructions and benchmarks !5261
• python api reference !5294
• python_api.md fixes !5297
• bucket.md to reflect 'summary' changes !5308
• update 'ais ls', add anonymous and more !5336
• main readme !5358
• s3 compatibility, 's3cmd' configuration and getting-started !5361
• main readme, index.md, python.md !5362
• amend and extend cli/config.md; fix config node reset !5363
• add 's3cmd.md' !5364

Tests

• random page, remote bucket !5234
• remote bucket, object props !5235
• extend cleanup to support (prior to test, remote bucket) !5240
• ensure more !5245
• fix iter-fields unit for recently added config extras !5290
• reverse proxy test against GCS !5329

Build

• upgrade gopkg !5223
• fix statsd github URL !5229
• new (and supported) JWT packages !5266

Performance micro-optimizations, bug fixes, refactoring

• aisloader: rewrite SGL freeing piece !5236
• refactor low-level is-bucket, bucket-init, and friends !5239, !5241
• BMD: speed-up bucket init; add global namespace uname (const) to avoid allocations !5246
• BMD atomic update vs backend props !5247
• reuse buffer when iterating range template !5248
• devtools: refactor and simplify !5249
• bucket extra-props-to-update: include readonly; CLI: filter by provider !5256
• yet another case to skip err logging block devices !5273, !5295
• cluster config: allow critical updates !5298
• capacity: periodic log and refresh; version and build time; fixes !5315
• local playground: loopback device sizing; fixes !5321
• refactor and move docker pkg (formerly containers) !5331
• list-objects flags: unify, rename, document, and reference !5332
• config: restart-required selection; feature flags !5334
• blog image fix !5347
• assorted fixes and refactoring !5354
• CI/CD: skip-ci & python-tests-only conflict fix !5365

3.10

Highlights

• Global (cluster) config meta-version 2
• Bucket metadata (BMD) meta-version 2
• Python SDK
• Build: Go 1.18; upgrade all packages
• MessagePack archiving

Global Config v2 & Bucket Metadata v2

• bump up cluster config's meta-version; handle backward compatibility !5122, !5123, !5124
• redefine and extend feature flags !5132
• capacity and space watermarks (cleanup, low/high, OOS) !5143, !5144
• BMD meta-v2 with backward-compatible logic to load v1 !5145, !5150
• BMD meta-v2: simplify mirroring configuration !5151
• add transport section !5153
• support loading previous meta-version !5154
• readonly knobs; backward compatibility !5155
• introduce cos.size type and "kb", "mb", etc. suffixes !5156
• cos.size vs backward compatibility !5157
• add memsys section and tunables !5158
• log flush time interval !5159
• join-cluster-at-startup timeout !5161
• add log-stats time, add validators !5171
• continued refactoring !5172
• add tcb (transform-and-copy-bucket) section; stream bundle multiplier; compression !5173
• revise docs/configuration.md !5180

msgpack archiving; list, create, and read msgpack formatted shards

• msgp (generic) shard !5100, !5101
• fix preambles in the msgp generated files !5086
• alternative msgp; sgl as io.ByteScanner; Python unpack to test !5103
• assorted text fixes; more msgpack !5104
• archiving formats: add msgpack (in addition to tar, tgz, zip) !5107, !5108
• add integration tests !5100, !5101

Python SDK

• basic bucket and object operations !5109
• introduce pydantic; refactor list-buckets !5120
• CI: add Python SDK test !5112
• improve list-objects API !5163
• refactor API calls for usability: !5128
• move direct requests call out of list objects, create and delete bucket, cluster map !5130
• put-object & head-bucket API !5141
• improve list object API !5163
• fix return type in _request_raw method !5164
• add head-object method !5166
• add setUp and tearDown in tests !5167
• improve creating temporary file !5168
• add evict-bucket and delete-object; add props argument for list-objects !5175
• get-object returns a stream instead of the entire content !5184
• update README, docs !5178, !5190
• fix naming; add read-all method for streams !5196
• list_object to support pagination !5198
• improve object streaming; rename msg.py -> types.py !5203
• Python SDK tutorial !5218

CLI

• show cluster config feature flags in human-readable format !5133
• fix setting backend bucket props !5152
• add ais object rm --all to remove all in one shot !5195
• verbose and non-verbose multi-object operations !5197

Tests

• re-enable smoke-test for Cloud buckets !5098
• msgp (generic) shard !5100, !5101
• add missing wait for put-copies; BMD string !5204
• failback; devtools: wait-cluster; transport !5209
• dummy keepalive tracker !5211

Docs

• website: tech blog on promoting local and shared files !5082
• explain msgp usage; add msgp-update to Makefile !5085
• update msgp generator usage !5094
• msgpack !5110
• revise and amend configuration.md !5180
• performance tuning !5115, !5116
• development, first-time usage, production deployments (summary) !5127
• getting started (recommendations) !5129
• local playground, tracking access time (more edits) !5139
• fix references, add a reference !5140
• erasure coding (space utilization ratio, IO performance) !5142

Build

• upgrade fasthttp package !5092
• upgrade direct dependencies !5096, !5097
• transition to Go 1.18 !5106
• upgrade all dependencies except 1.18 !5113
• upgrade all dependencies except 1.18; v3.10-rc2 !5181
• upgrade all dependencies including 1.18; v3.10-rc3 !5186
• upgrade all minors !5200

Bug fixes and performance improvements; continuous refactoring

• skip bucket name, etc. validation when initializing lom !5083
• follow-up !5084
• unsafe string: inline and use it consistently !5087
• write-msg-pack (ref) !5088
• control-plane broadcast & unicast: result value factories and decoders !5089, !5090, !5091, !5093, !5095
• global rebalance: dynamic reg/unreg streams; housekeeper name suffix !5099
• fix sending http request body on redirect: recompute content-length !5102
• not restoring atime when !5105
• multi-object operations !5111
• local from-scratch deployment: add user-friendly tips !5114
• follow-up !5117, !5118
• write-policy: both metadata and data; bump up config's meta-version; backward compatibility (major) !5121
• deploy: fix Dockerfiles and related scripts !5126
• assorted usability tweaks !5134
• local playground: print ports AIS listens on !5135
• remove bench/soaktest; rewrite devtools/readme !5136
• config: write policy for data must be 'immediate' !5137
• docker + LVM deployment: no disks fix !5146
• move mock-bmd => cluster/mock (refactoring) !5147
• docker + LVM deployment: no disks !5148
• local playground: pgrep to check; quiet run-cmd !5160, !5162
• fix deploy.sh script for older bash versions !5165
• target startup: skip docker union mounts when enumerating drives !5169, !5170
• dedup fix HEAD(obj) system attributes !5174
• get-object lock/unlock (refactoring) !5176
• URI parsing, config validations, and assorted ref !5177, !5179
• making "rlock" exception to forcefully remove corrupted !5182
• dont-lookup-remote-bucket = true: proceed silently, log-wise !5183, !5185
• remove +build directive (obsolete) !5188, !5189
• get random proxy (ref) !5191
• cluster map (ref); node equality !5192
• PUT object vs remote backend returning status=503 !5193
• multi-object operation, range template (ref) !5194
• aisloader works !5199
• when shutting down warn with details !5201
• api: xaction wait-IC min waiting time !5202
• compute-checksum and clone-md (ref) !5205
• memsys: mem-free to include buff/cache !5206
• memsys: revise sys/mem (major) !5207
• lif to lom conversion: free on error; EC error handling !5208
• revise xs/rename-bucket !5210
• deferred unlock via lightweight lif (ref) !5212
• lom as a reader, lif as unlocker !5213
• get-from-neighbor (gfn): reg hk housekeepr under lock; abort to self-expire !5215
• lom copy: allocate with name !5216
• lom xattr (ref) !5217

3.9

AIS v3.9 is substantial productization and performance-improving upgrade. Much of the codebase has been refactored for consistency, with numerous micro-optimization and stabilization fixes across the board.

Highlights

• promote: redefine to handle remote file shares; collaborate when promoting via entire cluster; add usability options; productize;
• xmeta: extend to also dump in a human-readable format: a) erasure-coded metadata and b) object metadata;
• memory usage and fragmentation: consistently use mem-pooling (via sync.Pool) for all control structures in the datapath;
• optimistic concurrency when running batch prefetch jobs; refactor and productize;
• optimize PUT datapath;
• core logic to deconflict running concurrent xactions (asynchronous jobs): bucket rename vs bucket copy, put a node into maintenance mode vs offline ETL, and similar;
• extend and reinforce resilvering logic to withstand simultaneous disk losses/attachments - at runtime and with no downtime;
• stabilize global rebalance to successfully pass multiple hours of random node "kills" and restarts - node-left and node-joined events - in presence of stressful data traffic;
• self-healing: object metadata cache to support recovery upon mountpath events (e.g., drive failures);
• error handling: phase out generic fmt.Errorf and consistently use assorted error types instead;
• additional options to speedup listing of very large buckets (list-objects);
• numerous micro-optimizing improvements: fast datapath query (DPQ) and many more.

Promote files and directories

• refactor as a 2-phase transaction and auto-detect file share (initial) !4929
• auto-detect file share and distribute the work between target nodes !4945
• add test; add target node => IC notifications !4975
• extend test coverage; reinforce global UUID when promoting via entire cluster !4976
• rename api.Promote; add test permutations and checks !4985
• remove redundant control structures; cleanup !4987
• add API options delete-src (delete source) and overwrite-dst (overwrite destination)!4988
• fix extra-copy optimization with full refactoring !4989
• revise/optimize HEAD(object) implementation and utilize it when promoting with overwrite-dst=false (major) !4991
• extend object write transaction (OWT) to support the flow !4992
• support in (i.e., transmitted), out (i.e., received) and locally-promoted stats counters - files/objects and bytes !4993
• introduce confirmed file share; add user option not to auto-detect file share !5019
• CLI: add overwrite-dst and delete-src E2E tests !5024
• consolidate control, eliminate ambiguity !5045
• increase test coverage !5047, !5063
• add all test permutations to cover (ais | cloud | remote-ais) bucket vs. (non-redundant | EC | n-way mirror) !5068

ETL

• add CLI to view stored ETL code and specification !4925
• handle target-down; test !4933
• redefine and improve ETL API (!4947, !4966, !5022), including:
  • manage (CRUD wise) persistently stored ETL definitions
  • eliminate redundant URL path parameters
  • enforce uniqueness of the user-provided ETL name
• remove (obsolete) embedded ETL-specific annotations from the init spec (pod template)
• support stopping and restarting ETLs !5005, !5056
• update ETL docs and fix minor bugs !4984, !5022

Global Rebalance

• global rebalance status: always respond with total (cumulative) stats counters !4905
• generic fs.Walk for global rebalance (refactoring) !4889, !4930
• get-status & health !4934
• global rebalance status: reimplement to optimize !4936
• devtools: merge WaitForRebalanceToStart and WaitForAllResilver !4937
• tweak/optimize receive logic !4994
• abort via stage notifications from other target nodes (major) !5015
• transport streams vs receive errors; assorted fixes !5040
• tweak preemption logic (when rebalance triggering events arrive back to back) !5057
• assorted fixes: global rebalance vs n-way mirroring & resilvering !5071, !5072
• consistent renames and continuous refactoring !5075

Resilvering (in presence of drive failures and attachments)

• tools and stats: wait-for-all-resilvers, multi-snap API !4888
• resilvering vs copy management (major) !4865, !4866, !4867
• resilvering: tweak is-active time interval !4882
• support losing multiple disks (mountpaths) simultaneously !4884
• multiple overlapping add/remove disk operations: fixes !4894
• resilvering as scrubbing: recover objects to their expected (default) locations !4900
• resilvering: interval-of-inactivity multiplier !4974
• resilvering under stress in presence of lost mountpath(s) !5058

Asynchronous Jobs (aka xactions)

• when aborting and propagating abort to the control-plane caller, make sure not to lose the original cause for the abort !4886
• fix put-xaction finishing logic !4887
• aborting jobs: propagate the original cause through channels and APIs !4890, !4891
• revise lookup by only-running and/or by UUID !4897
• move xaction and xreg packages with refactoring !4898
• clarify running vs not finished xaction !4908
• registry: fix matching logic, remove redundant code !4911, !4912
• registry: amend housekeeping !4913
• registry: continued refactoring and cleanup !4914
• "limited coexistence" between running and about-to-run services (new) !4915, !4916, !4917
• xact package: revise and optimize abort-checking concurrency !4923
• registry: continued simplification and cleanup !4940
• reinforce global UUID for all cluster-wide xactions !4978
• IC notifications vs transactional xactions: same rules for all !4982
• more stateful info: propagate xaction reference all the way into local PUT flow (major) !4995
• copy-object -- xaction -- promote: continued refactoring !5002
• registry: micro-optimizations and cleanup !5028

CLI

• PUT: add an option not to load (skip loading) object metadata; amend docs; refactor and cleanup !4859
• add a command to view ETL code/spec !4925
• fix: do not add --help flag to the subcommands of subcommands !4926
• amend 'show config' to include CLI config (in addition to cluster and local configs); fix cluster-unreachable error !4983
• revise 'flag-is-set' for Boolean flags !5021, !5023, !5030
• copy bucket: add --force option !5042
• add start etl command !5056
• ais show cluster: add support for refresh=<time-interval> and count options !5076
• update CLI docs !5078
• enable 'ais show storage' and friends to run continuously and refresh periodically !5079

Testing

• test fixes to align with changes in the core !4861
• add ensure-num-mountpaths helper, and reinforce !4892
• use api.WaitForXaction instead of tutils.WaitForXactionByID !4893
• re-enable one fs-checker test, allow more time for mountpaths !4903
• add more checks when downloading object !4910
• extend CLI e2e promote test !4932
• WaitFor follow-up !4943
• fix e2e AuthN messages !4952
• retry upon failure to recover a damaged erasure-coded object !4986
• amend and extend EC tests !4990
• revise and enable bucket-rename-and-copy test !5060

CI/CD (continuous integration)

• add CI job that runs on multiple cloud buckets !5027
• add 1.18-rc1 version to build check !5044
• add test-short-minimal to test a single-node cluster !5046
• make AWS_REGION global env variable !5050
• update test-long stage !5059

Bug fixes, performance improvements; continuous refactoring

• LOM load to return distinct types: syscall-error and corrupted-error !4849
• LOM vs n-way mirroring: fix and revise caching of the metadata !4850
• list-objects: add fast mode --only-names !4851
• introduce permission to overwrite disconnected backend !4852
• api: refactor PUT API; fix devtools !4853
• optimize PUT latency by allowing not to load object metadata !4854
• aisloader bench: do not run goroutine per each PUT request !4855
• reinforce access time atime (major) !4856
• reintroduce no-metadata error; fix n-way stress; refactor !4857
• build: fix deprecation warning on MacOS !4858
• when copying objects differentiate between copying == mirroring and all other scenarios !4860
• simplify LOM from-fs logic !4862
• general: don't use regex to validate names and UUIDs !4863, !4864
• assorted fixes !4868
• preserve atime across LOM caches !4869
• storage cleanup: leftover copies, corrupted and missing metadata !4870
• refactor cmn and api packages !4871
• list-objects: use-cache option !4872
• consistently use HTTP status 507 throughout; assorted fixes !4875
• eliminate redundant mirroring !4876
• get-cold (aka cold-GET) follow-up !4877
• object write transaction (OWT) fusion !4878
• prefetch: support optimistic concurrency (major) !4879
• name locker: fit two structures into 24 bytes !4880, !4881
• disable/detach mountpath: graceful (admin request) and immediate (FSHC) !4883
• move health package under fs/health !4885
• bucket summary and obj-list query: move, refactor, and simplify !4895, !4896
• control-plane: always free call-results back to pool !4899
• api: eliminate code duplication !4901
• general: deprecate and remove query objects !4902
• refactor ais package !4904
• control-plane transactions: refactor, reduce code !4906
• initial ETL get API implementation !4907
• control-plane transactions: follow-up !4909
• introduce read-only (but still configurable) timeouts: cplane-operation and max-keepalive !4918
• intra-cluster transport streams: tweak termination logic !4919
• slab allocator: amend pooling of the SGL control structures !4920
• transport s...