Releases: NVIDIA/aistore
Releases · NVIDIA/aistore
3.21
Highlights
- cold GET: extract and micro-optimize the flow
- sync Cloud bucket
- leverage validate-warm-GET bucket config, and
- extend it to support non-versioned Cloud buckets, and
- optionally, delete (remotely deleted) objects
- bucket sizing and counting:
- support very large buckets that are not necessarily present in the cluster;
- unify
ais ls --summary
andais storage summary
to utilize the same control message and flags
- list, summarize, and lookup the properties of remote buckets without adding them to cluster's BMD
- HTTPS:
- support TLS configuration to authenticate clients
- switch cluster from HTTP to HTTPS, and vice versa
- optimize metadata cache
- optimize capacity management
- bug fixes, performance improvements
Core
- set
prime-time
to amend local generation of globally unique IDs !6325 - multi-object (archive, copy, transform) jobs: transport endpoint !6326
- core: (maintenance, decommission, shutdown) transition w/ rebalancing !6327
- core: (maintenance, decommission, shutdown) transition w/ rebalancing !6328
- intra-cluster transport: make receive-side stats optional !6329
- intra-cluster transport: reduce receive side contention !6330
- fix channel full condition; rebalance-cluster; transport !6331
- feature flags: add
limited-coexistence
; transport: track closed endpoints !6334 - fix
prime-time
: addcaller-is-primary
; up cli module !6335 - switch existing cluster between HTTPS and HTTP !6336
- Go 1.21: use built-in
min
andmax
functions !6337 list-objects
(remote-bucket-and-only-remote-props); Go 1.21clear
built-in !6339- Go 1.20: use typed atomic pointer, remove unsafe !6343
- core: assorted micro-optimizations; remove read locks !6346
- tweak multi-error
join-err
, remove error channel (minor) !6347 - [API change] capacity management !6348
- xxhash; field-align
vol
package !6349 - bucket: new-query help; silent GET; test tools !6350
- etl: adding
fqn
param to spec templates !6351 - low-level control structs:
bucket
,namespace
!6352 - etl: Keras template fix !6355
- etl: fix hello-world ais-etl tests !6356
- core: don't recompute
uname
hash !6359 - repackage HRW methods !6361
- core: lom cache v2 (major update) !6362
- refactor: downloader's diff resolver; control plane (receive BMD) !6363
- core: lom metadata cache (cont-ed) !6365
dsort
: error handling, assorted cleanups, more scripted tests !6366- core transactions: concurrency !6368
- downloader: throttle; wait !6369
- optimize cold GET !6370
- global rebalance: log; minor edits !6373
- core: update backend 'get-reader' API (all supported backends) !6374
- core:
validate-warm-get
to support non-versioned buckets, and more !6375 validate-warm-get
to support non-versioned buckets !6376- [API change] silent
HEAD(object)
request !6378 - core: add
load-unsafe
(the faster way to load local metadata) !6382 - total disk size: compute at startup, recompute on change !6383
- [API change] new bucket summary; unify
list-objects
andsummary
!6384, !6386, !6387 - add
config.Rom
to consolidate assorted "read-mostly" config values; refactor and unify !6388 - [API change] new bucket summary (major update) !6390
- mountpath jogger: support bucket query !6392
- backend providers: do not include (
checksum
,version
) if not asked to !6394 - python: updated bucket info API !6395
- feature flags: don't-add-remote & don't-head-remote; log: add s3 module; verbosity; !6398
- support listing remote buckets without adding them to cluster's BMD !6399
- concurrent HEAD(object) vs evict/create bucket - fix the race !6400
- [API change] list and summarize remote buckets without adding remote buckets to cluster's BMD !6401
- datapath query (
dpq
) !6402 - Go-based API: response header to error message !6403
- [API change] new bucket summary !6405, !6406
- downloader: streamline and cleanup initialization sequence !6409
- HTTPS: support TLS configuration !6410, !6411, !6412, !6413, !6414, !6415, !6416
- assorted minor fixes !6417, !6418
- core: cold GET: fast path & slow path !6419
- cluster configuration: flip
validate-cold-get
!6420 - downloader (major update); [API change]: xaction registry !6422
validate-warm-get
: add scripted test utilizing remote ais cluster !6423- core: cold GET: fast path & slow path !6424, !6427
- feature flags: add
disable-fast-cold-get
;show performance latency
; up cli module !6425 - refactor ais/utils !6429
Bench: aisloader
and aisloader-composer
- skip list objects for 100% put load !6332
composer
: add playbook and script for intialaisloader
copy !6333composer
: add support foraisloader --filelist
option !6345- default value for duration should be infinite if num-epochs value is defined !6353
composer
: add epochs option for GET workloads !6354composer
: add cluster name prefix tonetdata
sources for easier filtering !6357- new bucket not to be listed; usability !6358
CLI
- typed
does-not-exist
error; misc !6358 - always print
dsort
job description !6367 show cluster
to report total num disks !6371show performance
: usability fixes, improvements !6379show performance
not to filter regex-selected zero columns !6380- attempt to copy/transform an empty remote bucket !6393
- new bucket summary; evict multiple buckets in one shot; pretty print !6396, !6397
ais show bucket
with an option to add remote bucket to cluster's BMD (effectively, create bucket) !6404ais search
: CLI command search results to include idiomatic extensions !6428
Build, CI, Deployment
- tests: upon node shutdown: wait for the node to stop (tcp) listening !6338
- CI: add gather-logs template for K8s tests !6340
- deploy: ais with HTTPS in minikube !6364
- build: bump
urllib
version !6372 - tests:
validate-warm-get
(scripted) !6423 - K8s playbooks: update kill
aisloader
command !6385 - docs: validate-warm-get; assorted !6377
- docs: add performance.md; inline help; rm all-columns flag (redundant) !6381
- build: upgrade all minors !6389
- CI: add
checkmarx
scan !6391 - build: upgrade
golangci-lint
, add linters !6407, !6408
3.20
Core
- tweak stop-maintenance logic; rebalance: cleanup log messages; assorted minor fixes !6288
- do not timestamp
err-aborted
message !6290 - [API change] dsort: remove extended metrics; add new counters; revise and refactor !6297
- list-objects; house-keeper;
aisloader
, logger (assorted fixes) !6298 - core stats: remove mutex and work channel - speed up !6299
- slab allocator: remove stats mutex, do not sort !6300
- consolidate and revise OOM handling !6301
- ETL: require admin access to create & delete; add feature flag !6302
- remove unused heartbeat tracker w/ minor ref !6308
- reimplement keep-alive mechanism (major) !6309
- keep-alive v2 (major update) !6312
- keep-alive v2: remove timeout stats (control structure and code) !6317
- keep-alive v2: add fast path !6320
- micro-optimize get-all-running (jobs); atomic heard-from/timed-out !6321
- node-restarted: remove 'lsof', use net dialer; fix node-decommissioning tests !6322
Tools and tests
- CI: update fspath (aka mountpath) config for minikube-based aistore deployments !6289
aisloader
: list and read s3 buckets directly !6291aisloader
: list, read, and write s3 buckets directly !6292- tests: K8s long tests (EchoGolang) fix !6293
aisloader
: fix cleanup option for s3 bucket benchmarks !6294aisloader
: reimplement direct get from s3 - use SDK !6295aisloader
: show progress when listing s3 directly !6296- CLI: add show details param to etl !6304
- tools: add check for ais etl deployment !6305
- tools: add
ETL_NAME
var for CLI tests !6310 aisloader-composer
: add playbook and script for clearing Linux Page Cache on all AIS targets !6311aisloader-composer
: add playbook for copying aws credentials !6314- tools: update check for aistore Kubernetes deployment !6315
- CI: update github action version (all modules) !6316
- CLI/ETL: support enumerated
arg-type
!6287, !6323
Build
- upgrade all OSS packages (minor versions) !6313
- transition to Go 1.21 !6318
3.19
Core
- [API change] archive and download logs (feature) !6172, !6175
- [API change] dsort: extend input format !6181
- [API change] dsort spec; CLI: print job spec !6204
- [API change] revise request spec (major upd) !6217
- [API change] dsort: is now 'xaction' as well !6253
- (downloader, dsort, ETL): disallow to run when out of space !6235
- handle "DNS lookup fail" as one of the unreachable err types; nlog flush-exit !6164
- when electing new primary; when joining nodes at startup !6165
- k8s: Change prod k8s and docker default to not log all to stderr !6166
- revise GFN !6167
- stats runner is now responsible to periodically flush logs !6170
- core: fail user attempt to abort global rebalance when !6184
- new Go API; assorted fixes !6189
- metasync BMD; up modules !6190
- downloader: return not-found when not found !6196
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- support S3 AWS profiles with alternative creds (feature) !6214
- core: state transition => rebalance => (point of no return) !6216
- amend low-level Go API check-response routine; add error type-code !6228, !6229
- control plane: deserialize original error from call result !6230
- xactions: when checking inactivity ("is idle") !6242 !6243
- primary readiness vs cluster shutdown !6244
- Go API: wait for xaction-related conditions !6245
- assorted tuneups: space cleanup; housekeeping (HK) callback; log !6246
- access control: when copying/transforming/dsorting to non-existing 'ais://' destination !6255
- core: a call to update stats should never block !6257
- core stats: add fast counters !6258 !6259 !6261
- sparsify latency stats !6260
- ETL: refactor and cleanup construction !6267
- deploy/dev: updated minikube scripts !6272
- new option to add Cloud bucket to aistore without checking accessibility !6275, !6277
- un-throttle PUT mirroring; assorted changes !6278
- feature: local generation of global (job) IDs !6280 !6282
Performance
- Add distributed loader scripts and playbooks for using aisloader with multiple hosts !6156
- pyaisloader: usability improvements !6215
- Update Grafana dashboard to include latency statistics !6249
- Reorganize benchmarks and related tools !6254
- aisloader: no need to call
rand
for 100% or 50% read/write workloads !6256 - aisloader-composer: add dashboard for DC network and disk !6266
- aisloader: add an option to randomize gateways !6279
- aisloader-composer: fix output files for GET bench !6283
Python
- sdk: update ETL templates (docker migration) !6168
- sdk: Release version 1.4.1 !6169
- sdk: ETL templates (compress + ffmpeg decode) !6185
- sdk: ETL templates (imagepullpolicy as always) !6191
- sdk: adding keras_transform template !6200
- sdk: ETL templates fix !6201
- sdk: ETL templates (ffmpeg decode transformer) !6205
- sdk: compress ETL template (updated usage) !6211
- sdk: torchvision sample transformer ETL template !6221
- sdk: fix comments (minor) !6240
- sdk: update version !6248
- sdk: increase timeout for torchvision transformer template (large image) !6252
- sdk: updated torchvision transform ETL !6262
- sdk: update dsort job info query and related tests !6265
- sdk: switch ETL init code 'transform_url' boolean flag to 'arg_type' string !6269
- docs: update ETL dev deployment for macOS !6163
- ETL: keras template minor fix !6213
- ETL: remove incorrect reference !6268
- ETL: add 'arg-type=FQN' (new) !6271
Datasets (resize, resort, and shuffle)
- [API change] dsort: extend input format !6181
- dsort input format: iterate list, iterate range !6186 !6187
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- add test scripts; memsys: init gmm only once !6192
- refactoring and renaming !6193
- move/consolidate error types; continued refactoring !6202
- Go API change; add dsort/api.go; CLI: print job spec !6203
- [API change]: dsort spec; CLI: print job spec !6204
- CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
- dsort: continued refactoring (major update) !6208, !6209, !6210
- free sgl on error; feature: any extension !6212
- [API change] revise request spec (major upd) !6217
- create destination on the fly !6218
- record content path to retain full shard name !6219
- output shard size estimation (rewrite) !6223
- add is-compressed; refactor dsort-mem !6227
- compressable shards (major) !6231
- output ext; rcb buffer; fixes !6232
- duplicated records (full coverage & stress); fixes !6233
- fix tests; add stress !6234
- rename subpackage, fix comments, refactor !6237
- remove dsort-context, rewrite initialization !6238
- static/stateless shard readers/writers; refactor and simplify !6239
- two goroutines per each shard-distributing request !6241
- [API change]: dsort: is now 'xaction' as well !6253
- dsort: support generic abort-xaction API !6264
- no need to block when sending shard records !6286
CLI
- archive and download logs (feature) !6180
- clarify "copying" vs "transforming" and "cached" vs "present" !6183
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- dsort: extend inline help, pretty-print job spec; update docs !6206
- dsort: Go API change; add dsort/api.go; CLI: print job spec !6203
- 'archive get' is now a shortcut (an alias) !6222
Build, test, and tools
- add test scripts; memsys: init gmm only once !6192
- tests and tools: cleanup around stop-maintenance, wait-rebalance" !6194
- deployment: update local deployment script to allow target-only deployment with defined primary host !6195
- deployment: optionally, skip deploying primary proxy !6197
- start using scripted integration tests; CLI: 'dsort src dst spec' !6198
- tools/generate shards: optimize buffer allocation !6224
- deploy/dev: Add ansible deployment scripts for deploying locally on multiple nodes !6199
- aistorage/CI docker image (lzma libraries) !6220
- tests: init with cleanup and without !6226
- CI: Retry stuck Python ETL tests in GitLab CI pipeline !6270
- remove
aisfs
(FUSE) !6273 - dev tools: readers; handle read from corrupted arch or non-arch !6250
Documentation
- update getting started !6161
- updated python sdk readme !6162
- update ETL dev deployment for macOS !6163
- update documentation with recent ETL changes !6173
- CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
3.18
Core
- add
htext
to track restarted state; target run and misc !5966 - cluster rebalance (scenarios) !5969, !5971, !5973, !5974, !5975, !5977, !5980, !5983, !5986, !5987, !5989, !5991, !5992, !5993, !5995, !6002
- add 'cluster-ready' helper; use it to reinforce !5976
- cleanup better when decommissioning; previous BMD at startup !5979
- fs: reliable remove-all !5981, !5982, !5984
- yet another buf pool !5985
- do not modify cluster map when starting up; always skip logging idle disks !5988
- rebalance (scenarios, major update) !5992
- [API change]: core: rebalance (scenarios) !5993
- rebalance (major update); when receiving new cluster map !5995
- up modules; handle housekeeper registration race !5994
- 'not present in the loaded cluster map' and similar startup validation !5996
- shutdown or decommission a node that's already in maintenance !5998
- transport: never establish a streaming connection to the peer that's in maintenance (or will be) !5999
metasync
just-in-time; assorted refactoring (minor) !6001- maintenance mode: pre & post vs keepalive & metasync; CLI: more colored cues !6004
- shutdown is also 'maintenance'; docs: adding-removing intro !6005
- add
meta
package !6006, !6007 - ETL: add arg-type parameter when initializing with code !6008
- archive v2: support empty template (tar entire bucket); atime !6013
- keep poi.atime in nanoseconds !6015
- archive v2: append to arch; refactoring !6017
- archive v2: up modules !6018
- archive v2: part four (major) !6019
- archive v2: detect an empty tar when appending, and handle !6020
- archive v2: part six !6022
- archive v2: mime detection !6024
- archive v2: extend 'append-to-arch' to support tar.gz !6025, !6027
- archive v2: tar and tgz append; fixes !6028
- log filenames; overlapping run vs node-restarted !6029
- archive v2: multi-object append-to-arch !6030
- archive v2: multi-object append-to-arch !6033
- archive v2: multi-object append-to-arch !6034
- cleanup disk utils (minor) !6035
- ios startup: run the command only once !6036
- hide AuthN secret !6038
- archive v2: append to zip !6041
- archive v2: append to msgpack !6043
- add
cmn/archive
package !6044 - archive v2: write and copy via new 'cmn/archive' !6045
- archive v2: append via new 'cmn/archive' !6046
- [API change] archive v2: MIME vs file extensions !6047
- ios: cleanup lsblk cache; CLI: refactor get-node-arg; up modules !6048
- archive v2: remove msgpack; refactor !6051
- archive v2: add '.tar.lz4' serialization (new) !6053
- archive v2: tar.lz4 cont-d !6054
- archive v2: lz4 features; checksum !6055
- s3 compat: run E2E tests with correct HTTP/HTTPS mode !6057
- [API change]: append to arch if exists !6062
- [API change] append to arch if doesn't exist; CLI cont-d !6064
- checksumming and buffering vs reader-from !6074
- core: content-length universally; revise write-json and friends !6075, !6076
- archive v2: [API change] put (files, dirs) with an option to append !6081, !6082
- archive v2: quiesce faster, refine continue-on-error logic !6083
- core: double-check target-in-maintenance, quiesce faster !6084
- archive v2: finalize cmn/archive package !6085
- archive v2: finalize cmn/archive package !6086
- log verbosity: core and modules !6087
- http client: disable compression; core: undefer & micro-optimize !6066
- append to (non-existing) arch: an option to create !6068
- mem-pool alloc/free symmetry: copy/transform & archive !6069
- copy/transform, multi-archive: refactor Rx logic and error handling !6071
- log verbosity: core and modules; remote cluster !6089
- log verbosity: core and modules; remote cluster !6091
- ec: minor refactoring !6092
- archive v2: WD basename; get with extraction; Range !6093
- archive v2: WD basename; get with extraction; Range !6094
- archive v2: tools/archive utils !6095
- archive v2: tools/archive utils !6096
- compile-out asserts; super-verbose logging; log module 'mirror' (ref) !6097
- log verbosity at runtime; log modules; remove glog; unify (major update) !6099
- fields iterator; size converter; log rotation (fixes) !6101
- [API change] get-bucket-info to count remote objects !6102
- [API change] get-bucket-info to count remote objects !6103
- [API change] get-bucket-info (part three); docs and CLI !6106
- list-objects vs buckets: revise and refactor, add validation, clarify !6107
- list-objects: introduce optional args (ref, cleanup) !6108
- list-objects: mem-pool msgpack buffers !6109
- kvdb: remove redundant err-not-found; amend dsort, downloader, authn !6110
- x-lso must idle more time !6111
- log modules (part three) !6112
- add
nlog
(new logger) !6113, !6122, !6124 - do not log perf counters when there's no change; sort the names !6119
- fix disk usage call for clusters on mac OS !6121
- log etl events: spec parsed, pod ready, hpull/hpush !6125
- cleanup
fs.PathError
; add object name validation !6127 - extend dsort to support
.tar.lz4
!6128 - aisloader: cleanup output when running with json option !6129
- xactions to return extended error info !6130
- xactions: add and return errors (major upd) !6131, !6132
- xactions: error handling cont-d; bucket-scope multi-object op-s !6133
- jobs error handling and reporting; tests !6134, !6135, !6136
- use Go 1.20
join-err
; reduce default pre-election interval !6141 - http request multiplexer !6142
- dsort: use common cmn/archive pkg (major upd) !6143
- archive v2: tar formats (USTAR, etc.) !6153
- archive v2: extend list-objects and GET to operate inside archives !6155
- archive v2: extend list-objects and GET to operate inside archives !6157
- control plane: consistently propagate cluster map !6158
- old EC metadata shouldn't terminate cluster-wide rebalance !6159
- core: when primary goes down it notifies !6160
CLI
- performance tabs: always show 'cluster idle' if idle !5997
- stop-maintenance w/ no rebalance; stats idle-ness (minor) !6000
- add
transform-url
flag (used when initializing ETL with code) !6010 - misc. improvements !6011
mountpath
completions; disable/detach; minor ref !6056- de-spaghettify put handler !6059
- archive multi-object (cont-d) !6060
- archive v2: CLI put, append, alias, docs !6072
- move 'gen-shards' and extend it to support all formats !6073
- archive v2: CLI: APPEND-to-arch is now on-par with multi-PUT !6077
- archive v2: CLI:
dry-run
option is back !6078 - archive v2: CLI: destination naming; dry-run; tips and examples !6079
- list-objects: extended help, template with no ranges !6104
- error and warning verbosity (major update) !6105
- log modules (part two); CLI archive: pre-parse and add tip, advice !6100
- archive help !6126
- archive v2: CLI and docs !6149
- archive v2: multiple CLI updates and improvements; is-archive bit !6150
- remove k8s apimachinery pkg (minor) !6151
Python
- sdk/python: Add option to pre-import modules when initializing ETL with code !5963
- sdk/python: Refactor ETL to provide name once on object creation !5968
- sdk/python: Release version 1.2.0 !5970
- sdk/python: Fix python package links !5978
- sdk/python: improve SDK testing, fix bucket eviction keep_md parameter !5990
- sdk/python: Release version 1.2.2 !6012
- sdk/python: Add dSort support in SDK !6032
- sdk/python: Add wait to dsort abort test !6037
- sdk/python: Fix dsort abort test on faster systems !6039
- sdk/python: Add get_url option to bucket, object group, and objects !6042
- sdk/python: Add interface for AIS sources that can be accessed via list of URLs !6052
- PyTorch: Add new PyTorch DataPipe to iterate over URLs from various AIS sources !6058
- update lint-tests to work with Python 3.11 (pytorch unsupported) !6061
- sdk/python: Include user agent in SDK requests !6067
- sdk/python Use msgpack content type when listing objects in bucket !6070
- sdk/python: Add msgspec to pyproject dependencies !6088
- sdk/python: Release SDK version 1.3.0 !6090
- sdk/python: bucket summary + bucket info !6098
- sdk/python: bck info integration test fix !6115
- sdk/python: bck summary integration test fix !6117
- sdk/python: bck summ + bck info fixes (revert and cleanup) !6120
- sdk/python: pyaisloader !6123
- pyaisloader: bucket utils (minor fixes) !6139
- sdk/python: bucket summary and info (minor fixes) !6140
pyaisloader
: total size/count fix !6146- sdk/python: Release version 1.4.0 !6152
Docs
- add lifecycle.md !6009
- WebDataset Blog Post part 1: Storing WebDataset in AIS !6014
- WebDataset Blog Post part 2: AIS ETL on WebDataset shards !6016
- update docs/tools and cmd/README !6050
- WebDataset blog post pt 3 -- PyTorch Datatpipe with ETL !6065
- archive v2: CLI put, append, alias, docs !6072
- archive v2: CLI: destination naming; dry-run; tips and examples !6079
- archive v2: CLI and docs !6149
Build, CI
- add local build options to k8s scripts and fix local registry !6021
- build: upgrade all minors !6040
- add
pyaisloader
stage !6148 - k8s: fix deployment scripts for compatibility with the latest
aisnode
image !6154
3.17
Table of Contents
- CLI v1.2
- Python SDK v1.1.2
- S3 compatibility and Botocore
- API changes
- Tests and Documentation
- Core: bug fixes and improvements
- Build and Continuous Integration
- Extensions: Downloader, dSort, ETL
See also:
CLI
- show all jobs !5645
- start/stop job/xaction !5660
- refresh rate and countdown; long-running 'show job' and friends !5651
- 'show log node-name' to mimic 'tail -f' !5652, !5654
- add custom duration flag and logic !5655
- 'ais config (cluster|node|cli)', 'ais config reset', and friends !5656
- bucket completions !5657
- set-config to show all updates; tweak iter-fields reflection !5658
- 'show job' to aggregate all categories and support all selections !5661
- transition to using job display names (major) !5663
- start, stop, show jobs and xactions (cont-d) !5665
- amend and restructure jobs !5666
- running xactions (completions) !5672
- tweak config json printout; get-config from memory !5673
- update backend config !5674
- update backend config (part two) !5677
- add footnote, marshal message only once !5678
- remove
xaction
term and subcommand (everything isjob
now) !5692 - suggest (targets, proxies, nodes) !5694, !5696
- revise bash completions script !5697
- remove 'xaction' (term and subcommand) !5698, !5699, !5700
- 'show cluster': separate cluster nodes from all other (tab-tab) completions !5701
- consolidate and refactor cluster map access !5704
- tweak
ais create bucket --props
&ais bucket props set
!5706 - extend 'job start' to support (resilver, copy-bucket, rename-bucket) !5715
- tweak listed props !5719
- remove (cleanup) download and dsort jobs !5721
- extend 'ais stop' to support --all|--regex !5722
- 'show job' verbose option; unify usage args; ref PUT/APPEND !5723
- rewrite command-not-found logic; add similar commands !5724
show jobs
(major) !5726, !5727- bash autocomplete ordering improvements !5728
- improvements (usability) !5729
- add
bucket cp
alias !5730 - flag printable name; split 'show job' in parts; usability !5736
- further unify stopping, waiting-for, and showing jobs !5744
- revise & amend 'show rebalance' - all permutations !5761
- universal start-end formatting; template refactoring !5762
- jobs grouping by name and, within name, by UUID !5764
- complete
etl-name
transition !5767 - ETL tools, UUID (part one) !5745, !5746, !5749, !5753, !5754, !5763
- fix download/dsort progress !5769
- new table to show target statistics !5788
- 'ais show performance' (new) !5791, !5793, !5800, !5802, !5803, !5809, !5810, !5811, !5812, !5816
- IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5820
- reduce code, simplify, cleanup !5821
- IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5823
- disk stats: add average read/write sizes !5824
- amend existing mountpath tab and add a new one !5833, !5834
- expect node unreachable when iterating '--refresh' !5837
- assorted usability; add 'no-color' config !5839
- 'ais show performance': average (GET, PUT, etc.) sizes on the fly !5840
- support new API to reset stats !5841, !5843
- 'ais show performance': refactor throughput, add latency !5844
- 'ais show performance': finalize latency tab !5847
- 'ais show performance' cont-d !5848
- 'ais show performance': finalize top-level tab !5850
- 'ais show performance': add cluster-level throughput, beautify !5852
- 'ais show performance': alias 'stats' and remove older code !5853
- 'ais show performance': disk table v2 !5855
- 'ais show performance': finalize disk table !5857
- 'ais show performance': new mountpaths/disks/capacity table !5858, !5859
- 'ais show performance': finalize capacity table !5861
- refactor and cleanup multi-object put !5862
- multi-object PUT: source dir, list/range; matching pattern !5865
- fix concatenation logic, refactor progress bar !5867
- copy bucket: support progress bars (copied objects and size) !5870
- consistent timeout management !5871
- copy/transform a list or range of objects: add progress bar !5873
- copy/transform with progress bar: style, reuse !5874
- multi-object PUT !5876
- rogress bar: all multi-object operations; universal 'wait-for' !5879
- PUT multi-object - all flavors !5880
- get multiple objects in one shot (""multi-object GET"") !5884
- GET destination & assorted fixes !5882
- copy-bucket: prepend prefix, command helps, examples !5889, !5891
- more inline help !5892
- assorted improvements (minor) !5900
- fix downloading with progress bar enabled !5903
- how-to text: how to reconfigure remote ais cluster !5932
- add CLI compatibility warning (new CLI vs old cluster and vice versa) !5952
- cluster membership-changing operations (shutdown, decommission, et al): improve usability !5956
Python: SDK library and ETL
- Add unit tests for cluster class !5675
- Add unit test for api client class !5676
- Fix python ETL test workflow !5680
- Add unit tests for Xaction class !5681
- Add unit test for object class, fix object put data from a filepath !5682
- Add unit tests for ETL class !5683
- Add unit tests for sdk utils !5685
- Restructure python subdirectories and containment !5686
- Run python unit tests as part of default python test make option, include python ETL tests in all python test runs !5688
- Only run python ETL tests when python labels are added !5702
- update test utils to support running Python tests on Windows !5716
- Add multi-object functionality !5720
- Add tests and validation to object ranges, support leading zeros !5731
- Add unit test for object group class !5732
- Update documentation for python multi-object ops !5734
- Use aws bucket for python sdk CI tests to test caching functions !5739
- Add string template support to object groups !5741
- Refactor all references from xaction to job !5742
- Update ETL runtime defaults and add new python 3.11 option !5743
- Fix remote bucket tests to avoid collisions !5748
- Increment cloudpickle version and update github action to use 3.11 for tests !5756
- Fix python test dependencies !5780
- Refactor to simplify typing and tests !5789
- Bump python SDK to 1.0.5 !5790
- Update python sdk version !5792
- Update pytorch integration README with compatibility issue !5794
- Patch sdk to support torchdata integration !5795
- Bump aistore package version !5798
- Fix pylint and formatting !5799
- Add ObjAttr type for returning additional object metadata !5805
- Address lint warnings, general improvements !5808
- Add writer option to object get !5813
- Split README for different python projects !5814
- Add PROMOTE functionality to objects !5818
- Standardize pylint version, fix lint errors !5822
- Improve object put behavior and add directory put options !5826
- Use Pathlib over os.path !5827
- Refactor multi-file put to bucket class !5830
- Improve job interface and update promote options !5832
- Update python package build tools, increase version to 1.0.9 !5835
- Improve example documentation !5842
- Improve input validation !5846
- Set up proper logging, update constants !5849
- Add job wait for idle status and fix job bucket filter !5863
- Add multi-object copy !5866
- Fix remote test fixture !5877
- Add multi-object ETL !5878
- Release 1.0.10 !5881
- Expand multi-object examples !5883
- Improve usability !5888
- Follow-up for copy/transform prepend !5890
- Improve object interface !5893
- Add prefix_filter option to bucket copy !5899
- Add flags and target options to bucket list_objects methods !5901
- Release 1.1.0 !5910
- Release 1.1.1 !5911
- Improve python sdk examples !5913
- Add cluster list_jobs, require id for individual job status query !5916
- Add support for multi-object archive !5920
- Update bucket params to use Bucket object, support namespaces !5925
- Add prefix filter to bucket transform !5949
- Release 1.1.2 !5950
- PyTorch: add support for
etl-name
inAISDataset
andAISFileLoaderIterDataPipe
!5957
S3 compatibility and Botocore
- Add botocore monkey patch alongside python SDK !5684
- Move botocore and pytorch packages to python top-level, separate from sdk !5691
- Add s3 compatibility testing with boto3 !5703
- list-objects vs HEAD !5705
- compute multipart md5 and set etag !5708
- Improve s3 compat test documentation and update validated tests !5747
- Pass S3 delete a list of objects !5854
- fix infinite loop when listing objects; add bucket name to list object response !5864
- return bucket creation date in UTC !5869
- new flag
NoRecursion
to support S3 delimiter feature !5930
API changes and new APIs
- add API to query multiple xactions via any IC proxy !5670, !5671
- yet another API to query xactions (new) !5687
- list-objects API: tweak listed props !5718
- [API change] flatten
xaction-snap
control structure (major upd) !5740 - [API change] ETL: tools, UUID (part nine) !5754
- [API change] ETL: new query parameter to specify transform name !5755, !5765
- [API change] init/start ETL to return xaction ID !5768
- [API change] GET(object) !5770, !5772, !5773, !5775
- [API change] remove xaction 'query-msg' (deprecated) !5783, !5784, !5785
- get-object API: amend comments !5769
- refactor
api
package (major) !5776 - [new API] query metric names and kinds ('counter', 'latency', 'throughput', et al.) !5796
- [API change] get node status !5811
- [API change] core stats: consistency between stats-querying APIs !5817
- [API change] PUT(object) !5828
- [API change] amend capacity-disks-filesystems control !5831
- [API change] add API to reset stats !...
3.16.rc2
v1.3.16 v3.16.rc2
3.12
Table of Contents
- Remote AIS clusters
- List objects
- List and summarize buckets
- S3 API
- CLI
- Documentation
- Tests
- Python SDK & ETL
- Core
- Refactoring & fixes
- Tools
- Build & CI
See also:
Remote AIS clusters
- list remote ais buckets; fix remote namespace containment !5567, !5621
- aliasing and bucket namespaces (major revision) !5622, !5623
- ais gateways to cache remote ais clusters (information) !5630
- extend remote ais errors - add cluster UUID !5631
- remote ais aliasing !5632
- [API change]: remote ais clusters (major) !5633, !5634
- [API change] add
HealthUptime
; use it to show remote clusters !5636 - remove support for multi-aliasing !5637
- alias vs bucket metadata !5638
List objects
- CLI to show
CACHED
(column) only when it's informative (and not when user asksais ls ... --cached
) !5554, !5555 - rename
list-objects
structures; regenerate message pack !5542, !5543 - amend and re-enforce single target rule for remote buckets !5556
- introduce location property, phase-out target URL (property) !5557, !5559
- add custom metadata, regenerate message pack; refactor !5564, !5565
list-objects
version 2 (major): !5583, !5584, !5586, !5589, !5590, !5591, !5601, !5602, !5605, !5607- [backend API change] mem-pool 'list-objects' pages !5604
list-objects
flow with a single target listing remote props (name, size, version) !5608- list remote pages v2: flow controls; ais gateways will now select single target to execute remote call !5609, !5610
list-objects
to include (in listing result) custom object properties !5614- consistency in custom props across:
get
,put
,head
object, andlist-objects
!5615, !5616
List & summarize buckets
- easy URL to support listing buckets via
/gs
,/az
, and/ais
endpoints !5486 list-buckets
andget-bucket-info
API; CLI summary table !5521, !5522list-buckets
version 2 (major update) !5524- unify 'list-buckets' and 'bucket-summary' !5530, !5531
- shorter and faster
list-buckets
version: w/ presence but wo/ summary !5535, !5536- [API change] addpresence
filter; eliminate bucket-info control structure (is redundant); refactor !5540 list-buckets
andbucket-summary
cont-d !5541- bucket sizing: always compute two sizes: on-disk and sum objs !5544
bucket-summary
on-disk (apparent) vs sum-all-objects, remote vs cached !5545bucket-summary
andlist-buckets
cont-d unification !5546, !5547list-buckets
by target: remote ais; hdfs and http; cloud; other !5567- rename/move bucket summary; align control structures !5568
- list remote ais buckets; fix remote namespace containment !5621
- [API change]
list-buckets
correctness for named bucket queries !5626
S3 API
- multipart upload !5458, !5460, !5467, !5468
- extend s3 API to support remote buckets (major) !5462
- persistent multipart state (major) !5466
- multipart GET !5472
- multipart list active uploads and upload parts (major) !5474
- return xml-formatted errors (compliance) !5480
- multipart: refine and simplify !5482
- revise
list-buckets
to include all providers !5485 - always set header
Server
when redirecting !5495 - amend s3 compatibility readme; add multipart example !5496
- multipart: sha256 fix; md5 not checked but computed;
aws s3api
example !5500 - xml-format errors (parts two and three) !5501
- revise
HEAD object
implementation; multipartETag
; miscellaneous fixes !5504 - compatibility: configurable root '/' access to AIS cluster !5572, !5573, !5574
- compatibility: assorted error codes !5620
CLI
- 'ais etl' to replace extension in transformed objects !5418
- standardize error and warning coloring !5493
- don't log and don't show backend secret !5506
- improve unknown-command messaging, provide hints !5512
- add
ais object ls
command; addpresent
flags !5514 - support updated HEAD(bucket) API; 'ais ls' and permutations to show obj props !5517
- static linkage with
cgo
disabled !5526 - rename aws, et al. backend provider constants !5528, !5529
- option to display sizes either in bytes or KiB, MiB etc. units !5532
- rename sub-packages; show-unmatched & hide-header !5534
- bucket
display-name
helper (and consistent usage) !5539 - flip bucket listing default to 'present' (was 'all accessible') !5550
- option
--all
to list both buckets and objects (in remote buckets) !5551 - multi-feature completions; action-warn and action-done !5575
- show config subsections in JSON; multi-feature completions !5576
- amend config JSON printout !5577
- extend and amplify: (config | section | json) !5578
- is-time-not-set, is-object-not-cached (for consistency) !5611
atime
&ctime
!5612- throughput must be averaged over an interval of time since the previous request !5643
Documentation
- amend main readme; add Reuse section !5375, !5376, !5378
- blog: Python SDK: Getting Started !5373
- add
curl
example to set backend bucket property !5380 - python/sdk: ETL !5397
- amend main readme: add PyTorch integration section !5406
- remote backend; native integration !5407
- revise
ulimit
, reference checked-in version oflimits.conf
!5429 - update CONTRIBUTING.md !5459
- blog: AIStore: Data Analysis w/ DataFrames demo !5469, !5473
- update s3cmd readme - add multipart example !5483
- restructure and extend getting-started; revise and amend
Local Playground
option !5484 - easy URL: add readme with extended examples and comments !5487
- REST API querying information !5641
- REST API listing buckets (remote namespace, presence and existence) !5644
- update ETL main readme, CLI readme, and examples !5471
- amend s3 compatibility readme, add multipart example !5496
- main readme: Installing from release binaries (add section) !5640
Tests
- python ETL !5410, !5461, !5476
- python
IO communicator
!5415 - tests: skipping short (minor) !5416
- pyhton/sdk: ETL stress test !5420
- add s3 multipart test !5482
- add ETL test for streaming with python runtime !5513
- re-enable and amend object-properties tests across all providers !5515
- amend
list-buckets
: add cases; up CLI !5569, !5570 - reinforce multi-object tests, fix
bucket-exists
helper !5606 - amend and fix multi-object range/list copy test !5619
- amend
list-buckets
test that checks expected result !5631
Python SDK & ETL
- ETL API part 1 !5370
- sdk version !5374
- code (re)formatting and misc !5381
- reorg Makefile dependencies !5390
- ETL APIs part 2 !5394
- bucket ETL transform API !5405
- bump sdk version to 1.0.3 !5408
- fix
IO communicator
fixes !5413 - fix transform bucket !5421, !5428
- add examples !5441
- add streaming runtime w/ refactoring and cleanup !5444
- streaming
init-code
with chunk-size param (major) !5446, !5447 - with temp debug !5448
init_code
changes for before and after functions !5449- remove debug w/ minor ref !5450
- functions
filter
(new),transform
, andbefore
/after
!5451 - runtime v2 fixes !5455
- bootstrapper; by reference !5457
- ETL API fixes !5464
- update readme (etl init_code) !5475
- extend
hpush
yaml with bitwise flags !5479 - remove unused ETL functions; add test for streaming with python runtime !5513
- update Jupyter notebook to reflect sdk changes !5548
Core
- global rebalance: initiating new vs receiving delayed ACKs !5426
- etl md: remove sgl, read-all !5470
- error handling: introduce type codes (major upgrade) !5477, !5481
- easy URL: support listing buckets via
/gs
,/az
, and/ais
endpoints !5486 - easy URL: add readme with extended examples and comments !5487
- user agent, internet browser: pretty-print JSON response !5492
- add
User-Agent
to http request headers !5494 - backward compatibility; remote backends !5498
- consolidate http headers; add type-cast fail !5503
- introduce error types unsupported and not-implemented-yet !5508
- refactor GET(object) !5507
- add
lom.rename-file
w/cleanups !5509 - simplify
lom.persist
&recache/refresh
!5510 - update
atime
on PUT !5511 - HEAD(bucket): extend the API in re: cluster metadata !5516
- [API change] add
api.GetBucketInfo
!5519 - enumerate and unify bucket and object 'presence' vs 'existence' !5520
fs
: add size-bucket and refactor x-bucket-summary !5523- backend providers as simple KVs (minor) !5525
- on the fly and (new) upon request remote bucket creation/addition !5538
- object props: remove get-node, add location and helpers !5561, !5562, !5563
- creating/adding buckets on the fly: further clarifications !5566
- debug pkg: assert-msg is redundant; minor ref !5579
cos-assert
; align (minor ref) !5587- periodic stats log: tweak idleness logic !5588
x-archive
to always close data mover !5594- key-value d...
3.11
Highlights
- Python SDK v1.0.1 with PyTorch integrations
- Authentication and Access Control
- GitHub Action to deploy and test local playground
- Bug fixes, documentation updates, performance improvements
Authentication and Access Control
- CI: fix AuthN to run !5264
- replace form3tech-oss/jwt and upgrade to jwt/v4 !5266
- prep for production !5267
- prep for production (major update) !5268
- prep for production (part three) !5269
- prep for production (part four) !5272
- prep for production (aisloader) !5275
- prep for production (clean_deploy) !5277
- extend error info, add usage examples, tweak CLI help !5279
- revamp
ais auth
CLI !5281 - rename APIs for consistency; add CLI to update existing role !5283
- E2E test: added basic AuthN test !5284
- make cluster optional for role add/set; randomize E2E tests; unify auth-enabled checks !5285
- rename AuthN entities for consistency !5288
- add api package, isolate server internals (major) !5289
- continued refactoring, assorted fixes !5291
- add 'tok' package to consolidate all jwt calls; continued refactoring !5292
- status 'forbidden' or 'unauthorized' to accompany all errors !5310
- allow 3rd party client lookup ht://bucket !5312
- separate enabling HTTPS for AuthN and AIS cluster when deploying !5313
- oad-token: if user-given location is empty then !5314
- secret handshake when adding/updating AIS clusters !5316
- revisit AIS calling and retrying; unify errors !5317
- login by non-superusers: require cluster (to be explicitly specified) !5326
- assorted fixes, improvements !5333
- introduce 'show-cluster' permission; reinforce admin access !5350
Python SDK
- add
copy_bucket
method !5225 - add method xact_status !5226
- move test to cluster ops file !5232, !5233
- add xaction start !5237
- restructuring (adding pytorch) !5238
- PyTorch plugin (dataset/aisio) implementation for aistore !5242
- fixed bug - list empty buckets !5251
- changed error handling !5258
- add iterator for bucket objects !5259
- improved pytorch plugin !5274
- release 0.9.0 fixes !5276
- cluster health api !5280
- health check for non existing urls !5282
- syncing aisio.py !5286
- rename_bucket() port !5304
- test suite changes !5305
- makefile dependency fix !5307
- cluster health api !5278
- added aisio example !5322
- restructure api (bucket) !5323
- fixing aisio example !5325
- aisio example fixes !5328
- restructure cluster !5335
- obj streamer fixes !5337
- blog post for ais file loader and lister !5338
- using session for requests !5341
- restructure object !5342
- object method name changes !5343, !5346
- restructure fixes !5348
- bucket api fixes !5351
- object get fix !5352
- restructure xaction operations !5353
- fixes and release prep !5355
- minor readme edits !5356
- readme changes wrt pytorch !5357
- api docs and make file fixes !5359
- removing aistore.pytorch imports from main init !5360
- makefile fix (pipx path) !5367
GitHub Action
- add github action for building ais !5265
- add github workflows for lint, python-tests and build !5270
- add
test-short
github workflow (Linux) !5271 - github/workflow: changed go lint version !5366
CLI
- confirm assorted operations: decommission, rm bucket,
rm -rf
!5224 - revamp
ais show cluster
; add K8s POD names !5243, !5244 - bucket extra-props-to-update; filter by provider !5256
- allow setting permission on per-bucket basis for a role !5309
- revise 'show config', add nested completions !5339
- revise 'set config', add local vs inherited !5344
- add --cluster filter flag to 'auth show role' command !5349
Bucket Summary
- revise bucket summary (major) !5230
- bucket summary: micro-optimize and simplify !5262
- bucket summary: multi-bucket; add min/max/avg counters; fixes !5299
- bucket summary: apparent vs on-disk sizing, formatting clarity !5300, !5302
Common environment and system filenames
- move and refactor common environment; AuthN version and build time !5318
- filename constants; environment variables !5324
- consolidate all configs under
$HOME/.config/ais
!5327
Misc. Features
- warn when decommissioning cluster, support '--rm-user-data' option !5221
- ability to decommission node and join it back with a new identity !5222
- anonymous public-access buckets: list-objects anonymously (w/ API change) !5330
- s3: the capability to set custom s3 endpoint !5257, !5260
Docs
- tweak ETL markdown !5227
- updated getting_started and etl markdown !5228
- changes to standalone deployment README !5253
- minor indentation fix to standalone readme !5254
- standalone ec2 deployment instructions and benchmarks !5261
- python api reference !5294
- python_api.md fixes !5297
- bucket.md to reflect 'summary' changes !5308
- update 'ais ls', add anonymous and more !5336
- main readme !5358
- s3 compatibility, 's3cmd' configuration and getting-started !5361
- main readme, index.md, python.md !5362
- amend and extend cli/config.md; fix
config node reset
!5363 - add 's3cmd.md' !5364
Tests
- random page, remote bucket !5234
- remote bucket, object props !5235
- extend cleanup to support (prior to test, remote bucket) !5240
- ensure more !5245
- fix
iter-fields
unit for recently added config extras !5290 - reverse proxy test against GCS !5329
Build
- upgrade gopkg !5223
- fix statsd github URL !5229
- new (and supported) JWT packages !5266
Performance micro-optimizations, bug fixes, refactoring
- aisloader: rewrite SGL freeing piece !5236
- refactor low-level is-bucket, bucket-init, and friends !5239, !5241
- BMD: speed-up bucket init; add global namespace uname (const) to avoid allocations !5246
- BMD atomic update vs backend props !5247
- reuse buffer when iterating range template !5248
- devtools: refactor and simplify !5249
- bucket extra-props-to-update: include readonly; CLI: filter by provider !5256
- yet another case to skip err logging block devices !5273, !5295
- cluster config: allow critical updates !5298
- capacity: periodic log and refresh; version and build time; fixes !5315
- local playground: loopback device sizing; fixes !5321
- refactor and move docker pkg (formerly containers) !5331
list-objects
flags: unify, rename, document, and reference !5332- config: restart-required selection; feature flags !5334
- blog image fix !5347
- assorted fixes and refactoring !5354
- CI/CD: skip-ci & python-tests-only conflict fix !5365
3.10
Highlights
- Global (cluster) config meta-version 2
- Bucket metadata (BMD) meta-version 2
- Python SDK
- Build: Go 1.18; upgrade all packages
- MessagePack archiving
Global Config v2 & Bucket Metadata v2
- bump up cluster config's meta-version; handle backward compatibility !5122, !5123, !5124
- redefine and extend feature flags !5132
- capacity and space watermarks (cleanup, low/high, OOS) !5143, !5144
- BMD meta-v2 with backward-compatible logic to load v1 !5145, !5150
- BMD meta-v2: simplify mirroring configuration !5151
- add transport section !5153
- support loading previous meta-version !5154
- readonly knobs; backward compatibility !5155
- introduce
cos.size
type and "kb", "mb", etc. suffixes !5156 cos.size
vs backward compatibility !5157- add
memsys
section and tunables !5158 - log flush time interval !5159
join-cluster-at-startup
timeout !5161- add
log-stats
time, add validators !5171 - continued refactoring !5172
- add
tcb
(transform-and-copy-bucket) section; stream bundle multiplier; compression !5173 - revise docs/configuration.md !5180
msgpack
archiving; list, create, and read msgpack
formatted shards
msgp
(generic) shard !5100, !5101- fix preambles in the
msgp
generated files !5086 - alternative
msgp
; sgl asio.ByteScanner
; Python unpack to test !5103 - assorted text fixes; more
msgpack
!5104 - archiving formats: add
msgpack
(in addition to tar, tgz, zip) !5107, !5108 - add integration tests !5100, !5101
Python SDK
- basic bucket and object operations !5109
- introduce pydantic; refactor
list-buckets
!5120 - CI: add Python SDK test !5112
- improve
list-objects
API !5163 - refactor API calls for usability: !5128
- move direct requests call out of list objects, create and delete bucket, cluster map !5130
put-object
&head-bucket
API !5141- improve list object API !5163
- fix return type in
_request_raw
method !5164 - add
head-object
method !5166 - add
setUp
andtearDown
in tests !5167 - improve creating temporary file !5168
- add
evict-bucket
anddelete-object
; add props argument forlist-objects
!5175 get-object
returns a stream instead of the entire content !5184- update README, docs !5178, !5190
- fix naming; add
read-all
method for streams !5196 list_object
to support pagination !5198- improve object streaming; rename
msg.py
->types.py
!5203 - Python SDK tutorial !5218
CLI
show cluster config
feature flags in human-readable format !5133- fix setting backend bucket props !5152
- add
ais object rm --all
to remove all in one shot !5195 - verbose and non-verbose multi-object operations !5197
Tests
- re-enable smoke-test for Cloud buckets !5098
msgp
(generic) shard !5100, !5101- add missing wait for put-copies; BMD string !5204
- failback;
devtools
: wait-cluster; transport !5209 - dummy keepalive tracker !5211
Docs
- website: tech blog on promoting local and shared files !5082
- explain
msgp
usage; addmsgp-update
to Makefile !5085 - update
msgp
generator usage !5094 msgpack
!5110- revise and amend configuration.md !5180
- performance tuning !5115, !5116
- development, first-time usage, production deployments (summary) !5127
- getting started (recommendations) !5129
- local playground, tracking access time (more edits) !5139
- fix references, add a reference !5140
- erasure coding (space utilization ratio, IO performance) !5142
Build
- upgrade
fasthttp
package !5092 - upgrade direct dependencies !5096, !5097
- transition to Go 1.18 !5106
- upgrade all dependencies except 1.18 !5113
- upgrade all dependencies except 1.18; v3.10-rc2 !5181
- upgrade all dependencies including 1.18; v3.10-rc3 !5186
- upgrade all minors !5200
Bug fixes and performance improvements; continuous refactoring
- skip bucket name, etc. validation when initializing
lom
!5083 - follow-up !5084
- unsafe string: inline and use it consistently !5087
- write-msg-pack (ref) !5088
- control-plane broadcast & unicast: result value factories and decoders !5089, !5090, !5091, !5093, !5095
- global rebalance: dynamic reg/unreg streams; housekeeper name suffix !5099
- fix sending http request body on redirect: recompute
content-length
!5102 - not restoring
atime
when !5105 - multi-object operations !5111
- local from-scratch deployment: add user-friendly tips !5114
- follow-up !5117, !5118
write-policy
: both metadata and data; bump up config's meta-version; backward compatibility (major) !5121- deploy: fix Dockerfiles and related scripts !5126
- assorted usability tweaks !5134
- local playground: print ports AIS listens on !5135
- remove
bench/soaktest
; rewritedevtools/readme
!5136 - config: write policy for data must be 'immediate' !5137
- docker + LVM deployment: no disks fix !5146
- move
mock-bmd
=> cluster/mock (refactoring) !5147 - docker + LVM deployment: no disks !5148
- local playground: pgrep to check; quiet run-cmd !5160, !5162
- fix
deploy.sh
script for older bash versions !5165 - target startup: skip docker union mounts when enumerating drives !5169, !5170
- dedup fix HEAD(obj) system attributes !5174
get-object
lock/unlock (refactoring) !5176- URI parsing, config validations, and assorted ref !5177, !5179
- making "rlock" exception to forcefully remove corrupted !5182
- dont-lookup-remote-bucket = true: proceed silently, log-wise !5183, !5185
- remove
+build
directive (obsolete) !5188, !5189 - get random proxy (ref) !5191
- cluster map (ref); node equality !5192
- PUT object vs remote backend returning status=503 !5193
- multi-object operation, range template (ref) !5194
aisloader
works !5199- when shutting down warn with details !5201
api
:xaction
wait-IC min waiting time !5202compute-checksum
andclone-md
(ref) !5205memsys
: mem-free to include buff/cache !5206memsys
: revise sys/mem (major) !5207lif
tolom
conversion: free on error; EC error handling !5208- revise xs/rename-bucket !5210
- deferred unlock via lightweight
lif
(ref) !5212 lom
as a reader,lif
as unlocker !5213- get-from-neighbor (
gfn
): reghk
housekeepr under lock; abort to self-expire !5215 lom
copy: allocate with name !5216lom
xattr (ref) !5217
3.9
AIS v3.9 is substantial productization and performance-improving upgrade. Much of the codebase has been refactored for consistency, with numerous micro-optimization and stabilization fixes across the board.
Highlights
- promote: redefine to handle remote file shares; collaborate when promoting via entire cluster; add usability options; productize;
- xmeta: extend to also dump in a human-readable format: a) erasure-coded metadata and b) object metadata;
- memory usage and fragmentation: consistently use mem-pooling (via
sync.Pool
) for all control structures in the datapath; - optimistic concurrency when running batch
prefetch
jobs; refactor and productize; - optimize PUT datapath;
- core logic to deconflict running concurrent
xactions
(asynchronous jobs): bucket rename vs bucket copy, put a node into maintenance mode vs offline ETL, and similar; - extend and reinforce resilvering logic to withstand simultaneous disk losses/attachments - at runtime and with no downtime;
- stabilize global rebalance to successfully pass multiple hours of random node "kills" and restarts - node-left and node-joined events - in presence of stressful data traffic;
- self-healing: object metadata cache to support recovery upon
mountpath
events (e.g., drive failures); - error handling: phase out generic
fmt.Errorf
and consistently use assorted error types instead; - additional options to speedup listing of very large buckets (list-objects);
- numerous micro-optimizing improvements: fast datapath query (
DPQ
) and many more.
Promote files and directories
- refactor as a 2-phase transaction and auto-detect file share (initial) !4929
- auto-detect file share and distribute the work between target nodes !4945
- add test; add target node => IC notifications !4975
- extend test coverage; reinforce global UUID when promoting via entire cluster !4976
- rename api.Promote; add test permutations and checks !4985
- remove redundant control structures; cleanup !4987
- add API options
delete-src
(delete source) andoverwrite-dst
(overwrite destination)!4988 - fix extra-copy optimization with full refactoring !4989
- revise/optimize
HEAD(object)
implementation and utilize it when promoting withoverwrite-dst=false
(major) !4991 - extend object write transaction (OWT) to support the flow !4992
- support in (i.e., transmitted), out (i.e., received) and locally-promoted stats counters - files/objects and bytes !4993
- introduce confirmed file share; add user option not to auto-detect file share !5019
- CLI: add
overwrite-dst
anddelete-src
E2E tests !5024 - consolidate control, eliminate ambiguity !5045
- increase test coverage !5047, !5063
- add all test permutations to cover (
ais
|cloud
|remote-ais
) bucket vs. (non-redundant |EC
|n-way mirror
) !5068
ETL
- add CLI to view stored ETL code and specification !4925
- handle target-down; test !4933
- redefine and improve ETL API (!4947, !4966, !5022), including:
- manage (CRUD wise) persistently stored ETL definitions
- eliminate redundant URL path parameters
- enforce uniqueness of the user-provided ETL name
- remove (obsolete) embedded ETL-specific annotations from the
init
spec (pod template) - support stopping and restarting ETLs !5005, !5056
- update ETL docs and fix minor bugs !4984, !5022
Global Rebalance
- global rebalance status: always respond with total (cumulative) stats counters !4905
- generic
fs.Walk
for global rebalance (refactoring) !4889, !4930 - get-status & health !4934
- global rebalance status: reimplement to optimize !4936
devtools
: mergeWaitForRebalanceToStart
andWaitForAllResilver
!4937- tweak/optimize receive logic !4994
- abort via stage notifications from other target nodes (major) !5015
- transport streams vs receive errors; assorted fixes !5040
- tweak preemption logic (when rebalance triggering events arrive back to back) !5057
- assorted fixes: global rebalance vs n-way mirroring & resilvering !5071, !5072
- consistent renames and continuous refactoring !5075
Resilvering (in presence of drive failures and attachments)
- tools and stats:
wait-for-all-resilvers
, multi-snap API !4888 - resilvering vs copy management (major) !4865, !4866, !4867
- resilvering: tweak is-active time interval !4882
- support losing multiple disks (mountpaths) simultaneously !4884
- multiple overlapping add/remove disk operations: fixes !4894
- resilvering as scrubbing: recover objects to their expected (default) locations !4900
- resilvering: interval-of-inactivity multiplier !4974
- resilvering under stress in presence of lost mountpath(s) !5058
Asynchronous Jobs (aka xactions
)
- when aborting and propagating abort to the control-plane caller, make sure not to lose the original cause for the abort !4886
- fix
put-xaction
finishing logic !4887 - aborting jobs: propagate the original cause through channels and APIs !4890, !4891
- revise lookup by
only-running
and/or by UUID !4897 - move
xaction
andxreg
packages with refactoring !4898 - clarify running vs not finished
xaction
!4908 - registry: fix matching logic, remove redundant code !4911, !4912
- registry: amend housekeeping !4913
- registry: continued refactoring and cleanup !4914
- "limited coexistence" between running and about-to-run services (new) !4915, !4916, !4917
xact
package: revise and optimize abort-checking concurrency !4923- registry: continued simplification and cleanup !4940
- reinforce global UUID for all cluster-wide
xactions
!4978 - IC notifications vs transactional
xactions
: same rules for all !4982 - more stateful info: propagate xaction reference all the way into local PUT flow (major) !4995
copy-object
--xaction
--promote
: continued refactoring !5002- registry: micro-optimizations and cleanup !5028
CLI
- PUT: add an option not to load (skip loading) object metadata; amend docs; refactor and cleanup !4859
- add a command to view ETL code/spec !4925
- fix: do not add
--help
flag to the subcommands of subcommands !4926 - amend 'show config' to include CLI config (in addition to cluster and local configs); fix
cluster-unreachable
error !4983 - revise 'flag-is-set' for Boolean flags !5021, !5023, !5030
- copy bucket: add
--force
option !5042 - add
start etl
command !5056 ais show cluster
: add support forrefresh=<time-interval>
andcount
options !5076- update CLI docs !5078
- enable 'ais show storage' and friends to run continuously and refresh periodically !5079
Testing
- test fixes to align with changes in the core !4861
- add
ensure-num-mountpaths
helper, and reinforce !4892 - use
api.WaitForXaction
instead oftutils.WaitForXactionByID
!4893 - re-enable one
fs-checker
test, allow more time for mountpaths !4903 - add more checks when downloading object !4910
- extend CLI e2e promote test !4932
WaitFor
follow-up !4943- fix e2e
AuthN
messages !4952 - retry upon failure to recover a damaged erasure-coded object !4986
- amend and extend EC tests !4990
- revise and enable
bucket-rename-and-copy
test !5060
CI/CD (continuous integration)
- add CI job that runs on multiple cloud buckets !5027
- add 1.18-rc1 version to build check !5044
- add
test-short-minimal
to test a single-node cluster !5046 - make
AWS_REGION
global env variable !5050 - update
test-long
stage !5059
Bug fixes, performance improvements; continuous refactoring
LOM
load to return distinct types:syscall-error
andcorrupted-error
!4849LOM
vs n-way mirroring: fix and revise caching of the metadata !4850list-objects
: add fast mode--only-names
!4851- introduce permission to overwrite disconnected backend !4852
api
: refactor PUT API; fixdevtools
!4853- optimize PUT latency by allowing not to load object metadata !4854
aisloader
bench: do not run goroutine per eachPUT
request !4855- reinforce access time
atime
(major) !4856 - reintroduce
no-metadata
error; fix n-way stress; refactor !4857 - build: fix deprecation warning on MacOS !4858
- when copying objects differentiate between copying == mirroring and all other scenarios !4860
- simplify
LOM
from-fs
logic !4862 - general: don't use regex to validate names and UUIDs !4863, !4864
- assorted fixes !4868
- preserve
atime
acrossLOM
caches !4869 - storage cleanup: leftover copies, corrupted and missing metadata !4870
- refactor
cmn
andapi
packages !4871 list-objects
:use-cache
option !4872- consistently use HTTP status 507 throughout; assorted fixes !4875
- eliminate redundant mirroring !4876
- get-cold (aka cold-GET) follow-up !4877
- object write transaction (
OWT
) fusion !4878 - prefetch: support optimistic concurrency (major) !4879
- name locker: fit two structures into 24 bytes !4880, !4881
- disable/detach mountpath: graceful (admin request) and immediate (FSHC) !4883
- move
health
package underfs/health
!4885 - bucket summary and
obj-list
query: move, refactor, and simplify !4895, !4896 - control-plane: always free
call-results
back to pool !4899 api
: eliminate code duplication !4901- general: deprecate and remove query objects !4902
- refactor
ais
package !4904 - control-plane transactions: refactor, reduce code !4906
- initial ETL get API implementation !4907
- control-plane transactions: follow-up !4909
- introduce read-only (but still configurable) timeouts:
cplane-operation
andmax-keepalive
!4918 - intra-cluster transport streams: tweak termination logic !4919
- slab allocator: amend pooling of the SGL control structures !4920
- transport s...