Skip to content

Releases: NVIDIA/aistore

3.21

05 Nov 22:20
Compare
Choose a tag to compare

Highlights

  • cold GET: extract and micro-optimize the flow
  • sync Cloud bucket
    • leverage validate-warm-GET bucket config, and
    • extend it to support non-versioned Cloud buckets, and
    • optionally, delete (remotely deleted) objects
  • bucket sizing and counting:
    • support very large buckets that are not necessarily present in the cluster;
    • unify ais ls --summary and ais storage summary to utilize the same control message and flags
  • list, summarize, and lookup the properties of remote buckets without adding them to cluster's BMD
  • HTTPS:
    • support TLS configuration to authenticate clients
    • switch cluster from HTTP to HTTPS, and vice versa
  • optimize metadata cache
  • optimize capacity management
  • bug fixes, performance improvements

Core

  • set prime-time to amend local generation of globally unique IDs !6325
  • multi-object (archive, copy, transform) jobs: transport endpoint !6326
  • core: (maintenance, decommission, shutdown) transition w/ rebalancing !6327
  • core: (maintenance, decommission, shutdown) transition w/ rebalancing !6328
  • intra-cluster transport: make receive-side stats optional !6329
  • intra-cluster transport: reduce receive side contention !6330
  • fix channel full condition; rebalance-cluster; transport !6331
  • feature flags: add limited-coexistence; transport: track closed endpoints !6334
  • fix prime-time: add caller-is-primary; up cli module !6335
  • switch existing cluster between HTTPS and HTTP !6336
  • Go 1.21: use built-in min and max functions !6337
  • list-objects(remote-bucket-and-only-remote-props); Go 1.21 clear built-in !6339
  • Go 1.20: use typed atomic pointer, remove unsafe !6343
  • core: assorted micro-optimizations; remove read locks !6346
  • tweak multi-error join-err, remove error channel (minor) !6347
  • [API change] capacity management !6348
  • xxhash; field-align vol package !6349
  • bucket: new-query help; silent GET; test tools !6350
  • etl: adding fqn param to spec templates !6351
  • low-level control structs: bucket, namespace !6352
  • etl: Keras template fix !6355
  • etl: fix hello-world ais-etl tests !6356
  • core: don't recompute uname hash !6359
  • repackage HRW methods !6361
  • core: lom cache v2 (major update) !6362
  • refactor: downloader's diff resolver; control plane (receive BMD) !6363
  • core: lom metadata cache (cont-ed) !6365
  • dsort: error handling, assorted cleanups, more scripted tests !6366
  • core transactions: concurrency !6368
  • downloader: throttle; wait !6369
  • optimize cold GET !6370
  • global rebalance: log; minor edits !6373
  • core: update backend 'get-reader' API (all supported backends) !6374
  • core: validate-warm-get to support non-versioned buckets, and more !6375
  • validate-warm-get to support non-versioned buckets !6376
  • [API change] silent HEAD(object) request !6378
  • core: add load-unsafe (the faster way to load local metadata) !6382
  • total disk size: compute at startup, recompute on change !6383
  • [API change] new bucket summary; unify list-objects and summary !6384, !6386, !6387
  • add config.Rom to consolidate assorted "read-mostly" config values; refactor and unify !6388
  • [API change] new bucket summary (major update) !6390
  • mountpath jogger: support bucket query !6392
  • backend providers: do not include (checksum, version) if not asked to !6394
  • python: updated bucket info API !6395
  • feature flags: don't-add-remote & don't-head-remote; log: add s3 module; verbosity; !6398
  • support listing remote buckets without adding them to cluster's BMD !6399
  • concurrent HEAD(object) vs evict/create bucket - fix the race !6400
  • [API change] list and summarize remote buckets without adding remote buckets to cluster's BMD !6401
  • datapath query (dpq) !6402
  • Go-based API: response header to error message !6403
  • [API change] new bucket summary !6405, !6406
  • downloader: streamline and cleanup initialization sequence !6409
  • HTTPS: support TLS configuration !6410, !6411, !6412, !6413, !6414, !6415, !6416
  • assorted minor fixes !6417, !6418
  • core: cold GET: fast path & slow path !6419
  • cluster configuration: flip validate-cold-get !6420
  • downloader (major update); [API change]: xaction registry !6422
  • validate-warm-get: add scripted test utilizing remote ais cluster !6423
  • core: cold GET: fast path & slow path !6424, !6427
  • feature flags: add disable-fast-cold-get; show performance latency; up cli module !6425
  • refactor ais/utils !6429

Bench: aisloader and aisloader-composer

  • skip list objects for 100% put load !6332
  • composer: add playbook and script for intial aisloader copy !6333
  • composer: add support for aisloader --filelist option !6345
  • default value for duration should be infinite if num-epochs value is defined !6353
  • composer: add epochs option for GET workloads !6354
  • composer: add cluster name prefix to netdata sources for easier filtering !6357
  • new bucket not to be listed; usability !6358

CLI

  • typed does-not-exist error; misc !6358
  • always print dsort job description !6367
  • show cluster to report total num disks !6371
  • show performance: usability fixes, improvements !6379
  • show performance not to filter regex-selected zero columns !6380
  • attempt to copy/transform an empty remote bucket !6393
  • new bucket summary; evict multiple buckets in one shot; pretty print !6396, !6397
  • ais show bucket with an option to add remote bucket to cluster's BMD (effectively, create bucket) !6404
  • ais search: CLI command search results to include idiomatic extensions !6428

Build, CI, Deployment

  • tests: upon node shutdown: wait for the node to stop (tcp) listening !6338
  • CI: add gather-logs template for K8s tests !6340
  • deploy: ais with HTTPS in minikube !6364
  • build: bump urllib version !6372
  • tests: validate-warm-get (scripted) !6423
  • K8s playbooks: update kill aisloader command !6385
  • docs: validate-warm-get; assorted !6377
  • docs: add performance.md; inline help; rm all-columns flag (redundant) !6381
  • build: upgrade all minors !6389
  • CI: add checkmarx scan !6391
  • build: upgrade golangci-lint, add linters !6407, !6408

3.20

12 Sep 15:14
Compare
Choose a tag to compare

Core

  • tweak stop-maintenance logic; rebalance: cleanup log messages; assorted minor fixes !6288
  • do not timestamp err-aborted message !6290
  • [API change] dsort: remove extended metrics; add new counters; revise and refactor !6297
  • list-objects; house-keeper; aisloader, logger (assorted fixes) !6298
  • core stats: remove mutex and work channel - speed up !6299
  • slab allocator: remove stats mutex, do not sort !6300
  • consolidate and revise OOM handling !6301
  • ETL: require admin access to create & delete; add feature flag !6302
  • remove unused heartbeat tracker w/ minor ref !6308
  • reimplement keep-alive mechanism (major) !6309
  • keep-alive v2 (major update) !6312
  • keep-alive v2: remove timeout stats (control structure and code) !6317
  • keep-alive v2: add fast path !6320
  • micro-optimize get-all-running (jobs); atomic heard-from/timed-out !6321
  • node-restarted: remove 'lsof', use net dialer; fix node-decommissioning tests !6322

Tools and tests

  • CI: update fspath (aka mountpath) config for minikube-based aistore deployments !6289
  • aisloader: list and read s3 buckets directly !6291
  • aisloader: list, read, and write s3 buckets directly !6292
  • tests: K8s long tests (EchoGolang) fix !6293
  • aisloader: fix cleanup option for s3 bucket benchmarks !6294
  • aisloader: reimplement direct get from s3 - use SDK !6295
  • aisloader: show progress when listing s3 directly !6296
  • CLI: add show details param to etl !6304
  • tools: add check for ais etl deployment !6305
  • tools: add ETL_NAME var for CLI tests !6310
  • aisloader-composer: add playbook and script for clearing Linux Page Cache on all AIS targets !6311
  • aisloader-composer: add playbook for copying aws credentials !6314
  • tools: update check for aistore Kubernetes deployment !6315
  • CI: update github action version (all modules) !6316
  • CLI/ETL: support enumerated arg-type !6287, !6323

Build

  • upgrade all OSS packages (minor versions) !6313
  • transition to Go 1.21 !6318

3.19

29 Aug 17:31
Compare
Choose a tag to compare

Core

  • [API change] archive and download logs (feature) !6172, !6175
  • [API change] dsort: extend input format !6181
  • [API change] dsort spec; CLI: print job spec !6204
  • [API change] revise request spec (major upd) !6217
  • [API change] dsort: is now 'xaction' as well !6253
  • (downloader, dsort, ETL): disallow to run when out of space !6235
  • handle "DNS lookup fail" as one of the unreachable err types; nlog flush-exit !6164
  • when electing new primary; when joining nodes at startup !6165
  • k8s: Change prod k8s and docker default to not log all to stderr !6166
  • revise GFN !6167
  • stats runner is now responsible to periodically flush logs !6170
  • core: fail user attempt to abort global rebalance when !6184
  • new Go API; assorted fixes !6189
  • metasync BMD; up modules !6190
  • downloader: return not-found when not found !6196
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • support S3 AWS profiles with alternative creds (feature) !6214
  • core: state transition => rebalance => (point of no return) !6216
  • amend low-level Go API check-response routine; add error type-code !6228, !6229
  • control plane: deserialize original error from call result !6230
  • xactions: when checking inactivity ("is idle") !6242 !6243
  • primary readiness vs cluster shutdown !6244
  • Go API: wait for xaction-related conditions !6245
  • assorted tuneups: space cleanup; housekeeping (HK) callback; log !6246
  • access control: when copying/transforming/dsorting to non-existing 'ais://' destination !6255
  • core: a call to update stats should never block !6257
  • core stats: add fast counters !6258 !6259 !6261
  • sparsify latency stats !6260
  • ETL: refactor and cleanup construction !6267
  • deploy/dev: updated minikube scripts !6272
  • new option to add Cloud bucket to aistore without checking accessibility !6275, !6277
  • un-throttle PUT mirroring; assorted changes !6278
  • feature: local generation of global (job) IDs !6280 !6282

Performance

  • Add distributed loader scripts and playbooks for using aisloader with multiple hosts !6156
  • pyaisloader: usability improvements !6215
  • Update Grafana dashboard to include latency statistics !6249
  • Reorganize benchmarks and related tools !6254
  • aisloader: no need to call rand for 100% or 50% read/write workloads !6256
  • aisloader-composer: add dashboard for DC network and disk !6266
  • aisloader: add an option to randomize gateways !6279
  • aisloader-composer: fix output files for GET bench !6283

Python

  • sdk: update ETL templates (docker migration) !6168
  • sdk: Release version 1.4.1 !6169
  • sdk: ETL templates (compress + ffmpeg decode) !6185
  • sdk: ETL templates (imagepullpolicy as always) !6191
  • sdk: adding keras_transform template !6200
  • sdk: ETL templates fix !6201
  • sdk: ETL templates (ffmpeg decode transformer) !6205
  • sdk: compress ETL template (updated usage) !6211
  • sdk: torchvision sample transformer ETL template !6221
  • sdk: fix comments (minor) !6240
  • sdk: update version !6248
  • sdk: increase timeout for torchvision transformer template (large image) !6252
  • sdk: updated torchvision transform ETL !6262
  • sdk: update dsort job info query and related tests !6265
  • sdk: switch ETL init code 'transform_url' boolean flag to 'arg_type' string !6269
  • docs: update ETL dev deployment for macOS !6163
  • ETL: keras template minor fix !6213
  • ETL: remove incorrect reference !6268
  • ETL: add 'arg-type=FQN' (new) !6271

Datasets (resize, resort, and shuffle)

  • [API change] dsort: extend input format !6181
  • dsort input format: iterate list, iterate range !6186 !6187
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • add test scripts; memsys: init gmm only once !6192
  • refactoring and renaming !6193
  • move/consolidate error types; continued refactoring !6202
  • Go API change; add dsort/api.go; CLI: print job spec !6203
  • [API change]: dsort spec; CLI: print job spec !6204
  • CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
  • dsort: continued refactoring (major update) !6208, !6209, !6210
  • free sgl on error; feature: any extension !6212
  • [API change] revise request spec (major upd) !6217
  • create destination on the fly !6218
  • record content path to retain full shard name !6219
  • output shard size estimation (rewrite) !6223
  • add is-compressed; refactor dsort-mem !6227
  • compressable shards (major) !6231
  • output ext; rcb buffer; fixes !6232
  • duplicated records (full coverage & stress); fixes !6233
  • fix tests; add stress !6234
  • rename subpackage, fix comments, refactor !6237
  • remove dsort-context, rewrite initialization !6238
  • static/stateless shard readers/writers; refactor and simplify !6239
  • two goroutines per each shard-distributing request !6241
  • [API change]: dsort: is now 'xaction' as well !6253
  • dsort: support generic abort-xaction API !6264
  • no need to block when sending shard records !6286

CLI

  • archive and download logs (feature) !6180
  • clarify "copying" vs "transforming" and "cached" vs "present" !6183
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • dsort: extend inline help, pretty-print job spec; update docs !6206
  • dsort: Go API change; add dsort/api.go; CLI: print job spec !6203
  • 'archive get' is now a shortcut (an alias) !6222

Build, test, and tools

  • add test scripts; memsys: init gmm only once !6192
  • tests and tools: cleanup around stop-maintenance, wait-rebalance" !6194
  • deployment: update local deployment script to allow target-only deployment with defined primary host !6195
  • deployment: optionally, skip deploying primary proxy !6197
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • tools/generate shards: optimize buffer allocation !6224
  • deploy/dev: Add ansible deployment scripts for deploying locally on multiple nodes !6199
  • aistorage/CI docker image (lzma libraries) !6220
  • tests: init with cleanup and without !6226
  • CI: Retry stuck Python ETL tests in GitLab CI pipeline !6270
  • remove aisfs (FUSE) !6273
  • dev tools: readers; handle read from corrupted arch or non-arch !6250

Documentation

  • update getting started !6161
  • updated python sdk readme !6162
  • update ETL dev deployment for macOS !6163
  • update documentation with recent ETL changes !6173
  • CLI/dsort: extend inline help, pretty-print job spec; update docs !6206

3.18

09 Jul 18:56
Compare
Choose a tag to compare

Core

  • add htext to track restarted state; target run and misc !5966
  • cluster rebalance (scenarios) !5969, !5971, !5973, !5974, !5975, !5977, !5980, !5983, !5986, !5987, !5989, !5991, !5992, !5993, !5995, !6002
  • add 'cluster-ready' helper; use it to reinforce !5976
  • cleanup better when decommissioning; previous BMD at startup !5979
  • fs: reliable remove-all !5981, !5982, !5984
  • yet another buf pool !5985
  • do not modify cluster map when starting up; always skip logging idle disks !5988
  • rebalance (scenarios, major update) !5992
  • [API change]: core: rebalance (scenarios) !5993
  • rebalance (major update); when receiving new cluster map !5995
  • up modules; handle housekeeper registration race !5994
  • 'not present in the loaded cluster map' and similar startup validation !5996
  • shutdown or decommission a node that's already in maintenance !5998
  • transport: never establish a streaming connection to the peer that's in maintenance (or will be) !5999
  • metasync just-in-time; assorted refactoring (minor) !6001
  • maintenance mode: pre & post vs keepalive & metasync; CLI: more colored cues !6004
  • shutdown is also 'maintenance'; docs: adding-removing intro !6005
  • add meta package !6006, !6007
  • ETL: add arg-type parameter when initializing with code !6008
  • archive v2: support empty template (tar entire bucket); atime !6013
  • keep poi.atime in nanoseconds !6015
  • archive v2: append to arch; refactoring !6017
  • archive v2: up modules !6018
  • archive v2: part four (major) !6019
  • archive v2: detect an empty tar when appending, and handle !6020
  • archive v2: part six !6022
  • archive v2: mime detection !6024
  • archive v2: extend 'append-to-arch' to support tar.gz !6025, !6027
  • archive v2: tar and tgz append; fixes !6028
  • log filenames; overlapping run vs node-restarted !6029
  • archive v2: multi-object append-to-arch !6030
  • archive v2: multi-object append-to-arch !6033
  • archive v2: multi-object append-to-arch !6034
  • cleanup disk utils (minor) !6035
  • ios startup: run the command only once !6036
  • hide AuthN secret !6038
  • archive v2: append to zip !6041
  • archive v2: append to msgpack !6043
  • add cmn/archive package !6044
  • archive v2: write and copy via new 'cmn/archive' !6045
  • archive v2: append via new 'cmn/archive' !6046
  • [API change] archive v2: MIME vs file extensions !6047
  • ios: cleanup lsblk cache; CLI: refactor get-node-arg; up modules !6048
  • archive v2: remove msgpack; refactor !6051
  • archive v2: add '.tar.lz4' serialization (new) !6053
  • archive v2: tar.lz4 cont-d !6054
  • archive v2: lz4 features; checksum !6055
  • s3 compat: run E2E tests with correct HTTP/HTTPS mode !6057
  • [API change]: append to arch if exists !6062
  • [API change] append to arch if doesn't exist; CLI cont-d !6064
  • checksumming and buffering vs reader-from !6074
  • core: content-length universally; revise write-json and friends !6075, !6076
  • archive v2: [API change] put (files, dirs) with an option to append !6081, !6082
  • archive v2: quiesce faster, refine continue-on-error logic !6083
  • core: double-check target-in-maintenance, quiesce faster !6084
  • archive v2: finalize cmn/archive package !6085
  • archive v2: finalize cmn/archive package !6086
  • log verbosity: core and modules !6087
  • http client: disable compression; core: undefer & micro-optimize !6066
  • append to (non-existing) arch: an option to create !6068
  • mem-pool alloc/free symmetry: copy/transform & archive !6069
  • copy/transform, multi-archive: refactor Rx logic and error handling !6071
  • log verbosity: core and modules; remote cluster !6089
  • log verbosity: core and modules; remote cluster !6091
  • ec: minor refactoring !6092
  • archive v2: WD basename; get with extraction; Range !6093
  • archive v2: WD basename; get with extraction; Range !6094
  • archive v2: tools/archive utils !6095
  • archive v2: tools/archive utils !6096
  • compile-out asserts; super-verbose logging; log module 'mirror' (ref) !6097
  • log verbosity at runtime; log modules; remove glog; unify (major update) !6099
  • fields iterator; size converter; log rotation (fixes) !6101
  • [API change] get-bucket-info to count remote objects !6102
  • [API change] get-bucket-info to count remote objects !6103
  • [API change] get-bucket-info (part three); docs and CLI !6106
  • list-objects vs buckets: revise and refactor, add validation, clarify !6107
  • list-objects: introduce optional args (ref, cleanup) !6108
  • list-objects: mem-pool msgpack buffers !6109
  • kvdb: remove redundant err-not-found; amend dsort, downloader, authn !6110
  • x-lso must idle more time !6111
  • log modules (part three) !6112
  • add nlog (new logger) !6113, !6122, !6124
  • do not log perf counters when there's no change; sort the names !6119
  • fix disk usage call for clusters on mac OS !6121
  • log etl events: spec parsed, pod ready, hpull/hpush !6125
  • cleanup fs.PathError; add object name validation !6127
  • extend dsort to support .tar.lz4 !6128
  • aisloader: cleanup output when running with json option !6129
  • xactions to return extended error info !6130
  • xactions: add and return errors (major upd) !6131, !6132
  • xactions: error handling cont-d; bucket-scope multi-object op-s !6133
  • jobs error handling and reporting; tests !6134, !6135, !6136
  • use Go 1.20 join-err; reduce default pre-election interval !6141
  • http request multiplexer !6142
  • dsort: use common cmn/archive pkg (major upd) !6143
  • archive v2: tar formats (USTAR, etc.) !6153
  • archive v2: extend list-objects and GET to operate inside archives !6155
  • archive v2: extend list-objects and GET to operate inside archives !6157
  • control plane: consistently propagate cluster map !6158
  • old EC metadata shouldn't terminate cluster-wide rebalance !6159
  • core: when primary goes down it notifies !6160

CLI

  • performance tabs: always show 'cluster idle' if idle !5997
  • stop-maintenance w/ no rebalance; stats idle-ness (minor) !6000
  • add transform-url flag (used when initializing ETL with code) !6010
  • misc. improvements !6011
  • mountpath completions; disable/detach; minor ref !6056
  • de-spaghettify put handler !6059
  • archive multi-object (cont-d) !6060
  • archive v2: CLI put, append, alias, docs !6072
  • move 'gen-shards' and extend it to support all formats !6073
  • archive v2: CLI: APPEND-to-arch is now on-par with multi-PUT !6077
  • archive v2: CLI: dry-run option is back !6078
  • archive v2: CLI: destination naming; dry-run; tips and examples !6079
  • list-objects: extended help, template with no ranges !6104
  • error and warning verbosity (major update) !6105
  • log modules (part two); CLI archive: pre-parse and add tip, advice !6100
  • archive help !6126
  • archive v2: CLI and docs !6149
  • archive v2: multiple CLI updates and improvements; is-archive bit !6150
  • remove k8s apimachinery pkg (minor) !6151

Python

  • sdk/python: Add option to pre-import modules when initializing ETL with code !5963
  • sdk/python: Refactor ETL to provide name once on object creation !5968
  • sdk/python: Release version 1.2.0 !5970
  • sdk/python: Fix python package links !5978
  • sdk/python: improve SDK testing, fix bucket eviction keep_md parameter !5990
  • sdk/python: Release version 1.2.2 !6012
  • sdk/python: Add dSort support in SDK !6032
  • sdk/python: Add wait to dsort abort test !6037
  • sdk/python: Fix dsort abort test on faster systems !6039
  • sdk/python: Add get_url option to bucket, object group, and objects !6042
  • sdk/python: Add interface for AIS sources that can be accessed via list of URLs !6052
  • PyTorch: Add new PyTorch DataPipe to iterate over URLs from various AIS sources !6058
  • update lint-tests to work with Python 3.11 (pytorch unsupported) !6061
  • sdk/python: Include user agent in SDK requests !6067
  • sdk/python Use msgpack content type when listing objects in bucket !6070
  • sdk/python: Add msgspec to pyproject dependencies !6088
  • sdk/python: Release SDK version 1.3.0 !6090
  • sdk/python: bucket summary + bucket info !6098
  • sdk/python: bck info integration test fix !6115
  • sdk/python: bck summary integration test fix !6117
  • sdk/python: bck summ + bck info fixes (revert and cleanup) !6120
  • sdk/python: pyaisloader !6123
  • pyaisloader: bucket utils (minor fixes) !6139
  • sdk/python: bucket summary and info (minor fixes) !6140
  • pyaisloader: total size/count fix !6146
  • sdk/python: Release version 1.4.0 !6152

Docs

  • add lifecycle.md !6009
  • WebDataset Blog Post part 1: Storing WebDataset in AIS !6014
  • WebDataset Blog Post part 2: AIS ETL on WebDataset shards !6016
  • update docs/tools and cmd/README !6050
  • WebDataset blog post pt 3 -- PyTorch Datatpipe with ETL !6065
  • archive v2: CLI put, append, alias, docs !6072
  • archive v2: CLI: destination naming; dry-run; tips and examples !6079
  • archive v2: CLI and docs !6149

Build, CI

  • add local build options to k8s scripts and fix local registry !6021
  • build: upgrade all minors !6040
  • add pyaisloader stage !6148
  • k8s: fix deployment scripts for compatibility with the latest aisnode image !6154

3.17

11 Apr 00:46
Compare
Choose a tag to compare

Table of Contents

  • CLI v1.2
  • Python SDK v1.1.2
  • S3 compatibility and Botocore
  • API changes
  • Tests and Documentation
  • Core: bug fixes and improvements
  • Build and Continuous Integration
  • Extensions: Downloader, dSort, ETL

See also:

CLI

  • show all jobs !5645
  • start/stop job/xaction !5660
  • refresh rate and countdown; long-running 'show job' and friends !5651
  • 'show log node-name' to mimic 'tail -f' !5652, !5654
  • add custom duration flag and logic !5655
  • 'ais config (cluster|node|cli)', 'ais config reset', and friends !5656
  • bucket completions !5657
  • set-config to show all updates; tweak iter-fields reflection !5658
  • 'show job' to aggregate all categories and support all selections !5661
  • transition to using job display names (major) !5663
  • start, stop, show jobs and xactions (cont-d) !5665
  • amend and restructure jobs !5666
  • running xactions (completions) !5672
  • tweak config json printout; get-config from memory !5673
  • update backend config !5674
  • update backend config (part two) !5677
  • add footnote, marshal message only once !5678
  • remove xaction term and subcommand (everything is job now) !5692
  • suggest (targets, proxies, nodes) !5694, !5696
  • revise bash completions script !5697
  • remove 'xaction' (term and subcommand) !5698, !5699, !5700
  • 'show cluster': separate cluster nodes from all other (tab-tab) completions !5701
  • consolidate and refactor cluster map access !5704
  • tweak ais create bucket --props & ais bucket props set !5706
  • extend 'job start' to support (resilver, copy-bucket, rename-bucket) !5715
  • tweak listed props !5719
  • remove (cleanup) download and dsort jobs !5721
  • extend 'ais stop' to support --all|--regex !5722
  • 'show job' verbose option; unify usage args; ref PUT/APPEND !5723
  • rewrite command-not-found logic; add similar commands !5724
  • show jobs (major) !5726, !5727
  • bash autocomplete ordering improvements !5728
  • improvements (usability) !5729
  • add bucket cp alias !5730
  • flag printable name; split 'show job' in parts; usability !5736
  • further unify stopping, waiting-for, and showing jobs !5744
  • revise & amend 'show rebalance' - all permutations !5761
  • universal start-end formatting; template refactoring !5762
  • jobs grouping by name and, within name, by UUID !5764
  • complete etl-name transition !5767
  • ETL tools, UUID (part one) !5745, !5746, !5749, !5753, !5754, !5763
  • fix download/dsort progress !5769
  • new table to show target statistics !5788
  • 'ais show performance' (new) !5791, !5793, !5800, !5802, !5803, !5809, !5810, !5811, !5812, !5816
  • IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5820
  • reduce code, simplify, cleanup !5821
  • IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5823
  • disk stats: add average read/write sizes !5824
  • amend existing mountpath tab and add a new one !5833, !5834
  • expect node unreachable when iterating '--refresh' !5837
  • assorted usability; add 'no-color' config !5839
  • 'ais show performance': average (GET, PUT, etc.) sizes on the fly !5840
  • support new API to reset stats !5841, !5843
  • 'ais show performance': refactor throughput, add latency !5844
  • 'ais show performance': finalize latency tab !5847
  • 'ais show performance' cont-d !5848
  • 'ais show performance': finalize top-level tab !5850
  • 'ais show performance': add cluster-level throughput, beautify !5852
  • 'ais show performance': alias 'stats' and remove older code !5853
  • 'ais show performance': disk table v2 !5855
  • 'ais show performance': finalize disk table !5857
  • 'ais show performance': new mountpaths/disks/capacity table !5858, !5859
  • 'ais show performance': finalize capacity table !5861
  • refactor and cleanup multi-object put !5862
  • multi-object PUT: source dir, list/range; matching pattern !5865
  • fix concatenation logic, refactor progress bar !5867
  • copy bucket: support progress bars (copied objects and size) !5870
  • consistent timeout management !5871
  • copy/transform a list or range of objects: add progress bar !5873
  • copy/transform with progress bar: style, reuse !5874
  • multi-object PUT !5876
  • rogress bar: all multi-object operations; universal 'wait-for' !5879
  • PUT multi-object - all flavors !5880
  • get multiple objects in one shot (""multi-object GET"") !5884
  • GET destination & assorted fixes !5882
  • copy-bucket: prepend prefix, command helps, examples !5889, !5891
  • more inline help !5892
  • assorted improvements (minor) !5900
  • fix downloading with progress bar enabled !5903
  • how-to text: how to reconfigure remote ais cluster !5932
  • add CLI compatibility warning (new CLI vs old cluster and vice versa) !5952
  • cluster membership-changing operations (shutdown, decommission, et al): improve usability !5956

Python: SDK library and ETL

  • Add unit tests for cluster class !5675
  • Add unit test for api client class !5676
  • Fix python ETL test workflow !5680
  • Add unit tests for Xaction class !5681
  • Add unit test for object class, fix object put data from a filepath !5682
  • Add unit tests for ETL class !5683
  • Add unit tests for sdk utils !5685
  • Restructure python subdirectories and containment !5686
  • Run python unit tests as part of default python test make option, include python ETL tests in all python test runs !5688
  • Only run python ETL tests when python labels are added !5702
  • update test utils to support running Python tests on Windows !5716
  • Add multi-object functionality !5720
  • Add tests and validation to object ranges, support leading zeros !5731
  • Add unit test for object group class !5732
  • Update documentation for python multi-object ops !5734
  • Use aws bucket for python sdk CI tests to test caching functions !5739
  • Add string template support to object groups !5741
  • Refactor all references from xaction to job !5742
  • Update ETL runtime defaults and add new python 3.11 option !5743
  • Fix remote bucket tests to avoid collisions !5748
  • Increment cloudpickle version and update github action to use 3.11 for tests !5756
  • Fix python test dependencies !5780
  • Refactor to simplify typing and tests !5789
  • Bump python SDK to 1.0.5 !5790
  • Update python sdk version !5792
  • Update pytorch integration README with compatibility issue !5794
  • Patch sdk to support torchdata integration !5795
  • Bump aistore package version !5798
  • Fix pylint and formatting !5799
  • Add ObjAttr type for returning additional object metadata !5805
  • Address lint warnings, general improvements !5808
  • Add writer option to object get !5813
  • Split README for different python projects !5814
  • Add PROMOTE functionality to objects !5818
  • Standardize pylint version, fix lint errors !5822
  • Improve object put behavior and add directory put options !5826
  • Use Pathlib over os.path !5827
  • Refactor multi-file put to bucket class !5830
  • Improve job interface and update promote options !5832
  • Update python package build tools, increase version to 1.0.9 !5835
  • Improve example documentation !5842
  • Improve input validation !5846
  • Set up proper logging, update constants !5849
  • Add job wait for idle status and fix job bucket filter !5863
  • Add multi-object copy !5866
  • Fix remote test fixture !5877
  • Add multi-object ETL !5878
  • Release 1.0.10 !5881
  • Expand multi-object examples !5883
  • Improve usability !5888
  • Follow-up for copy/transform prepend !5890
  • Improve object interface !5893
  • Add prefix_filter option to bucket copy !5899
  • Add flags and target options to bucket list_objects methods !5901
  • Release 1.1.0 !5910
  • Release 1.1.1 !5911
  • Improve python sdk examples !5913
  • Add cluster list_jobs, require id for individual job status query !5916
  • Add support for multi-object archive !5920
  • Update bucket params to use Bucket object, support namespaces !5925
  • Add prefix filter to bucket transform !5949
  • Release 1.1.2 !5950
  • PyTorch: add support for etl-name in AISDataset and AISFileLoaderIterDataPipe !5957

S3 compatibility and Botocore

  • Add botocore monkey patch alongside python SDK !5684
  • Move botocore and pytorch packages to python top-level, separate from sdk !5691
  • Add s3 compatibility testing with boto3 !5703
  • list-objects vs HEAD !5705
  • compute multipart md5 and set etag !5708
  • Improve s3 compat test documentation and update validated tests !5747
  • Pass S3 delete a list of objects !5854
  • fix infinite loop when listing objects; add bucket name to list object response !5864
  • return bucket creation date in UTC !5869
  • new flag NoRecursion to support S3 delimiter feature !5930

API changes and new APIs

  • add API to query multiple xactions via any IC proxy !5670, !5671
  • yet another API to query xactions (new) !5687
  • list-objects API: tweak listed props !5718
  • [API change] flatten xaction-snap control structure (major upd) !5740
  • [API change] ETL: tools, UUID (part nine) !5754
  • [API change] ETL: new query parameter to specify transform name !5755, !5765
  • [API change] init/start ETL to return xaction ID !5768
  • [API change] GET(object) !5770, !5772, !5773, !5775
  • [API change] remove xaction 'query-msg' (deprecated) !5783, !5784, !5785
  • get-object API: amend comments !5769
  • refactor api package (major) !5776
  • [new API] query metric names and kinds ('counter', 'latency', 'throughput', et al.) !5796
  • [API change] get node status !5811
  • [API change] core stats: consistency between stats-querying APIs !5817
  • [API change] PUT(object) !5828
  • [API change] amend capacity-disks-filesystems control !5831
  • [API change] add API to reset stats !...

3.16.rc2

31 Mar 14:23
Compare
Choose a tag to compare
v1.3.16

v3.16.rc2

3.12

13 Nov 19:47
Compare
Choose a tag to compare

Table of Contents

  • Remote AIS clusters
  • List objects
  • List and summarize buckets
  • S3 API
  • CLI
  • Documentation
  • Tests
  • Python SDK & ETL
  • Core
  • Refactoring & fixes
  • Tools
  • Build & CI

See also:

Remote AIS clusters

  • list remote ais buckets; fix remote namespace containment !5567, !5621
  • aliasing and bucket namespaces (major revision) !5622, !5623
  • ais gateways to cache remote ais clusters (information) !5630
  • extend remote ais errors - add cluster UUID !5631
  • remote ais aliasing !5632
  • [API change]: remote ais clusters (major) !5633, !5634
  • [API change] add HealthUptime; use it to show remote clusters !5636
  • remove support for multi-aliasing !5637
  • alias vs bucket metadata !5638

List objects

  • CLI to show CACHED (column) only when it's informative (and not when user asks ais ls ... --cached) !5554, !5555
  • rename list-objects structures; regenerate message pack !5542, !5543
  • amend and re-enforce single target rule for remote buckets !5556
  • introduce location property, phase-out target URL (property) !5557, !5559
  • add custom metadata, regenerate message pack; refactor !5564, !5565
  • list-objects version 2 (major): !5583, !5584, !5586, !5589, !5590, !5591, !5601, !5602, !5605, !5607
  • [backend API change] mem-pool 'list-objects' pages !5604
  • list-objects flow with a single target listing remote props (name, size, version) !5608
  • list remote pages v2: flow controls; ais gateways will now select single target to execute remote call !5609, !5610
  • list-objects to include (in listing result) custom object properties !5614
  • consistency in custom props across: get, put, head object, and list-objects !5615, !5616

List & summarize buckets

  • easy URL to support listing buckets via /gs, /az, and /ais endpoints !5486
  • list-buckets and get-bucket-info API; CLI summary table !5521, !5522
  • list-buckets version 2 (major update) !5524
  • unify 'list-buckets' and 'bucket-summary' !5530, !5531
  • shorter and faster list-buckets version: w/ presence but wo/ summary !5535, !5536- [API change] add presence filter; eliminate bucket-info control structure (is redundant); refactor !5540
  • list-buckets and bucket-summary cont-d !5541
  • bucket sizing: always compute two sizes: on-disk and sum objs !5544
  • bucket-summary on-disk (apparent) vs sum-all-objects, remote vs cached !5545
  • bucket-summary and list-buckets cont-d unification !5546, !5547
  • list-buckets by target: remote ais; hdfs and http; cloud; other !5567
  • rename/move bucket summary; align control structures !5568
  • list remote ais buckets; fix remote namespace containment !5621
  • [API change] list-buckets correctness for named bucket queries !5626

S3 API

  • multipart upload !5458, !5460, !5467, !5468
  • extend s3 API to support remote buckets (major) !5462
  • persistent multipart state (major) !5466
  • multipart GET !5472
  • multipart list active uploads and upload parts (major) !5474
  • return xml-formatted errors (compliance) !5480
  • multipart: refine and simplify !5482
  • revise list-buckets to include all providers !5485
  • always set header Server when redirecting !5495
  • amend s3 compatibility readme; add multipart example !5496
  • multipart: sha256 fix; md5 not checked but computed; aws s3api example !5500
  • xml-format errors (parts two and three) !5501
  • revise HEAD object implementation; multipart ETag; miscellaneous fixes !5504
  • compatibility: configurable root '/' access to AIS cluster !5572, !5573, !5574
  • compatibility: assorted error codes !5620

CLI

  • 'ais etl' to replace extension in transformed objects !5418
  • standardize error and warning coloring !5493
  • don't log and don't show backend secret !5506
  • improve unknown-command messaging, provide hints !5512
  • add ais object ls command; add present flags !5514
  • support updated HEAD(bucket) API; 'ais ls' and permutations to show obj props !5517
  • static linkage with cgo disabled !5526
  • rename aws, et al. backend provider constants !5528, !5529
  • option to display sizes either in bytes or KiB, MiB etc. units !5532
  • rename sub-packages; show-unmatched & hide-header !5534
  • bucket display-name helper (and consistent usage) !5539
  • flip bucket listing default to 'present' (was 'all accessible') !5550
  • option --all to list both buckets and objects (in remote buckets) !5551
  • multi-feature completions; action-warn and action-done !5575
  • show config subsections in JSON; multi-feature completions !5576
  • amend config JSON printout !5577
  • extend and amplify: (config | section | json) !5578
  • is-time-not-set, is-object-not-cached (for consistency) !5611
  • atime & ctime !5612
  • throughput must be averaged over an interval of time since the previous request !5643

Documentation

Tests

  • python ETL !5410, !5461, !5476
  • python IO communicator !5415
  • tests: skipping short (minor) !5416
  • pyhton/sdk: ETL stress test !5420
  • add s3 multipart test !5482
  • add ETL test for streaming with python runtime !5513
  • re-enable and amend object-properties tests across all providers !5515
  • amend list-buckets: add cases; up CLI !5569, !5570
  • reinforce multi-object tests, fix bucket-exists helper !5606
  • amend and fix multi-object range/list copy test !5619
  • amend list-buckets test that checks expected result !5631

Python SDK & ETL

  • ETL API part 1 !5370
  • sdk version !5374
  • code (re)formatting and misc !5381
  • reorg Makefile dependencies !5390
  • ETL APIs part 2 !5394
  • bucket ETL transform API !5405
  • bump sdk version to 1.0.3 !5408
  • fix IO communicator fixes !5413
  • fix transform bucket !5421, !5428
  • add examples !5441
  • add streaming runtime w/ refactoring and cleanup !5444
  • streaming init-code with chunk-size param (major) !5446, !5447
  • with temp debug !5448
  • init_code changes for before and after functions !5449
  • remove debug w/ minor ref !5450
  • functions filter (new), transform, and before/after !5451
  • runtime v2 fixes !5455
  • bootstrapper; by reference !5457
  • ETL API fixes !5464
  • update readme (etl init_code) !5475
  • extend hpush yaml with bitwise flags !5479
  • remove unused ETL functions; add test for streaming with python runtime !5513
  • update Jupyter notebook to reflect sdk changes !5548

Core

  • global rebalance: initiating new vs receiving delayed ACKs !5426
  • etl md: remove sgl, read-all !5470
  • error handling: introduce type codes (major upgrade) !5477, !5481
  • easy URL: support listing buckets via /gs, /az, and /ais endpoints !5486
  • easy URL: add readme with extended examples and comments !5487
  • user agent, internet browser: pretty-print JSON response !5492
  • add User-Agent to http request headers !5494
  • backward compatibility; remote backends !5498
  • consolidate http headers; add type-cast fail !5503
  • introduce error types unsupported and not-implemented-yet !5508
  • refactor GET(object) !5507
  • add lom.rename-file w/cleanups !5509
  • simplify lom.persist & recache/refresh !5510
  • update atime on PUT !5511
  • HEAD(bucket): extend the API in re: cluster metadata !5516
  • [API change] add api.GetBucketInfo !5519
  • enumerate and unify bucket and object 'presence' vs 'existence' !5520
  • fs: add size-bucket and refactor x-bucket-summary !5523
  • backend providers as simple KVs (minor) !5525
  • on the fly and (new) upon request remote bucket creation/addition !5538
  • object props: remove get-node, add location and helpers !5561, !5562, !5563
  • creating/adding buckets on the fly: further clarifications !5566
  • debug pkg: assert-msg is redundant; minor ref !5579
  • cos-assert; align (minor ref) !5587
  • periodic stats log: tweak idleness logic !5588
  • x-archive to always close data mover !5594
  • key-value d...

3.11

18 Jul 23:56
Compare
Choose a tag to compare

Highlights

Authentication and Access Control

  • CI: fix AuthN to run !5264
  • replace form3tech-oss/jwt and upgrade to jwt/v4 !5266
  • prep for production !5267
  • prep for production (major update) !5268
  • prep for production (part three) !5269
  • prep for production (part four) !5272
  • prep for production (aisloader) !5275
  • prep for production (clean_deploy) !5277
  • extend error info, add usage examples, tweak CLI help !5279
  • revamp ais auth CLI !5281
  • rename APIs for consistency; add CLI to update existing role !5283
  • E2E test: added basic AuthN test !5284
  • make cluster optional for role add/set; randomize E2E tests; unify auth-enabled checks !5285
  • rename AuthN entities for consistency !5288
  • add api package, isolate server internals (major) !5289
  • continued refactoring, assorted fixes !5291
  • add 'tok' package to consolidate all jwt calls; continued refactoring !5292
  • status 'forbidden' or 'unauthorized' to accompany all errors !5310
  • allow 3rd party client lookup ht://bucket !5312
  • separate enabling HTTPS for AuthN and AIS cluster when deploying !5313
  • oad-token: if user-given location is empty then !5314
  • secret handshake when adding/updating AIS clusters !5316
  • revisit AIS calling and retrying; unify errors !5317
  • login by non-superusers: require cluster (to be explicitly specified) !5326
  • assorted fixes, improvements !5333
  • introduce 'show-cluster' permission; reinforce admin access !5350

Python SDK

  • add copy_bucket method !5225
  • add method xact_status !5226
  • move test to cluster ops file !5232, !5233
  • add xaction start !5237
  • restructuring (adding pytorch) !5238
  • PyTorch plugin (dataset/aisio) implementation for aistore !5242
  • fixed bug - list empty buckets !5251
  • changed error handling !5258
  • add iterator for bucket objects !5259
  • improved pytorch plugin !5274
  • release 0.9.0 fixes !5276
  • cluster health api !5280
  • health check for non existing urls !5282
  • syncing aisio.py !5286
  • rename_bucket() port !5304
  • test suite changes !5305
  • makefile dependency fix !5307
  • cluster health api !5278
  • added aisio example !5322
  • restructure api (bucket) !5323
  • fixing aisio example !5325
  • aisio example fixes !5328
  • restructure cluster !5335
  • obj streamer fixes !5337
  • blog post for ais file loader and lister !5338
  • using session for requests !5341
  • restructure object !5342
  • object method name changes !5343, !5346
  • restructure fixes !5348
  • bucket api fixes !5351
  • object get fix !5352
  • restructure xaction operations !5353
  • fixes and release prep !5355
  • minor readme edits !5356
  • readme changes wrt pytorch !5357
  • api docs and make file fixes !5359
  • removing aistore.pytorch imports from main init !5360
  • makefile fix (pipx path) !5367

GitHub Action

  • add github action for building ais !5265
  • add github workflows for lint, python-tests and build !5270
  • add test-short github workflow (Linux) !5271
  • github/workflow: changed go lint version !5366

CLI

  • confirm assorted operations: decommission, rm bucket, rm -rf !5224
  • revamp ais show cluster; add K8s POD names !5243, !5244
  • bucket extra-props-to-update; filter by provider !5256
  • allow setting permission on per-bucket basis for a role !5309
  • revise 'show config', add nested completions !5339
  • revise 'set config', add local vs inherited !5344
  • add --cluster filter flag to 'auth show role' command !5349

Bucket Summary

  • revise bucket summary (major) !5230
  • bucket summary: micro-optimize and simplify !5262
  • bucket summary: multi-bucket; add min/max/avg counters; fixes !5299
  • bucket summary: apparent vs on-disk sizing, formatting clarity !5300, !5302

Common environment and system filenames

  • move and refactor common environment; AuthN version and build time !5318
  • filename constants; environment variables !5324
  • consolidate all configs under $HOME/.config/ais !5327

Misc. Features

  • warn when decommissioning cluster, support '--rm-user-data' option !5221
  • ability to decommission node and join it back with a new identity !5222
  • anonymous public-access buckets: list-objects anonymously (w/ API change) !5330
  • s3: the capability to set custom s3 endpoint !5257, !5260

Docs

  • tweak ETL markdown !5227
  • updated getting_started and etl markdown !5228
  • changes to standalone deployment README !5253
  • minor indentation fix to standalone readme !5254
  • standalone ec2 deployment instructions and benchmarks !5261
  • python api reference !5294
  • python_api.md fixes !5297
  • bucket.md to reflect 'summary' changes !5308
  • update 'ais ls', add anonymous and more !5336
  • main readme !5358
  • s3 compatibility, 's3cmd' configuration and getting-started !5361
  • main readme, index.md, python.md !5362
  • amend and extend cli/config.md; fix config node reset !5363
  • add 's3cmd.md' !5364

Tests

  • random page, remote bucket !5234
  • remote bucket, object props !5235
  • extend cleanup to support (prior to test, remote bucket) !5240
  • ensure more !5245
  • fix iter-fields unit for recently added config extras !5290
  • reverse proxy test against GCS !5329

Build

  • upgrade gopkg !5223
  • fix statsd github URL !5229
  • new (and supported) JWT packages !5266

Performance micro-optimizations, bug fixes, refactoring

  • aisloader: rewrite SGL freeing piece !5236
  • refactor low-level is-bucket, bucket-init, and friends !5239, !5241
  • BMD: speed-up bucket init; add global namespace uname (const) to avoid allocations !5246
  • BMD atomic update vs backend props !5247
  • reuse buffer when iterating range template !5248
  • devtools: refactor and simplify !5249
  • bucket extra-props-to-update: include readonly; CLI: filter by provider !5256
  • yet another case to skip err logging block devices !5273, !5295
  • cluster config: allow critical updates !5298
  • capacity: periodic log and refresh; version and build time; fixes !5315
  • local playground: loopback device sizing; fixes !5321
  • refactor and move docker pkg (formerly containers) !5331
  • list-objects flags: unify, rename, document, and reference !5332
  • config: restart-required selection; feature flags !5334
  • blog image fix !5347
  • assorted fixes and refactoring !5354
  • CI/CD: skip-ci & python-tests-only conflict fix !5365

3.10

23 May 23:28
Compare
Choose a tag to compare

Highlights

  • Global (cluster) config meta-version 2
  • Bucket metadata (BMD) meta-version 2
  • Python SDK
  • Build: Go 1.18; upgrade all packages
  • MessagePack archiving

Global Config v2 & Bucket Metadata v2

  • bump up cluster config's meta-version; handle backward compatibility !5122, !5123, !5124
  • redefine and extend feature flags !5132
  • capacity and space watermarks (cleanup, low/high, OOS) !5143, !5144
  • BMD meta-v2 with backward-compatible logic to load v1 !5145, !5150
  • BMD meta-v2: simplify mirroring configuration !5151
  • add transport section !5153
  • support loading previous meta-version !5154
  • readonly knobs; backward compatibility !5155
  • introduce cos.size type and "kb", "mb", etc. suffixes !5156
  • cos.size vs backward compatibility !5157
  • add memsys section and tunables !5158
  • log flush time interval !5159
  • join-cluster-at-startup timeout !5161
  • add log-stats time, add validators !5171
  • continued refactoring !5172
  • add tcb (transform-and-copy-bucket) section; stream bundle multiplier; compression !5173
  • revise docs/configuration.md !5180

msgpack archiving; list, create, and read msgpack formatted shards

  • msgp (generic) shard !5100, !5101
  • fix preambles in the msgp generated files !5086
  • alternative msgp; sgl as io.ByteScanner; Python unpack to test !5103
  • assorted text fixes; more msgpack !5104
  • archiving formats: add msgpack (in addition to tar, tgz, zip) !5107, !5108
  • add integration tests !5100, !5101

Python SDK

  • basic bucket and object operations !5109
  • introduce pydantic; refactor list-buckets !5120
  • CI: add Python SDK test !5112
  • improve list-objects API !5163
  • refactor API calls for usability: !5128
  • move direct requests call out of list objects, create and delete bucket, cluster map !5130
  • put-object & head-bucket API !5141
  • improve list object API !5163
  • fix return type in _request_raw method !5164
  • add head-object method !5166
  • add setUp and tearDown in tests !5167
  • improve creating temporary file !5168
  • add evict-bucket and delete-object; add props argument for list-objects !5175
  • get-object returns a stream instead of the entire content !5184
  • update README, docs !5178, !5190
  • fix naming; add read-all method for streams !5196
  • list_object to support pagination !5198
  • improve object streaming; rename msg.py -> types.py !5203
  • Python SDK tutorial !5218

CLI

  • show cluster config feature flags in human-readable format !5133
  • fix setting backend bucket props !5152
  • add ais object rm --all to remove all in one shot !5195
  • verbose and non-verbose multi-object operations !5197

Tests

  • re-enable smoke-test for Cloud buckets !5098
  • msgp (generic) shard !5100, !5101
  • add missing wait for put-copies; BMD string !5204
  • failback; devtools: wait-cluster; transport !5209
  • dummy keepalive tracker !5211

Docs

  • website: tech blog on promoting local and shared files !5082
  • explain msgp usage; add msgp-update to Makefile !5085
  • update msgp generator usage !5094
  • msgpack !5110
  • revise and amend configuration.md !5180
  • performance tuning !5115, !5116
  • development, first-time usage, production deployments (summary) !5127
  • getting started (recommendations) !5129
  • local playground, tracking access time (more edits) !5139
  • fix references, add a reference !5140
  • erasure coding (space utilization ratio, IO performance) !5142

Build

  • upgrade fasthttp package !5092
  • upgrade direct dependencies !5096, !5097
  • transition to Go 1.18 !5106
  • upgrade all dependencies except 1.18 !5113
  • upgrade all dependencies except 1.18; v3.10-rc2 !5181
  • upgrade all dependencies including 1.18; v3.10-rc3 !5186
  • upgrade all minors !5200

Bug fixes and performance improvements; continuous refactoring

  • skip bucket name, etc. validation when initializing lom !5083
  • follow-up !5084
  • unsafe string: inline and use it consistently !5087
  • write-msg-pack (ref) !5088
  • control-plane broadcast & unicast: result value factories and decoders !5089, !5090, !5091, !5093, !5095
  • global rebalance: dynamic reg/unreg streams; housekeeper name suffix !5099
  • fix sending http request body on redirect: recompute content-length !5102
  • not restoring atime when !5105
  • multi-object operations !5111
  • local from-scratch deployment: add user-friendly tips !5114
  • follow-up !5117, !5118
  • write-policy: both metadata and data; bump up config's meta-version; backward compatibility (major) !5121
  • deploy: fix Dockerfiles and related scripts !5126
  • assorted usability tweaks !5134
  • local playground: print ports AIS listens on !5135
  • remove bench/soaktest; rewrite devtools/readme !5136
  • config: write policy for data must be 'immediate' !5137
  • docker + LVM deployment: no disks fix !5146
  • move mock-bmd => cluster/mock (refactoring) !5147
  • docker + LVM deployment: no disks !5148
  • local playground: pgrep to check; quiet run-cmd !5160, !5162
  • fix deploy.sh script for older bash versions !5165
  • target startup: skip docker union mounts when enumerating drives !5169, !5170
  • dedup fix HEAD(obj) system attributes !5174
  • get-object lock/unlock (refactoring) !5176
  • URI parsing, config validations, and assorted ref !5177, !5179
  • making "rlock" exception to forcefully remove corrupted !5182
  • dont-lookup-remote-bucket = true: proceed silently, log-wise !5183, !5185
  • remove +build directive (obsolete) !5188, !5189
  • get random proxy (ref) !5191
  • cluster map (ref); node equality !5192
  • PUT object vs remote backend returning status=503 !5193
  • multi-object operation, range template (ref) !5194
  • aisloader works !5199
  • when shutting down warn with details !5201
  • api: xaction wait-IC min waiting time !5202
  • compute-checksum and clone-md (ref) !5205
  • memsys: mem-free to include buff/cache !5206
  • memsys: revise sys/mem (major) !5207
  • lif to lom conversion: free on error; EC error handling !5208
  • revise xs/rename-bucket !5210
  • deferred unlock via lightweight lif (ref) !5212
  • lom as a reader, lif as unlocker !5213
  • get-from-neighbor (gfn): reg hk housekeepr under lock; abort to self-expire !5215
  • lom copy: allocate with name !5216
  • lom xattr (ref) !5217

3.9

15 Mar 17:35
Compare
Choose a tag to compare
3.9

AIS v3.9 is substantial productization and performance-improving upgrade. Much of the codebase has been refactored for consistency, with numerous micro-optimization and stabilization fixes across the board.

Highlights

  • promote: redefine to handle remote file shares; collaborate when promoting via entire cluster; add usability options; productize;
  • xmeta: extend to also dump in a human-readable format: a) erasure-coded metadata and b) object metadata;
  • memory usage and fragmentation: consistently use mem-pooling (via sync.Pool) for all control structures in the datapath;
  • optimistic concurrency when running batch prefetch jobs; refactor and productize;
  • optimize PUT datapath;
  • core logic to deconflict running concurrent xactions (asynchronous jobs): bucket rename vs bucket copy, put a node into maintenance mode vs offline ETL, and similar;
  • extend and reinforce resilvering logic to withstand simultaneous disk losses/attachments - at runtime and with no downtime;
  • stabilize global rebalance to successfully pass multiple hours of random node "kills" and restarts - node-left and node-joined events - in presence of stressful data traffic;
  • self-healing: object metadata cache to support recovery upon mountpath events (e.g., drive failures);
  • error handling: phase out generic fmt.Errorf and consistently use assorted error types instead;
  • additional options to speedup listing of very large buckets (list-objects);
  • numerous micro-optimizing improvements: fast datapath query (DPQ) and many more.

Promote files and directories

  • refactor as a 2-phase transaction and auto-detect file share (initial) !4929
  • auto-detect file share and distribute the work between target nodes !4945
  • add test; add target node => IC notifications !4975
  • extend test coverage; reinforce global UUID when promoting via entire cluster !4976
  • rename api.Promote; add test permutations and checks !4985
  • remove redundant control structures; cleanup !4987
  • add API options delete-src (delete source) and overwrite-dst (overwrite destination)!4988
  • fix extra-copy optimization with full refactoring !4989
  • revise/optimize HEAD(object) implementation and utilize it when promoting with overwrite-dst=false (major) !4991
  • extend object write transaction (OWT) to support the flow !4992
  • support in (i.e., transmitted), out (i.e., received) and locally-promoted stats counters - files/objects and bytes !4993
  • introduce confirmed file share; add user option not to auto-detect file share !5019
  • CLI: add overwrite-dst and delete-src E2E tests !5024
  • consolidate control, eliminate ambiguity !5045
  • increase test coverage !5047, !5063
  • add all test permutations to cover (ais | cloud | remote-ais) bucket vs. (non-redundant | EC | n-way mirror) !5068

ETL

  • add CLI to view stored ETL code and specification !4925
  • handle target-down; test !4933
  • redefine and improve ETL API (!4947, !4966, !5022), including:
    • manage (CRUD wise) persistently stored ETL definitions
    • eliminate redundant URL path parameters
    • enforce uniqueness of the user-provided ETL name
  • remove (obsolete) embedded ETL-specific annotations from the init spec (pod template)
  • support stopping and restarting ETLs !5005, !5056
  • update ETL docs and fix minor bugs !4984, !5022

Global Rebalance

  • global rebalance status: always respond with total (cumulative) stats counters !4905
  • generic fs.Walk for global rebalance (refactoring) !4889, !4930
  • get-status & health !4934
  • global rebalance status: reimplement to optimize !4936
  • devtools: merge WaitForRebalanceToStart and WaitForAllResilver !4937
  • tweak/optimize receive logic !4994
  • abort via stage notifications from other target nodes (major) !5015
  • transport streams vs receive errors; assorted fixes !5040
  • tweak preemption logic (when rebalance triggering events arrive back to back) !5057
  • assorted fixes: global rebalance vs n-way mirroring & resilvering !5071, !5072
  • consistent renames and continuous refactoring !5075

Resilvering (in presence of drive failures and attachments)

  • tools and stats: wait-for-all-resilvers, multi-snap API !4888
  • resilvering vs copy management (major) !4865, !4866, !4867
  • resilvering: tweak is-active time interval !4882
  • support losing multiple disks (mountpaths) simultaneously !4884
  • multiple overlapping add/remove disk operations: fixes !4894
  • resilvering as scrubbing: recover objects to their expected (default) locations !4900
  • resilvering: interval-of-inactivity multiplier !4974
  • resilvering under stress in presence of lost mountpath(s) !5058

Asynchronous Jobs (aka xactions)

  • when aborting and propagating abort to the control-plane caller, make sure not to lose the original cause for the abort !4886
  • fix put-xaction finishing logic !4887
  • aborting jobs: propagate the original cause through channels and APIs !4890, !4891
  • revise lookup by only-running and/or by UUID !4897
  • move xaction and xreg packages with refactoring !4898
  • clarify running vs not finished xaction !4908
  • registry: fix matching logic, remove redundant code !4911, !4912
  • registry: amend housekeeping !4913
  • registry: continued refactoring and cleanup !4914
  • "limited coexistence" between running and about-to-run services (new) !4915, !4916, !4917
  • xact package: revise and optimize abort-checking concurrency !4923
  • registry: continued simplification and cleanup !4940
  • reinforce global UUID for all cluster-wide xactions !4978
  • IC notifications vs transactional xactions: same rules for all !4982
  • more stateful info: propagate xaction reference all the way into local PUT flow (major) !4995
  • copy-object -- xaction -- promote: continued refactoring !5002
  • registry: micro-optimizations and cleanup !5028

CLI

  • PUT: add an option not to load (skip loading) object metadata; amend docs; refactor and cleanup !4859
  • add a command to view ETL code/spec !4925
  • fix: do not add --help flag to the subcommands of subcommands !4926
  • amend 'show config' to include CLI config (in addition to cluster and local configs); fix cluster-unreachable error !4983
  • revise 'flag-is-set' for Boolean flags !5021, !5023, !5030
  • copy bucket: add --force option !5042
  • add start etl command !5056
  • ais show cluster: add support for refresh=<time-interval> and count options !5076
  • update CLI docs !5078
  • enable 'ais show storage' and friends to run continuously and refresh periodically !5079

Testing

  • test fixes to align with changes in the core !4861
  • add ensure-num-mountpaths helper, and reinforce !4892
  • use api.WaitForXaction instead of tutils.WaitForXactionByID !4893
  • re-enable one fs-checker test, allow more time for mountpaths !4903
  • add more checks when downloading object !4910
  • extend CLI e2e promote test !4932
  • WaitFor follow-up !4943
  • fix e2e AuthN messages !4952
  • retry upon failure to recover a damaged erasure-coded object !4986
  • amend and extend EC tests !4990
  • revise and enable bucket-rename-and-copy test !5060

CI/CD (continuous integration)

  • add CI job that runs on multiple cloud buckets !5027
  • add 1.18-rc1 version to build check !5044
  • add test-short-minimal to test a single-node cluster !5046
  • make AWS_REGION global env variable !5050
  • update test-long stage !5059

Bug fixes, performance improvements; continuous refactoring

  • LOM load to return distinct types: syscall-error and corrupted-error !4849
  • LOM vs n-way mirroring: fix and revise caching of the metadata !4850
  • list-objects: add fast mode --only-names !4851
  • introduce permission to overwrite disconnected backend !4852
  • api: refactor PUT API; fix devtools !4853
  • optimize PUT latency by allowing not to load object metadata !4854
  • aisloader bench: do not run goroutine per each PUT request !4855
  • reinforce access time atime (major) !4856
  • reintroduce no-metadata error; fix n-way stress; refactor !4857
  • build: fix deprecation warning on MacOS !4858
  • when copying objects differentiate between copying == mirroring and all other scenarios !4860
  • simplify LOM from-fs logic !4862
  • general: don't use regex to validate names and UUIDs !4863, !4864
  • assorted fixes !4868
  • preserve atime across LOM caches !4869
  • storage cleanup: leftover copies, corrupted and missing metadata !4870
  • refactor cmn and api packages !4871
  • list-objects: use-cache option !4872
  • consistently use HTTP status 507 throughout; assorted fixes !4875
  • eliminate redundant mirroring !4876
  • get-cold (aka cold-GET) follow-up !4877
  • object write transaction (OWT) fusion !4878
  • prefetch: support optimistic concurrency (major) !4879
  • name locker: fit two structures into 24 bytes !4880, !4881
  • disable/detach mountpath: graceful (admin request) and immediate (FSHC) !4883
  • move health package under fs/health !4885
  • bucket summary and obj-list query: move, refactor, and simplify !4895, !4896
  • control-plane: always free call-results back to pool !4899
  • api: eliminate code duplication !4901
  • general: deprecate and remove query objects !4902
  • refactor ais package !4904
  • control-plane transactions: refactor, reduce code !4906
  • initial ETL get API implementation !4907
  • control-plane transactions: follow-up !4909
  • introduce read-only (but still configurable) timeouts: cplane-operation and max-keepalive !4918
  • intra-cluster transport streams: tweak termination logic !4919
  • slab allocator: amend pooling of the SGL control structures !4920
  • transport s...