Skip to content

3.19
Compare
Choose a tag to compare
@alex-aizman alex-aizman released this 29 Aug 17:31
v1.3.19
7dcc067

Core

  • [API change] archive and download logs (feature) !6172, !6175
  • [API change] dsort: extend input format !6181
  • [API change] dsort spec; CLI: print job spec !6204
  • [API change] revise request spec (major upd) !6217
  • [API change] dsort: is now 'xaction' as well !6253
  • (downloader, dsort, ETL): disallow to run when out of space !6235
  • handle "DNS lookup fail" as one of the unreachable err types; nlog flush-exit !6164
  • when electing new primary; when joining nodes at startup !6165
  • k8s: Change prod k8s and docker default to not log all to stderr !6166
  • revise GFN !6167
  • stats runner is now responsible to periodically flush logs !6170
  • core: fail user attempt to abort global rebalance when !6184
  • new Go API; assorted fixes !6189
  • metasync BMD; up modules !6190
  • downloader: return not-found when not found !6196
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • support S3 AWS profiles with alternative creds (feature) !6214
  • core: state transition => rebalance => (point of no return) !6216
  • amend low-level Go API check-response routine; add error type-code !6228, !6229
  • control plane: deserialize original error from call result !6230
  • xactions: when checking inactivity ("is idle") !6242 !6243
  • primary readiness vs cluster shutdown !6244
  • Go API: wait for xaction-related conditions !6245
  • assorted tuneups: space cleanup; housekeeping (HK) callback; log !6246
  • access control: when copying/transforming/dsorting to non-existing 'ais://' destination !6255
  • core: a call to update stats should never block !6257
  • core stats: add fast counters !6258 !6259 !6261
  • sparsify latency stats !6260
  • ETL: refactor and cleanup construction !6267
  • deploy/dev: updated minikube scripts !6272
  • new option to add Cloud bucket to aistore without checking accessibility !6275, !6277
  • un-throttle PUT mirroring; assorted changes !6278
  • feature: local generation of global (job) IDs !6280 !6282

Performance

  • Add distributed loader scripts and playbooks for using aisloader with multiple hosts !6156
  • pyaisloader: usability improvements !6215
  • Update Grafana dashboard to include latency statistics !6249
  • Reorganize benchmarks and related tools !6254
  • aisloader: no need to call rand for 100% or 50% read/write workloads !6256
  • aisloader-composer: add dashboard for DC network and disk !6266
  • aisloader: add an option to randomize gateways !6279
  • aisloader-composer: fix output files for GET bench !6283

Python

  • sdk: update ETL templates (docker migration) !6168
  • sdk: Release version 1.4.1 !6169
  • sdk: ETL templates (compress + ffmpeg decode) !6185
  • sdk: ETL templates (imagepullpolicy as always) !6191
  • sdk: adding keras_transform template !6200
  • sdk: ETL templates fix !6201
  • sdk: ETL templates (ffmpeg decode transformer) !6205
  • sdk: compress ETL template (updated usage) !6211
  • sdk: torchvision sample transformer ETL template !6221
  • sdk: fix comments (minor) !6240
  • sdk: update version !6248
  • sdk: increase timeout for torchvision transformer template (large image) !6252
  • sdk: updated torchvision transform ETL !6262
  • sdk: update dsort job info query and related tests !6265
  • sdk: switch ETL init code 'transform_url' boolean flag to 'arg_type' string !6269
  • docs: update ETL dev deployment for macOS !6163
  • ETL: keras template minor fix !6213
  • ETL: remove incorrect reference !6268
  • ETL: add 'arg-type=FQN' (new) !6271

Datasets (resize, resort, and shuffle)

  • [API change] dsort: extend input format !6181
  • dsort input format: iterate list, iterate range !6186 !6187
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • add test scripts; memsys: init gmm only once !6192
  • refactoring and renaming !6193
  • move/consolidate error types; continued refactoring !6202
  • Go API change; add dsort/api.go; CLI: print job spec !6203
  • [API change]: dsort spec; CLI: print job spec !6204
  • CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
  • dsort: continued refactoring (major update) !6208, !6209, !6210
  • free sgl on error; feature: any extension !6212
  • [API change] revise request spec (major upd) !6217
  • create destination on the fly !6218
  • record content path to retain full shard name !6219
  • output shard size estimation (rewrite) !6223
  • add is-compressed; refactor dsort-mem !6227
  • compressable shards (major) !6231
  • output ext; rcb buffer; fixes !6232
  • duplicated records (full coverage & stress); fixes !6233
  • fix tests; add stress !6234
  • rename subpackage, fix comments, refactor !6237
  • remove dsort-context, rewrite initialization !6238
  • static/stateless shard readers/writers; refactor and simplify !6239
  • two goroutines per each shard-distributing request !6241
  • [API change]: dsort: is now 'xaction' as well !6253
  • dsort: support generic abort-xaction API !6264
  • no need to block when sending shard records !6286

CLI

  • archive and download logs (feature) !6180
  • clarify "copying" vs "transforming" and "cached" vs "present" !6183
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • dsort: extend inline help, pretty-print job spec; update docs !6206
  • dsort: Go API change; add dsort/api.go; CLI: print job spec !6203
  • 'archive get' is now a shortcut (an alias) !6222

Build, test, and tools

  • add test scripts; memsys: init gmm only once !6192
  • tests and tools: cleanup around stop-maintenance, wait-rebalance" !6194
  • deployment: update local deployment script to allow target-only deployment with defined primary host !6195
  • deployment: optionally, skip deploying primary proxy !6197
  • start using scripted integration tests; CLI: 'dsort src dst spec' !6198
  • tools/generate shards: optimize buffer allocation !6224
  • deploy/dev: Add ansible deployment scripts for deploying locally on multiple nodes !6199
  • aistorage/CI docker image (lzma libraries) !6220
  • tests: init with cleanup and without !6226
  • CI: Retry stuck Python ETL tests in GitLab CI pipeline !6270
  • remove aisfs (FUSE) !6273
  • dev tools: readers; handle read from corrupted arch or non-arch !6250

Documentation

  • update getting started !6161
  • updated python sdk readme !6162
  • update ETL dev deployment for macOS !6163
  • update documentation with recent ETL changes !6173
  • CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
Assets 8
Loading