Skip to content

3.12
Compare
Choose a tag to compare
@alex-aizman alex-aizman released this 13 Nov 19:47
v1.3.15
85636aa

Table of Contents

  • Remote AIS clusters
  • List objects
  • List and summarize buckets
  • S3 API
  • CLI
  • Documentation
  • Tests
  • Python SDK & ETL
  • Core
  • Refactoring & fixes
  • Tools
  • Build & CI

See also:

Remote AIS clusters

  • list remote ais buckets; fix remote namespace containment !5567, !5621
  • aliasing and bucket namespaces (major revision) !5622, !5623
  • ais gateways to cache remote ais clusters (information) !5630
  • extend remote ais errors - add cluster UUID !5631
  • remote ais aliasing !5632
  • [API change]: remote ais clusters (major) !5633, !5634
  • [API change] add HealthUptime; use it to show remote clusters !5636
  • remove support for multi-aliasing !5637
  • alias vs bucket metadata !5638

List objects

  • CLI to show CACHED (column) only when it's informative (and not when user asks ais ls ... --cached) !5554, !5555
  • rename list-objects structures; regenerate message pack !5542, !5543
  • amend and re-enforce single target rule for remote buckets !5556
  • introduce location property, phase-out target URL (property) !5557, !5559
  • add custom metadata, regenerate message pack; refactor !5564, !5565
  • list-objects version 2 (major): !5583, !5584, !5586, !5589, !5590, !5591, !5601, !5602, !5605, !5607
  • [backend API change] mem-pool 'list-objects' pages !5604
  • list-objects flow with a single target listing remote props (name, size, version) !5608
  • list remote pages v2: flow controls; ais gateways will now select single target to execute remote call !5609, !5610
  • list-objects to include (in listing result) custom object properties !5614
  • consistency in custom props across: get, put, head object, and list-objects !5615, !5616

List & summarize buckets

  • easy URL to support listing buckets via /gs, /az, and /ais endpoints !5486
  • list-buckets and get-bucket-info API; CLI summary table !5521, !5522
  • list-buckets version 2 (major update) !5524
  • unify 'list-buckets' and 'bucket-summary' !5530, !5531
  • shorter and faster list-buckets version: w/ presence but wo/ summary !5535, !5536- [API change] add presence filter; eliminate bucket-info control structure (is redundant); refactor !5540
  • list-buckets and bucket-summary cont-d !5541
  • bucket sizing: always compute two sizes: on-disk and sum objs !5544
  • bucket-summary on-disk (apparent) vs sum-all-objects, remote vs cached !5545
  • bucket-summary and list-buckets cont-d unification !5546, !5547
  • list-buckets by target: remote ais; hdfs and http; cloud; other !5567
  • rename/move bucket summary; align control structures !5568
  • list remote ais buckets; fix remote namespace containment !5621
  • [API change] list-buckets correctness for named bucket queries !5626

S3 API

  • multipart upload !5458, !5460, !5467, !5468
  • extend s3 API to support remote buckets (major) !5462
  • persistent multipart state (major) !5466
  • multipart GET !5472
  • multipart list active uploads and upload parts (major) !5474
  • return xml-formatted errors (compliance) !5480
  • multipart: refine and simplify !5482
  • revise list-buckets to include all providers !5485
  • always set header Server when redirecting !5495
  • amend s3 compatibility readme; add multipart example !5496
  • multipart: sha256 fix; md5 not checked but computed; aws s3api example !5500
  • xml-format errors (parts two and three) !5501
  • revise HEAD object implementation; multipart ETag; miscellaneous fixes !5504
  • compatibility: configurable root '/' access to AIS cluster !5572, !5573, !5574
  • compatibility: assorted error codes !5620

CLI

  • 'ais etl' to replace extension in transformed objects !5418
  • standardize error and warning coloring !5493
  • don't log and don't show backend secret !5506
  • improve unknown-command messaging, provide hints !5512
  • add ais object ls command; add present flags !5514
  • support updated HEAD(bucket) API; 'ais ls' and permutations to show obj props !5517
  • static linkage with cgo disabled !5526
  • rename aws, et al. backend provider constants !5528, !5529
  • option to display sizes either in bytes or KiB, MiB etc. units !5532
  • rename sub-packages; show-unmatched & hide-header !5534
  • bucket display-name helper (and consistent usage) !5539
  • flip bucket listing default to 'present' (was 'all accessible') !5550
  • option --all to list both buckets and objects (in remote buckets) !5551
  • multi-feature completions; action-warn and action-done !5575
  • show config subsections in JSON; multi-feature completions !5576
  • amend config JSON printout !5577
  • extend and amplify: (config | section | json) !5578
  • is-time-not-set, is-object-not-cached (for consistency) !5611
  • atime & ctime !5612
  • throughput must be averaged over an interval of time since the previous request !5643

Documentation

Tests

  • python ETL !5410, !5461, !5476
  • python IO communicator !5415
  • tests: skipping short (minor) !5416
  • pyhton/sdk: ETL stress test !5420
  • add s3 multipart test !5482
  • add ETL test for streaming with python runtime !5513
  • re-enable and amend object-properties tests across all providers !5515
  • amend list-buckets: add cases; up CLI !5569, !5570
  • reinforce multi-object tests, fix bucket-exists helper !5606
  • amend and fix multi-object range/list copy test !5619
  • amend list-buckets test that checks expected result !5631

Python SDK & ETL

  • ETL API part 1 !5370
  • sdk version !5374
  • code (re)formatting and misc !5381
  • reorg Makefile dependencies !5390
  • ETL APIs part 2 !5394
  • bucket ETL transform API !5405
  • bump sdk version to 1.0.3 !5408
  • fix IO communicator fixes !5413
  • fix transform bucket !5421, !5428
  • add examples !5441
  • add streaming runtime w/ refactoring and cleanup !5444
  • streaming init-code with chunk-size param (major) !5446, !5447
  • with temp debug !5448
  • init_code changes for before and after functions !5449
  • remove debug w/ minor ref !5450
  • functions filter (new), transform, and before/after !5451
  • runtime v2 fixes !5455
  • bootstrapper; by reference !5457
  • ETL API fixes !5464
  • update readme (etl init_code) !5475
  • extend hpush yaml with bitwise flags !5479
  • remove unused ETL functions; add test for streaming with python runtime !5513
  • update Jupyter notebook to reflect sdk changes !5548

Core

  • global rebalance: initiating new vs receiving delayed ACKs !5426
  • etl md: remove sgl, read-all !5470
  • error handling: introduce type codes (major upgrade) !5477, !5481
  • easy URL: support listing buckets via /gs, /az, and /ais endpoints !5486
  • easy URL: add readme with extended examples and comments !5487
  • user agent, internet browser: pretty-print JSON response !5492
  • add User-Agent to http request headers !5494
  • backward compatibility; remote backends !5498
  • consolidate http headers; add type-cast fail !5503
  • introduce error types unsupported and not-implemented-yet !5508
  • refactor GET(object) !5507
  • add lom.rename-file w/cleanups !5509
  • simplify lom.persist & recache/refresh !5510
  • update atime on PUT !5511
  • HEAD(bucket): extend the API in re: cluster metadata !5516
  • [API change] add api.GetBucketInfo !5519
  • enumerate and unify bucket and object 'presence' vs 'existence' !5520
  • fs: add size-bucket and refactor x-bucket-summary !5523
  • backend providers as simple KVs (minor) !5525
  • on the fly and (new) upon request remote bucket creation/addition !5538
  • object props: remove get-node, add location and helpers !5561, !5562, !5563
  • creating/adding buckets on the fly: further clarifications !5566
  • debug pkg: assert-msg is redundant; minor ref !5579
  • cos-assert; align (minor ref) !5587
  • periodic stats log: tweak idleness logic !5588
  • x-archive to always close data mover !5594
  • key-value driver; probabilistic filter !5595, !5596
  • embed stop channel !5597
  • transport: let individual streams define max header size !5598
  • intra-cluster transport: grow receive header, simplify rx-loop !5599
  • intra-cluster transport: within-read retries !5603
  • aws backend: rewrite list-objects versioning part, speed-up working with versioned buckets !5612, !5613
  • consistency in custom props across: get, put, head object, and list-objects !5615, !5616
  • simplify bucket try-init logic to add buckets on the fly !5628
  • notifications, control plane (ref) !5629
  • cluster and node 'started-up' is now uptime (was bool) !5635
  • [API change] add HealthUptime and use it to ais show remote clusters !5636

Refactoring & fixes

  • make deploy: mark aws secret optional to unblock startup !5379
  • consolidate xaction, move xs package !5383
  • move dbdriver, simplify !5384
  • GKE, cluster restart: amend false-positive SIE (storage integrity error) detection !5412
  • defer freeing etl-md sgl !5414
  • fieldalignment analyzer !5431, !5432, !5433, !5434, !5435, !5437, !5438, !5440, !5442
  • fix missing curl in aisnode k8s image !5427
  • http REST: parsing etc. helpers !5488
  • rename aws, etc. provider constants !5527
  • gofmt to replace interface{} with any !5533
  • refactor and revise objwalk pkg !5551, !5553
  • delete object vs 503 status; duplicated trname (transport endpoint name) !5614

Tools

  • scripts: revise github-release: tar.gz binaries, sha256(resulting tar.gz); naming convention !5411
  • add devtools/trand for tests; use strong randomization for product !5465
  • tools: revise get-primary, wait-for-cluster-state, wait-smap-sync !5571, !5580, !5581
  • aisloader: writing non-ais buckets !5600
  • tools: add buckets-contain helper !5627
  • scripts: amend local multi-cluster deployment to --override-backends and more !5617, !5618
  • scripts: add install_from_binaries !5640

Build & CI

  • github workflow: setting go version to 1.18.3 !5369
  • undo lint and Go version downgrades !5371, !5372
  • add .dockerignore !5382
  • cluster-minimal image: use alpine latest, remove azure !5385
  • deploy/k8s: fixes; minimal minikube deployment !5386
  • pull latest minikube image in GitHub workflow !5387, !5388
  • fully transition to Ubuntu 22; apt-get changes !5389
  • add limits.conf to all docker images !5391
  • deploy/prod: docker Makefile !5392
  • gitlab pipeline to build and push docker images !5393
  • add github workflow to build docker images !5395
  • two-stage Dockerfile to produce aisnode for minikube !5398
  • add Makefile for dev/k8s scripts !5399
  • add short CI stage to the k8s yaml template !5401
  • cleanup deploy/dev/k8s readme and scripts !5402
  • add cross-compile option !5409
  • upgrade all minors !5417
  • rebuild and use new docker images !5436
  • fix latest lint issues !5423
  • fix gitlab Dockerfile !5425
  • golangci: re-enable nolintlint, run go1.17, comment-out disabled !5439
  • run python/etl manually (was part of test-short) !5502
  • cli module to have its own Makefile !5505
  • github-release; never-nil (minor) !5422
  • lint: re-enable nolintlint !5424
  • update docker github workflow !5430
  • pylint: fix warnings !5453
  • golangci v.49 !5489
  • fully transition to Go 1.19 (part two) !5491
  • docker/cluster-minimal: configure aws and gcp backends !5499
  • gitlab/CLI: force back e2e test suite; upgrade and build !5552
  • upgrade all minor versions !5560
  • add early return (linter) !5582
  • update pylint settings to skip generated files !5642
Assets 8