Skip to content

Commit bc6e0e0

Browse files
Nahemah1022Nahemah1022
committed
docs: update blob downloader readme
Signed-off-by: Tony Chen <a122774007@gmail.com>
1 parent a51c7ca commit bc6e0e0

File tree

1 file changed

+117
-119
lines changed

1 file changed

+117
-119
lines changed

‎docs/blob_downloader.md‎

Lines changed: 117 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -1,63 +1,35 @@
1-
## Background
2-
3-
AIStore supports multiple ways to populate itself with existing datasets, including (but not limited to):
4-
5-
* **on demand**, often during the first epoch;
6-
* **copy** entire bucket or its selected virtual subdirectories;
7-
* **copy** multiple matching objects;
8-
* **archive** multiple objects
9-
* **prefetch** remote bucket or parts of thereof;
10-
* **download** raw http(s) addressable directories, including (but not limited to) Cloud storages;
11-
* **promote** NFS or SMB shares accessible by one or multiple (or all) AIS target nodes;
12-
13-
> The on-demand "way" is maybe the most popular, whereby users just start running their workloads against a [remote bucket](/docs/providers.md) with AIS cluster positioned as an intermediate fast tier.
14-
15-
But there's more. In particular, v3.22 introduces a special facility to download very large remote objects a.k.a. BLOBs.
16-
17-
We call this (new facility):
18-
191
## Blob Downloader
202

21-
AIS blob downloader features multiple concurrent workers - chunk readers - that run in parallel and, well, read certain fixed-size chunks from the remote object.
22-
23-
User can control (or tune-up) the number of workers and the chunk size(s), among other configurable tunables. The tunables themselves are backed up by system defaults - in particular:
24-
25-
| Name | Comment |
26-
| --- | --- |
27-
| default chunk size | 2 MiB |
28-
| minimum chunk size | 32 KiB |
29-
| maximum chunk size | 16 MiB |
30-
| default number of workers | 4 |
31-
32-
In addition to massively parallel reading (**), blob downloader also:
3+
Blob downloader is AIStore's facility for **downloading large remote objects (BLOBs)** using **concurrent range-reads**.
4+
Instead of pulling a 10–100+ GiB object with a single sequential stream, blob downloader:
335

34-
* stores and _finalizes_ (checksums, replicates, erasure codes - as per bucket configuration) downloaded object;
35-
* optionally(**), concurrently transmits the loaded content to requesting user.
6+
- **splits the object into chunks** (configurable chunk size),
7+
- **fetches those chunks in parallel** from the remote backend (configurable number of workers),
8+
- **writes them directly into AIStore's chunked object layout** so all target disks are writing in parallel, effectively aggregating the full disk write bandwidth of the node.
369

37-
> (**) assuming sufficient and _not_ rate-limited network bandwidth
10+
![Blob Downloader](/docs/assets/blob_downloader/blob_downloader_workflow.png)
3811

39-
> (**) see [GET](#2-get-via-blob-downloader) section below
12+
The result is that, beyond a certain object size, blob downloader can deliver **much higher throughput** than a regular cold GET. In our internal benchmarks, a 4 GiB S3 object fetched with blob downloader was up to **4× faster** than a monolithic cold GET.
4013

41-
## Flavors
14+
Blob downloader is also **load‑aware**: it consults AIStore's internal load advisors to avoid overcommitting memory or disks, backing off when the node is under pressure and running at full speed when the system has headroom.
4215

43-
For users, blob downloader is currently(**) available in 3 distinct flavors:
16+
For a deeper dive into the internals and detailed benchmarks, see the [blog post](https://aistore.nvidia.com/blog/2025/11/26/blob-downloader).
4417

45-
| Name | Go API | CLI |
46-
| --- | --- | --- |
47-
| 1. `blob-download` job | [api.BlobDownload](https://github.com/NVIDIA/aistore/blob/main/api/blob.go) | `ais blob-download` |
48-
| 2. `GET` request | [api.GetObject](https://github.com/NVIDIA/aistore/blob/main/api/object.go) and friends | `ais get` |
49-
| 3. `prefetch` job | [api.Prefetch](https://github.com/NVIDIA/aistore/blob/main/api/multiobj.go) | `ais prefetch` |
18+
---
5019

20+
## Usage
5121

52-
> (**) There's a plan to integrate blob downloader with [Internet Downloader](downloader.md) and, generally, all supported mechanisms that one way or another read remote objects and files.
22+
AIStore exposes blob download functionality through three distinct interfaces, each suited to different use cases.
5323

54-
> (**) At the time of this writing, none of the above is supported (yet) in our [Python SDK](https://github.com/NVIDIA/aistore/tree/main/python/aistore/sdk).
24+
- **Single object blob-download job** – explicitly start a blob-download job for one or more objects.
25+
- **Prefetch + blob-threshold** – route large objects in the prefetch job through blob downloader.
26+
- **Streaming GET** – stream a large object from blob downloader while it is being cached in AIS.
5527

56-
Rest of this text talks separately about each of the 3 "flavors" providing additional details, insights, and context.
28+
### 1. Single object blob-download job
5729

58-
## 1. Usage
30+
Use this when you want direct control over which/how objects are fetched with blob downloader.
5931

60-
To put some of the blob downloader's functionality into immediate perspective, let's see some CLI:
32+
**Help and options**:
6133

6234
```console
6335
$ ais blob-download --help
@@ -72,100 +44,126 @@ USAGE:
7244
ais blob-download BUCKET/OBJECT_NAME [command options]
7345

7446
OPTIONS:
75-
--chunk-size value Chunk size in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k; see '--units')
76-
--latest Check in-cluster metadata and, possibly, GET, download, prefetch, or otherwise copy the latest object version
77-
from the associated remote bucket;
78-
the option provides operation-level control over object versioning (and version synchronization)
79-
without the need to change the corresponding bucket configuration: 'versioning.validate_warm_get';
80-
see also:
81-
- 'ais show bucket BUCKET versioning'
82-
- 'ais bucket props set BUCKET versioning'
83-
- 'ais ls --check-versions'
84-
supported commands include:
85-
- 'ais cp', 'ais prefetch', 'ais get'
86-
--list value Comma-separated list of object or file names, e.g.:
87-
--list 'o1,o2,o3'
88-
--list "abc/1.tar, abc/1.cls, abc/1.jpeg"
89-
or, when listing files and/or directories:
90-
--list "/home/docs, /home/abc/1.tar, /home/abc/1.jpeg"
91-
--non-verbose, --nv Non-verbose (quiet) output, minimized reporting, fewer warnings
92-
--num-workers value Number of concurrent blob-downloading workers (readers); system default when omitted or zero (default: 0)
93-
--progress Show progress bar(s) and progress of execution in real time
94-
--refresh value Time interval for continuous monitoring; can be also used to update progress bar (at a given interval);
95-
valid time units: ns, us (or µs), ms, s (default), m, h
96-
--timeout value Maximum time to wait for a job to finish; if omitted: wait forever or until Ctrl-C;
97-
valid time units: ns, us (or µs), ms, s (default), m, h
98-
--wait Wait for an asynchronous operation to finish (optionally, use '--timeout' to limit the waiting time)
99-
--help, -h Show help
47+
chunk-size value Chunk size in IEC or SI units, or "raw" bytes (e.g.: 4mb, 1MiB, 1048576, 128k)
48+
num-workers value Number of concurrent blob-downloading workers (readers); system default when omitted or zero (default: 0)
49+
list value Comma-separated list of object or file names
50+
latest Check and optionally synchronize the latest object version from the remote bucket
51+
progress Show progress bar(s) in real time
52+
wait Block until the job finishes (optionally use '--timeout' to limit waiting time)
53+
...
10054
```
10155

102-
## 2. GET via blob downloader
56+
**Examples**:
10357

104-
Some of the common use cases boil down to the following:
58+
- **Single large object**
10559

106-
* user "knows" the size of an object to be read (or downloaded) from remote (cold) storage;
107-
* there's also an idea of a certain size _threshold_ beyond which the latency of the operation becomes prohibitive.
60+
```console
61+
$ ais blob-download s3://my-bucket/large-model.bin \
62+
--chunk-size 4MiB \
63+
--num-workers 8 \
64+
--wait --progress
65+
```
10866

109-
Thus, when the size in question is greater than the _threshold_ there's a motivation to speed up.
67+
- **Multiple objects in one job**
11068

111-
To meet this motivation, AIS now supports `GET` request with additional (and optional) http headers:
69+
```console
70+
$ ais blob-download s3://my-bucket \
71+
--list "obj1.tar,obj2.bin,obj3.dat" \
72+
--chunk-size 8MiB \
73+
--num-workers 4 \
74+
--wait --progress
75+
```
11276

113-
| Header | Values (examples) | Comments |
114-
| --- | --- | --- |
115-
| `ais-blob-download` | "true", "" | NOTE: to engage blob downloader, this http header must be present and must be "true" (or "y", "yes", "on" case-insensitive) |
116-
| `ais-blob-chunk` | "1mb", "1234567", "128KiB" | [system defaults](#blob-downloader) above |
117-
| `ais-blob-workers` | "3", "7", "16" | ditto |
77+
### 2. Prefetch with blob-threshold
11878

119-
* HTTP headers that AIStore recognizes and supports are always prefixed with "ais-". For the most recently updated list (of headers), please see [the source](https://github.com/NVIDIA/aistore/blob/main/api/apc/headers.go).
79+
`prefetch` is AIStore's **multi‑object “warm‑up” job** for remote buckets. When you add a **blob size threshold**, it automatically decides which objects are large enough to benefit from blob downloader:
12080

121-
## 3. Prefetch remote buckets w/ blob size threshold
81+
- Objects **≥ `--blob-threshold`** are fetched via blob downloader (parallel range‑reads, chunked writes).
82+
- Objects **< `--blob-threshold`** are fetched with the normal cold GET path.
12283

123-
`Prefetch` is another batch operation, one of the supported job types that can be invoked both via Go or Python call, or command line.
84+
This lets you get the large‑object gains of blob downloader by just tuning prefetch's knobs.
12485

125-
The idea of size threshold applies here as well, with the only difference being the _scope_: single object in [GET](#2-get-via-blob-downloader), all matching objects in `prefetch`.
126-
127-
> The `prefetch` operation supports multi-object selection via the usual `--list`, `--template`, and `--prefix` options.
128-
129-
But first thing first, let's see an example.
86+
**Example**:
13087

13188
```console
132-
$ ais ls s3://abc
89+
# Inspect a remote bucket
90+
$ ais ls s3://my-bucket
13391
NAME SIZE CACHED
134-
aisloader 39.30MiB no
135-
largefile 5.76GiB no
136-
smallfile 100.00MiB no
92+
model.ckpt 12.50GiB no
93+
dataset.tar 8.30GiB no
94+
config.json 4.20KiB no
95+
96+
# Prefetch with 1 GiB threshold:
97+
# - objects ≥ threshold use blob downloader (parallel chunks)
98+
# - objects < threshold use standard cold GET
99+
$ ais prefetch s3://my-bucket \
100+
--blob-threshold 1GiB \
101+
--blob-chunk-size 8MiB \
102+
--wait --progress
103+
prefetch-objects[E-abc123]: prefetch entire bucket s3://my-bucket
137104
```
138105

139-
Given the bucket (above), we now run `prefetch` with 1MB size threshold:
106+
Key prefetch options:
140107

141-
```console
142-
$ ais prefetch s3://abc --blob-threshold 1mb
143-
prefetch-objects[E-w0gjdm1z]: prefetch entire bucket s3://abc. To monitor the progress, run 'ais show job E-w0gjdm1z'
144-
```
108+
- **`--blob-threshold SIZE`**: turn blob downloader on for objects at/above `SIZE`.
109+
- **`--blob-chunk-size SIZE`** (if available in your build): override default blob chunk size for this prefetch.
110+
- **`--prefix` / `--list` / `--template`**: scope which objects are prefetched.
145111

146-
But notice, `prefetch` stats do not move:
112+
### 3. Streaming GET (Python SDK Only)
147113

148-
```console
149-
$ ais show job E-w0gjdm1z
150-
NODE ID KIND BUCKET OBJECTS BYTES START END STATE
151-
CAHt8081 E-w0gjdm1z prefetch-listrange s3://abc - - 10:08:24 - Running
152-
```
114+
In addition to CLI jobs, blob downloader can be used to stream large objects while they are concurrently downloaded in the cluster. This is useful when you want to feed data directly into an application (for example, model loading or preprocessing) and still keep a local cached copy in AIS.
153115

154-
And that is because it is the blob downloader that actually does all the work behind the scenes:
116+
```python
117+
from aistore import Client
118+
from aistore.sdk.blob_download_config import BlobDownloadConfig
155119

156-
```console
157-
$ ais show job blob-download
158-
blob-download[lP3Lpe5jJ]
159-
NODE ID KIND BUCKET OBJECTS BYTES START END STATE
160-
CAHt8081 lP3Lpe5jJ blob-download s3://abc - 20.00MiB 10:08:25 - Running
161-
```
120+
# Set up AIS client and bucket
121+
client = Client("AIS_ENDPOINT")
122+
bucket = client.bucket(name="my_bucket", provider="aws")
162123

163-
The work that shortly thereafter results in:
124+
# Configure blob downloader (4 MiB chunks, 16 workers)
125+
blob_cfg = BlobDownloadConfig(chunk_size="4MiB", num_workers="16")
164126

165-
```console
166-
$ ais ls s3://abc
167-
NAME SIZE CACHED
168-
aisloader 39.30MiB yes
169-
largefile 5.76GiB yes
170-
smallfile 100.00MiB yes
127+
# Stream large object using blob downloader settings
128+
reader = bucket.object("my_large_object").get_reader(blob_download_config=blob_cfg)
129+
data = reader.read_all()
171130
```
131+
132+
---
133+
134+
## Selecting an effective blob-threshold for prefetch
135+
136+
The ideal `--blob-threshold` depends on your cluster (CPU, disks, network), backend (S3/GCS/…​), and object size distribution.
137+
Running full `prefetch` experiments for many candidate values can easily take **hours**, so instead we recommend using a **shorter single‑object blob-download benchmark** to pick a good starting point and then using that value directly in your prefetch job.
138+
139+
To do this in practice, **compare cold GET vs. blob-download on a single object**:
140+
141+
1. **Pick a representative large remote object** in your bucket (for example, a model shard or big archive).
142+
2. **Evict it from AIStore** to ensure a cold path:
143+
144+
```console
145+
$ ais evict s3://my-bucket --list "large-model.bin"
146+
```
147+
148+
3. **Measure cold GET time** for that object:
149+
150+
```console
151+
$ time ais get s3://my-bucket/large-model.bin /dev/null
152+
```
153+
154+
4. **Measure blob-download time** for the same object:
155+
156+
```console
157+
$ ais evict s3://my-bucket --list "large-model.bin"
158+
159+
$ time ais blob-download s3://my-bucket/large-model.bin --wait
160+
```
161+
162+
5. Repeat the above for a few object sizes (for example: 64 MiB, 256 MiB, 1 GiB, 4 GiB) until you see a pattern:
163+
164+
- **Below some size**, cold GET is as fast or faster (blob overhead dominates).
165+
- **Above that size**, blob-download is consistently faster.
166+
167+
The **crossover size** where blob-download _wins_ is your **blob-threshold** for prefetch: use that size as `--blob-threshold` when you run your real `ais prefetch` job. This single‑object comparison gives you a quick, reasonable approximation.
168+
169+
In our internal 1.56 TiB S3 benchmark, applying this method led us to a threshold of about **256 MiB**. This value provided the best trade‑off for that specific cluster and workload and delivered roughly **2.3× faster** end‑to‑end prefetch compared to a pure cold‑GET baseline.

0 commit comments

Comments
 (0)