MLPerf Storage RetinaNet CLOSED Division Failure

[austingnanaraj]

Date: June 24, 2026 03:42 UTC
Test: RetinaNet Training - 4 Client, CLOSED Division

Summary
MLPerf Storage RetinaNet training benchmark in CLOSED division failed during warmup phase with a RuntimeError: dispatch failure in the s3dlio library. The test successfully passed validation and completed file listing (20+ minutes) but crashed immediately when attempting to read data during the warmup iteration.
Test Configuration (Redacted)
Parameter Value
Benchmark mlpstorage 3.0.16
Division CLOSED
Workload RetinaNet Training
Model RetinaNet B200
Clients 4 nodes (HOST1, HOST2, HOST3, HOST4)
Processes 16 (4 per client)
Accelerators 16 B200
Memory/Host 754 GB
Dataset 50,203,282 files (full dataset)
Epochs 8
Batch Size 24
Storage Configuration:
Library: s3dlio
Endpoints: 4 S3 endpoints (REDACTED)
Region: us-east-1
Load Balancing: round_robin
Concurrency: 64 in-flight per rank
Workers: 8 per rank
❌ Failure Details
Error Message:
RuntimeError: dispatch failure
Error Location:
python
File: dlio_benchmark/reader/_s3_iterable_mixin.py, line 424
Function: _s3_stream_s3dlio()
Code: while batch := item_iter.collect_batch(collect_n):
Full Stack Trace:
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "torch/utils/data/_utils/worker.py", line 374, in _worker_loop
data = fetcher.fetch(index)
File "torch/utils/data/_utils/fetch.py", line 44, in fetch
data = next(self.dataset_iter)
File "dlio_benchmark/data_loader/torch_data_loader.py", line 403, in iter
for _batch in self.reader.next():
File "dlio_benchmark/reader/image_reader_s3_iterable.py", line 83, in next
yield from self._s3_stream_next()
File "dlio_benchmark/reader/_s3_iterable_mixin.py", line 484, in _s3_stream_next
yield from self._s3_stream_s3dlio(obj_keys)
File "dlio_benchmark/reader/_s3_iterable_mixin.py", line 424, in _s3_stream_s3dlio
while batch := item_iter.collect_batch(collect_n):
RuntimeError: dispatch failure

⏱️ Timeline
Phase Time Duration Status
Start 03:42:17 - ✅
Environment Validation 03:42:17-03:42:21 4s ✅
File Listing 03:42:25-04:02:47 20m 22s ✅
Reshard for Epoch 1 04:02:48-04:02:50 2.29s ✅
Warmup Iteration 04:02:50+ <1s ❌ FAILED
Total Runtime 03:42:17-04:03:06 20m 49s ❌
✅ What Worked
✅ Environment validation passed
✅ CLOSED division qualification passed
✅ MPI collection across 4 hosts successful
✅ File listing completed (50.2M files in 20m 22s)
✅ File sharding completed (3,137,705 files per rank)
✅ Epoch reshard completed (2.29s)
✅ DataLoader initialization successful

[ITER_SIMPLE] worker=2 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=6 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=5 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=3 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=1 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=0 reader=ImageReaderS3Iterable files_this_worker=392214 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
Error executing job with overrides: ['workload=retinanet_b200', '++workload.storage.storage_type=s3', '++workload.storage.storage_root=retinanet3', '++workload.dataset.skip_listing=true', '++workload.dataset.listing_validation_interval=10000', '++workload.dataset.num_files_train=50203282', '++workload.storage.storage_options.storage_library=s3dlio', '++workload.storage.storage_options.uri_scheme=s3', '++workload.storage.s3_force_path_style=true', '++workload.dataset.data_folder=retinanet_64p/retinanet']
Traceback (most recent call last):
File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/main.py", line 781, in run_benchmark
benchmark.run()
File "/root/storage/.venv/lib/python3.12/site-packages/dftracer/python/ai_common.py", line 170, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/main.py", line 666, in run
next(warmup_iter)
File "/root/storage/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 718, in next
data = self._next_data()
^^^^^^^^^^^^^^^^^
File "/root/storage/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1525, in _next_data
return self._process_data(data, worker_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/storage/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 1563, in _process_data
data.reraise()
File "/root/storage/.venv/lib/python3.12/site-packages/torch/_utils.py", line 774, in reraise
raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/root/storage/.venv/lib/python3.12/site-packages/torch/utils/data/_utils/worker.py", line 374, in _worker_loop
data = fetcher.fetch(index) # type: ignore[possibly-undefined]
^^^^^^^^^^^^^^^^^^^^
File "/root/storage/.venv/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = next(self.dataset_iter)
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/data_loader/torch_data_loader.py", line 403, in iter
for _batch in self.reader.next():
^^^^^^^^^^^^^^^^^^
File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/reader/image_reader_s3_iterable.py", line 83, in next
yield from self._s3_stream_next()
File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/reader/_s3_iterable_mixin.py", line 484, in _s3_stream_next
yield from self._s3_stream_s3dlio(obj_keys)
File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/reader/_s3_iterable_mixin.py", line 424, in _s3_stream_s3dlio
while batch := item_iter.collect_batch(collect_n):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: dispatch failure

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
[ITER_SIMPLE] worker=4 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
[ITER_SIMPLE] worker=7 reader=ImageReaderS3Iterable files_this_worker=392213 total_files=3137705
Error executing job with overrides: ['workload=retinanet_b200', '++workload.storage.storage_type=s3', '++workload.storage.storage_root=retinanet3', '++workload.dataset.skip_listing=true', '++workload.dataset.listing_validation_interval=10000', '++workload.dataset.num_files_train=50203282', '++workload.storage.storage_options.storage_library=s3dlio', '++workload.storage.storage_options.uri_scheme=s3', '++workload.storage.s3_force_path_style=true', '++workload.dataset.data_folder=retinanet_64p/retinanet']
Traceback (most recent call last):
File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/main.py", line 781, in run_benchmark
benchmark.run()
File "/root/storage/.venv/lib/python3.12/site-packages/dftracer/python/ai_common.py", line 170, in wrapper

[FileSystemGuy]

cc @russfellows — same s3dlio surface as #504, worth your eyes when you have the bandwidth.

The failure is in s3dlio's Rust runtime, not in DLIO or mlpstorage. The Python stack stops at the FFI boundary (_s3_iterable_mixin.py:424 → collect_batch()) and the string "dispatch failure" is generic Tokio terminology — a task couldn't be spawned, or a channel send/recv failed against a runtime that's in a bad state. Without RUST_BACKTRACE we can't tell which.

Configuration fanout

• 16 ranks (4 hosts × 4 procs)
• 8 DataLoader workers per rank → 128 worker processes total
• prefetch_window = 64 per worker (default at _s3_iterable_mixin.py:382)
• 4 S3 endpoints, round-robin per rank → 4 ranks per endpoint
• Peak in-flight per endpoint: (16 / 4) × 8 × 64 = 2048 concurrent GETs

That fanout lands on each endpoint within milliseconds of the warmup iter starting. The first worker whose Tokio runtime can't spawn the prefetch task set raises "dispatch failure", and PyTorch surfaces it as worker process 0. The [ITER_SIMPLE] worker=7 lines repeated ~10× after the failure are PyTorch re-reporting the dead worker during pool shutdown — benign noise, not a second failure.

Most likely causes (ranked)

1. Tokio runtime not properly initialized in the spawned worker. DLIO uses multiprocessing_context: "spawn" so each worker re-imports s3dlio fresh. If there's a startup race or fd-limit interaction, the runtime fails silently and the first dispatch call surfaces it.
2. fd-limit exhaustion. 128 workers × ~64 connections = ~8200 sockets. Default ulimit -n of 1024 (or even 65536, per-process) can be hit.
3. Endpoint-rotation state not transferring across spawn. obj_store_lib.py:213-237 selects an endpoint by OMPI_COMM_WORLD_RANK. Worker subprocess inherits the env var, but if any rotation state lives in a Python global rather than env, it won't transfer.
4. Credential refresh failure on cold worker. AWS creds are inherited via env. If anything resets them between spawn and first dispatch, the signed request fails.
5. Cold-start request burst overwhelming the endpoint at the per-endpoint math above. Listing was 20 min of low-concurrency LIST; the fetch burst is 2048 concurrent GETs at instant t0.

Debug recipe for the reporter (@austingnanaraj)

The fastest way to narrow this from five candidates to one — no code change needed:

1. Re-run with RUST_BACKTRACE=full in the env. That replaces "dispatch failure" with the actual Tokio error chain. Single most useful step.
2. Try --params storage.storage_options.prefetch_window=8 (down from default 64). If the failure moves or disappears, fanout is the culprit.
3. Try --num-processes such that you get 1 worker per rank (DataLoader-level num_workers). If single-worker succeeds, the issue is per-worker init or fanout.
4. Try a single S3 endpoint (unset S3_ENDPOINT_URIS, set AWS_ENDPOINT_URL to one endpoint). If single-endpoint works, rotation logic or per-endpoint overload is the culprit.
5. Check ulimit -n on the client hosts and confirm ≥ 65536 per-process, not just per-shell.

Side note (not the immediate bug)

The Hydra overrides shown include ++workload.dataset.skip_listing=true (auto-injected by mlpstorage's _apply_skip_listing_params), yet the timeline shows 20m 22s of "File Listing". Either skip_listing doesn't apply to the RetinaNet path, or listing_validation_interval=10000 is still doing 1-in-10000 revalidation across 50.2M files. Worth surfacing separately — the operator paid 20+ minutes for what they thought was disabled. Probably its own issue once #506 is resolved.

Nothing to change in mlpstorage

mlpstorage just forwards storage_options to DLIO; the actual dispatch happens in s3dlio's Rust runtime. Any code fix lives in s3dlio (preferred — make the runtime init failure surface a real error message instead of "dispatch failure") or DLIO (defensive: catch the Rust-side error and re-raise with the underlying Tokio context). Once we have RUST_BACKTRACE output we'll know which.

[russfellows]

Root cause fixed in s3dlio 0.9.102

The RuntimeError: dispatch failure was an opaque Rust→Python error translation — SdkError::DispatchFailure's Display impl emits only the string "dispatch failure", losing the underlying OS/TLS reason (connection refused, timeout, certificate error, etc.).

What 0.9.102 ships:

Change	Detail
Error chain	All S3 operation errors now include the full cause — e.g. "GET s3://…: dispatch failure: connection error: connection refused (os error 111)"
Connect timeout	Default raised 10 s → 20 s to survive cold-start fan-out across 50 M-file datasets; configurable via S3DLIO_CONNECT_TIMEOUT_SECS
Retry attempts	Configurable via S3DLIO_MAX_RETRY_ATTEMPTS (default 3, clamped ≥ 1)

How to apply:

# Fast-fail on first bad endpoint (test config, not prod):
S3DLIO_CONNECT_TIMEOUT_SECS=3 S3DLIO_MAX_RETRY_ATTEMPTS=1 mlpstorage …

# Warm-start fan-out (50 M files, 4 hosts):
S3DLIO_CONNECT_TIMEOUT_SECS=30 mlpstorage …

mlcommons/storage#531 (merged) bumps the s3dlio floor to >=0.9.102 in mlpstorage. The fix is live on PyPI — uv sync after pulling main will pick it up.

Could you retest with 0.9.102 and confirm the error message is now actionable? Happy to dig further if the root cause turns out to be something other than a timeout or connection refusal.

[FileSystemGuy]

Please see Issue#548