fix(dlio): strip URI scheme from storage.storage_root (#392)#459
Merged
Conversation
DLIO's obj_store_lib treats storage_root as a bare bucket/prefix and unconditionally prepends uri_scheme when building object URIs, so a user-supplied --params storage.storage_root=s3://bucket/path produced malformed URLs like s3://s3://bucket/path/... and every upload failed. Normalize storage.storage_root in process_dlio_params: if the value carries a <scheme>://, strip it before handing off to DLIO. The bare bucket form is unchanged. The proper fix still belongs in DLIO; this mitigation makes the reporter's command succeed against the current pin.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
russfellows
approved these changes
Jun 17, 2026
russfellows
left a comment
Contributor
There was a problem hiding this comment.
Approving per Curtis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #392.
Summary
When a user passes
--params storage.storage_root=s3://bucket/path, every S3 upload fails because DLIO'sobj_store_libbuilds URLs with a doubled scheme —s3://s3://bucket/path/.... The reporter saw "nothing in S3 / no bucket created" as a result.The doubled prefix originates in DLIO's
obj_store_lib.ObjStoreLibStorage:storage_factory.py:49passesargs.storage_rootstraight in asnamespace.obj_store_lib.py:159stores it verbatim.obj_store_lib.py:399-401unconditionally addsuri_scheme://when constructing object URIs:So a
storage_rootlikes3://mlperf-data/unet3dplus a key likedata/unet3d/train/img_000_of_168.npzproducess3://s3://mlperf-data/unet3d/data/unet3d/train/img_000_of_168.npz. MinIO rejects every put, and Keith's command exits without writing anything.--objectmode happens to avoid this because_apply_object_storage_paramssetsstorage_root = $BUCKET(bare bucket, no scheme). Users who hand-roll--params storage.storage_root=...hit it.What this PR does
DLIOBenchmark._strip_uri_schemeand normalizesstorage.storage_rootinprocess_dlio_paramsso any<scheme>://prefix is stripped before the override is handed off to DLIO. The bare-bucket form is untouched.s3://,s3://bucket/,az://,gs://, and bare-bucket cases, plus the reporter's exact command shape.The proper fix still belongs in DLIO's
obj_store_lib— once the pinned fork is patched andpyproject.toml:127is bumped, this normalization becomes a harmless no-op, since the normalized input is also valid input.Test plan
uv run pytest tests/unit -q→ 1372 passed, 4 skipped.mlpstorage training datagen --model unet3d --num-processes 1 --params storage.storage_library=s3dlio --params storage.storage_type=s3 --params storage.storage_root=s3://mlperf-data/unet3dagainst MinIO and confirm objects land ats3://mlperf-data/unet3d/data/unet3d/train/img_*.npz(singles3://).