resume failed checking for s3endpoint when localstorage/file served

[austingnanaraj]

reference #530

when served path only /mnt/e/retinanet_checkpoints
ConnectionError: ObjStoreLibStorage preflight failed: cannot reach bucket '/mnt/e/retinanet_checkpoints' via s3dlio at endpoint 'http://:9020'. Underlying error: RuntimeError: list_objects_v2 failed: failed to construct S3 request — check AWS_REGION, AWS_ENDPOINT_URL, and credential environment variables (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY)

• Check AWS_ENDPOINT_URL (current: 'http://:9020').
• Check credentials are valid for that endpoint.
• Check that bucket '/mnt/ecs/retinanet_checkpoints' exists at the endpoint.
  File "/root/storage/.venv/lib/python3.12/site-packages/dlio_benchmark/checkpointing/pytorch_obj_store_checkpointing.py", line 90, in get_instance

when served file://mnt/e/retinanet_checkpoints
Error Details
What Happened:
You specified checkpoint folder with file:// protocol (NFS storage)
DLIO used storage.storage_type=s3 globally for all storage
s3dlio tried to parse file:///mnt/e/... as an S3 bucket
URI parser failed - can't extract bucket name from file:// URI
All 32 MPI ranks crashed during initialization

ConnectionError: ObjStoreLibStorage preflight failed: cannot reach bucket 'file:///mnt/ecs/retinanet_checkpoints' via s3dlio at endpoint 'http://[REDACTED_IP]:9020'. Underlying error: RuntimeError: Bucket name cannot be empty in URI: s3://file:///mnt/e/retinanet_checkpoints/

• Check AWS_ENDPOINT_URL (current: 'http://[REDACTED_IP]:9020').
• Check credentials are valid for that endpoint.
• Check that bucket 'file:///mnt/e/retinanet_checkpoints' exists at the endpoint.

[FileSystemGuy]

@austingnanaraj You'll see that I've created a PR to enhance the argument checking for mlpstorage to look for conflicting option sets.
That doesn't fix the core bug in DLIO, but it does prevent the user from getting into DLIO to hit the bug.

I think the core of the problem you're having is that you're telling mlpstorage that you want to run a training test (using S3) but then you're providing arguments with parameters for a checkpointing test (using files). Please only provide arguments that relate to the test you're asking mlpstorage to run, and ones that do not conflict (eg: S3 versus file).

In general, it appears you're passing in every option that is possible (eg: the DLIO_LOG_LEVEL). Perhaps that is an attempt to be explicit on what value they all have? The first lines of output from the latest versions of mlpstorage are a listing of all the values of all the relevant CLI arguments, as a way of documenting everything, so you don't need to pass everything in the way you are.

The least risk way to use the tool is to only include the CLI options that you need to include as a result of the default value for that option not being what you want. That's the codepath that has been tested. Specifying the values of parameters that you don't need to change brings a risk that those values will be wrong and will break the test, it's better to use the defaults unless you need to change something.

mlpstorage closed training retinanet run object
--data-dir retinanet_64p/retinanet
--results-dir /tmp/mlperf_results_resume
--accelerator-type b200
--num-accelerators 32
--client-host-memory-in-gb 754
--hosts gizmoc5,gizmoc6,gizmoc7,gizmoc8
--params storage.storage_type=s3
--params storage.storage_root=retinanet3
--params dataset.skip_listing=true
--params dataset.listing_validation_interval=10000
--params dataset.num_files_train=50203282
--params workflow.checkpoint=true
--params checkpoint.checkpoint_folder="file://${CHECKPOINT_BASE}/${CHECKPOINT_DIR}"
--params checkpoint.steps_between_checkpoints="${CHECKPOINT_INTERVAL}"
--params checkpoint.epochs_between_checkpoints="${CHECKPOINT_EPOCHS}"
--params checkpoint.checkpoint_after_epoch=true
--params checkpoint.model_size=500
--mpi-params "-x AWS_ACCESS_KEY_ID -x AWS_SECRET_ACCESS_KEY -x AWS_REGION -x S3_ENDPOINT_URIS -x S3DLIO_MAX_CONCURRENCY -x S3DLIO_CONNECTION_TIMEOUT -x S3DLIO_OPERATION_TIMEOUT_SECS -x S3DLIO_READ_TIMEOUT -x DLIO_LOG_LEVEL -x RUST_LOG -x UCX_LOG_LEVEL -x MLPS_CLUSTER_COLLECTOR_SHARED_STAGING --map-by slot --mca pml ob1 --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0,virbr0 --mca btl_tcp_endpoint_timeout 1200 --mca btl_tcp_connect_timeout 600 --mca btl_tcp_sndbuf 131072 --mca btl_tcp_rcvbuf 131072 --mca oob_tcp_if_exclude lo,docker0,virbr0 --mca oob_tcp_connect_timeout 600 --mca btl_openib_allow_ib 0 --mca btl_openib_warn_no_device_params_found 0 --mca orte_tmpdir_base /tmp --mca plm_rsh_agent ssh --mca orte_base_help_aggregate 0"
--allow-run-as-root

[russfellows]

I'm reviewing this issue now.

[FileSystemGuy]

@russfellows Thank you for looking at this! I've already gotten a PR ready to build a guardrail in mlpstorage to catch conflicting storage access API types (eg: S3 versus file).