When I run the command below, the recommendation is to create 257173 files. However, when I run training, it appears to want 1,170,301. It aborts the run since I do not have that many objects in my S3 store. There is also a disallowed override error. Let me know if I am reading the output wrong.
mlpstorage open training retinanet datasize
--systemname klpgpu
--accelerator-type b200
--client-host-memory-in-gb 15
--max-accelerators 1
--results-dir ~/results
The output:
Setting attr from max_accelerators to 1
2026-07-15 14:04:48|STATUS:
2026-07-15 14:04:48|STATUS: --- Run Configuration (mlpstorage 3.0.42) ---
2026-07-15 14:04:48|STATUS: benchmark: training
2026-07-15 14:04:48|STATUS: command: datasize
2026-07-15 14:04:48|STATUS: mode: closed
2026-07-15 14:04:48|STATUS: orgname: keithpij
2026-07-15 14:04:48|STATUS: systemname: klpgpu
2026-07-15 14:04:48|STATUS: data_dir: [not set]
2026-07-15 14:04:48|STATUS: results_dir: /home/keithpij/results
2026-07-15 14:04:48|STATUS: data_access_protocol: [not set]
2026-07-15 14:04:48|STATUS: num_accelerators: [not set]
2026-07-15 14:04:48|STATUS: num_processes: 1
2026-07-15 14:04:48|STATUS: accelerator_type: b200
2026-07-15 14:04:48|STATUS: client_host_memory_in_gb: 15.0
2026-07-15 14:04:48|STATUS: hosts: 127.0.0.1
2026-07-15 14:04:48|STATUS: exec_type: mpi
2026-07-15 14:04:48|STATUS: mpi_bin: mpirun
2026-07-15 14:04:48|STATUS: loops: 1
2026-07-15 14:04:48|STATUS:
2026-07-15 14:04:48|STATUS: --- Environment ---
2026-07-15 14:04:48|STATUS: MLPERF_RESULTS_DIR: [not set]
2026-07-15 14:04:48|STATUS: MLPERF_SYSTEMNAME: klpgpu
2026-07-15 14:04:48|STATUS: MPI_RUN_BIN: [not set]
2026-07-15 14:04:48|STATUS: MPI_EXEC_BIN: [not set]
2026-07-15 14:04:48|STATUS:
⠋ Validating environment... 0:00:002026-07-15 14:04:50|INFO: Environment validation passed
⠋ Capturing or verifying code image... 0:00:002026-07-15 14:04:50|STATUS: code image match found at /home/keithpij/results/keithpij/code-4be7cfbc
2026-07-15 14:04:50|STATUS: Benchmark results directory: /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448
2026-07-15 14:04:50|INFO: Collector script staged at /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/collector-staging/mlps_collector.py (persisted as run artifact)
2026-07-15 14:04:50|INFO: Running MPI collection across 1 host(s)
2026-07-15 14:04:51|INFO: MPI collection completed successfully (1 hosts reported)
2026-07-15 14:04:51|INFO: skip_listing enabled: 1,170,301 train files → validation_interval=1,000 (~1,172 HEAD checks at startup)
⠋ Collecting cluster info... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 2/4 0:00:00
⠋ Collecting cluster info... 0:00:002026-07-15 14:04:51|INFO: Collector script staged at /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/collector-staging/mlps_collector.py (persisted as run artifact)
⠸ Collecting cluster info... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 2/4 0:00:00
⠸ Collecting via MPI... 0:00:002026-07-15 14:04:51|INFO: MPI collection completed successfully (1 hosts reported)
2026-07-15 14:04:51|INFO: CAP-01 skipped: destination not local (e.g., remote vector-DB backend)
2026-07-15 14:04:51|INFO: CAP-01 skipped: destination not local (e.g., remote vector-DB backend)
2026-07-15 14:04:51|WARNING: Measured cluster memory (15.5GiB) exceeds the declared --client-host-memory-in-gb budget (15.0GiB); sizing against the measured value so the recommendation matches run-time check_num_files_train and submission_checker rule 3.1.2. See storage#785.
2026-07-15 14:04:51|RESULT: Minimum file count dictated by dataset size to memory size ratio.
2026-07-15 14:04:51|RESULT: Number of training files: 257173
2026-07-15 14:04:51|RESULT: Number of training subfolders: 0
2026-07-15 14:04:51|RESULT: Total disk space required for training: 77.35GiB
2026-07-15 14:04:51|WARNING: The number of files required may be excessive for some filesystems. You can use the num_subfolders_train parameter to shard the dataset. To keep near 10,000 files per folder use "25x" subfolders by adding "--params dataset.num_subfolders_train=25"
2026-07-15 14:04:51|RESULT: Run the following command to generate data:
mlpstorage closed training retinanet datagen file --hosts 127.0.0.1 --exec-type=mpi --num-processes=1 --results-dir=/home/keithpij/results --data-dir=<INSERT_DATA_DIR> --systemname=klpgpu --params dataset.skip_listing=True dataset.listing_validation_interval=1000 dataset.num_files_train=257173 dataset.num_subfolders_train=0 dataset.total_disk_bytes=83055820561
Processing results... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 0:00:00
⠋ Collecting cluster info... 0:00:002026-07-15 14:04:51|INFO: Collector script staged at /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/collector-staging/mlps_collector.py (persisted as run artifact)
Processing results... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 0:00:00
2026-07-15 14:04:52|STATUS: Writing metadata for benchmark to: /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/training_20260715_140448_metadata.json
Training command:
mlpstorage open training retinanet run object --num-accelerators 1 --accelerator-type b200 --client-host-memory-in
-gb 15 --data-dir retinanet --results-dir ~/results --skip-validation --params storage.storage_library=s3dlio --params storage.storage_type=s3
The output from the training command:
Setting attr from num_accelerators to 1
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:49|STATUS: --- Run Configuration (mlpstorage 3.0.42) ---
2026-07-15 13:50:49|STATUS: benchmark: training
2026-07-15 13:50:49|STATUS: command: run
2026-07-15 13:50:49|STATUS: mode: open
2026-07-15 13:50:49|STATUS: orgname: keithpij
2026-07-15 13:50:49|STATUS: systemname: klpgpu
2026-07-15 13:50:49|STATUS: data_dir: retinanet
2026-07-15 13:50:49|STATUS: results_dir: /home/keithpij/results
2026-07-15 13:50:49|STATUS: data_access_protocol: object
2026-07-15 13:50:49|STATUS: num_accelerators: 1
2026-07-15 13:50:49|STATUS: num_processes: 1
2026-07-15 13:50:49|STATUS: accelerator_type: b200
2026-07-15 13:50:49|STATUS: client_host_memory_in_gb: 15.0
2026-07-15 13:50:49|STATUS: hosts: 127.0.0.1
2026-07-15 13:50:49|STATUS: exec_type: mpi
2026-07-15 13:50:49|STATUS: mpi_bin: mpirun
2026-07-15 13:50:49|STATUS: loops: 1
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:49|STATUS: --- Environment ---
2026-07-15 13:50:49|STATUS: MLPERF_RESULTS_DIR: [not set]
2026-07-15 13:50:49|STATUS: MLPERF_SYSTEMNAME: klpgpu
2026-07-15 13:50:49|STATUS: MPI_RUN_BIN: [not set]
2026-07-15 13:50:49|STATUS: MPI_EXEC_BIN: [not set]
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:49|STATUS: --- Object Storage (S3) ---
2026-07-15 13:50:49|STATUS: bucket: [not set]
2026-07-15 13:50:49|STATUS: storage_library: s3dlio
2026-07-15 13:50:49|STATUS: uri_scheme: s3
2026-07-15 13:50:49|STATUS: endpoint: [not set]
2026-07-15 13:50:49|STATUS: load_balance_strategy: round_robin
2026-07-15 13:50:49|STATUS: aws_region: us-east-1
2026-07-15 13:50:49|STATUS: aws_ca_bundle: [not set]
2026-07-15 13:50:49|STATUS: AWS_ACCESS_KEY_ID: [not set]
2026-07-15 13:50:49|STATUS: AWS_SECRET_ACCESS_KEY: [not set]
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:51|WARNING: Skipping environment validation (--skip-validation flag)
⠋ Capturing or verifying code image... 0:00:002026-07-15 13:50:51|STATUS: code image match found at /home/keithpij/results/keithpij/code-4be7cfbc
2026-07-15 13:50:51|STATUS: Benchmark results directory: /home/keithpij/results/open/keithpij/results/klpgpu/training/retinanet/run/20260715_135049
2026-07-15 13:50:51|INFO: Collector script staged at /home/keithpij/results/open/keithpij/results/klpgpu/training/retinanet/run/20260715_135049/collector-staging/mlps_collector.py (persisted as run artifact)
2026-07-15 13:50:51|INFO: Running MPI collection across 1 host(s)
2026-07-15 13:50:51|INFO: MPI collection completed successfully (1 hosts reported)
2026-07-15 13:50:52|INFO: --object mode: loading credentials from /home/keithpij/storage/.env
2026-07-15 13:50:52|INFO: --object mode: injected storage params (storage_type=s3, storage_root=mlcommons, library=s3dlio, uri_scheme=s3, force_path_style=True)
2026-07-15 13:50:52|INFO: skip_listing enabled: 1,170,301 train files → validation_interval=1,000 (~1,172 HEAD checks at startup)
2026-07-15 13:50:52|INFO: Created benchmark run: training_run_retinanet_20260715_135049
2026-07-15 13:50:52|STATUS: Verifying benchmark run for training_run_retinanet_20260715_135049
2026-07-15 13:50:52|RESULT: Minimum file count dictated by dataset size to memory size ratio.
2026-07-15 13:50:52|ERROR: INVALID: [INVALID] Disallowed parameter override: storage.storage_library = s3dlio (Parameter: Overrode Parameters, Expected: None, Actual: s3dlio)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_type = s3 (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_root = mlcommons (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_options.storage_library = s3dlio (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_options.uri_scheme = s3 (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.s3_force_path_style = true (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: dataset.skip_listing = True (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: dataset.listing_validation_interval = 1000 (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Benchmark run is INVALID due to 1 issues ([RunID(program='training', command='run', model='retinanet', run_datetime='20260715_135049')])
2026-07-15 13:50:52|ERROR: Invalid configuration found. Aborting benchmark run.
When I run the command below, the recommendation is to create 257173 files. However, when I run training, it appears to want 1,170,301. It aborts the run since I do not have that many objects in my S3 store. There is also a disallowed override error. Let me know if I am reading the output wrong.
mlpstorage open training retinanet datasize
--systemname klpgpu
--accelerator-type b200
--client-host-memory-in-gb 15
--max-accelerators 1
--results-dir ~/results
The output:
Setting attr from max_accelerators to 1
2026-07-15 14:04:48|STATUS:
2026-07-15 14:04:48|STATUS: --- Run Configuration (mlpstorage 3.0.42) ---
2026-07-15 14:04:48|STATUS: benchmark: training
2026-07-15 14:04:48|STATUS: command: datasize
2026-07-15 14:04:48|STATUS: mode: closed
2026-07-15 14:04:48|STATUS: orgname: keithpij
2026-07-15 14:04:48|STATUS: systemname: klpgpu
2026-07-15 14:04:48|STATUS: data_dir: [not set]
2026-07-15 14:04:48|STATUS: results_dir: /home/keithpij/results
2026-07-15 14:04:48|STATUS: data_access_protocol: [not set]
2026-07-15 14:04:48|STATUS: num_accelerators: [not set]
2026-07-15 14:04:48|STATUS: num_processes: 1
2026-07-15 14:04:48|STATUS: accelerator_type: b200
2026-07-15 14:04:48|STATUS: client_host_memory_in_gb: 15.0
2026-07-15 14:04:48|STATUS: hosts: 127.0.0.1
2026-07-15 14:04:48|STATUS: exec_type: mpi
2026-07-15 14:04:48|STATUS: mpi_bin: mpirun
2026-07-15 14:04:48|STATUS: loops: 1
2026-07-15 14:04:48|STATUS:
2026-07-15 14:04:48|STATUS: --- Environment ---
2026-07-15 14:04:48|STATUS: MLPERF_RESULTS_DIR: [not set]
2026-07-15 14:04:48|STATUS: MLPERF_SYSTEMNAME: klpgpu
2026-07-15 14:04:48|STATUS: MPI_RUN_BIN: [not set]
2026-07-15 14:04:48|STATUS: MPI_EXEC_BIN: [not set]
2026-07-15 14:04:48|STATUS:
⠋ Validating environment... 0:00:002026-07-15 14:04:50|INFO: Environment validation passed
⠋ Capturing or verifying code image... 0:00:002026-07-15 14:04:50|STATUS: code image match found at /home/keithpij/results/keithpij/code-4be7cfbc
2026-07-15 14:04:50|STATUS: Benchmark results directory: /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448
2026-07-15 14:04:50|INFO: Collector script staged at /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/collector-staging/mlps_collector.py (persisted as run artifact)
2026-07-15 14:04:50|INFO: Running MPI collection across 1 host(s)
2026-07-15 14:04:51|INFO: MPI collection completed successfully (1 hosts reported)
2026-07-15 14:04:51|INFO: skip_listing enabled: 1,170,301 train files → validation_interval=1,000 (~1,172 HEAD checks at startup)
⠋ Collecting cluster info... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 2/4 0:00:00
⠋ Collecting cluster info... 0:00:002026-07-15 14:04:51|INFO: Collector script staged at /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/collector-staging/mlps_collector.py (persisted as run artifact)
⠸ Collecting cluster info... ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 2/4 0:00:00
⠸ Collecting via MPI... 0:00:002026-07-15 14:04:51|INFO: MPI collection completed successfully (1 hosts reported)
2026-07-15 14:04:51|INFO: CAP-01 skipped: destination not local (e.g., remote vector-DB backend)
2026-07-15 14:04:51|INFO: CAP-01 skipped: destination not local (e.g., remote vector-DB backend)
2026-07-15 14:04:51|WARNING: Measured cluster memory (15.5GiB) exceeds the declared --client-host-memory-in-gb budget (15.0GiB); sizing against the measured value so the recommendation matches run-time check_num_files_train and submission_checker rule 3.1.2. See storage#785.
2026-07-15 14:04:51|RESULT: Minimum file count dictated by dataset size to memory size ratio.
2026-07-15 14:04:51|RESULT: Number of training files: 257173
2026-07-15 14:04:51|RESULT: Number of training subfolders: 0
2026-07-15 14:04:51|RESULT: Total disk space required for training: 77.35GiB
2026-07-15 14:04:51|WARNING: The number of files required may be excessive for some filesystems. You can use the num_subfolders_train parameter to shard the dataset. To keep near 10,000 files per folder use "25x" subfolders by adding "--params dataset.num_subfolders_train=25"
2026-07-15 14:04:51|RESULT: Run the following command to generate data:
mlpstorage closed training retinanet datagen file --hosts 127.0.0.1 --exec-type=mpi --num-processes=1 --results-dir=/home/keithpij/results --data-dir=<INSERT_DATA_DIR> --systemname=klpgpu --params dataset.skip_listing=True dataset.listing_validation_interval=1000 dataset.num_files_train=257173 dataset.num_subfolders_train=0 dataset.total_disk_bytes=83055820561
Processing results... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 0:00:00
⠋ Collecting cluster info... 0:00:002026-07-15 14:04:51|INFO: Collector script staged at /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/collector-staging/mlps_collector.py (persisted as run artifact)
Processing results... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4/4 0:00:00
2026-07-15 14:04:52|STATUS: Writing metadata for benchmark to: /home/keithpij/results/closed/keithpij/results/klpgpu/training/retinanet/datasize/20260715_140448/training_20260715_140448_metadata.json
Training command:
mlpstorage open training retinanet run object --num-accelerators 1 --accelerator-type b200 --client-host-memory-in
-gb 15 --data-dir retinanet --results-dir ~/results --skip-validation --params storage.storage_library=s3dlio --params storage.storage_type=s3
The output from the training command:
Setting attr from num_accelerators to 1
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:49|STATUS: --- Run Configuration (mlpstorage 3.0.42) ---
2026-07-15 13:50:49|STATUS: benchmark: training
2026-07-15 13:50:49|STATUS: command: run
2026-07-15 13:50:49|STATUS: mode: open
2026-07-15 13:50:49|STATUS: orgname: keithpij
2026-07-15 13:50:49|STATUS: systemname: klpgpu
2026-07-15 13:50:49|STATUS: data_dir: retinanet
2026-07-15 13:50:49|STATUS: results_dir: /home/keithpij/results
2026-07-15 13:50:49|STATUS: data_access_protocol: object
2026-07-15 13:50:49|STATUS: num_accelerators: 1
2026-07-15 13:50:49|STATUS: num_processes: 1
2026-07-15 13:50:49|STATUS: accelerator_type: b200
2026-07-15 13:50:49|STATUS: client_host_memory_in_gb: 15.0
2026-07-15 13:50:49|STATUS: hosts: 127.0.0.1
2026-07-15 13:50:49|STATUS: exec_type: mpi
2026-07-15 13:50:49|STATUS: mpi_bin: mpirun
2026-07-15 13:50:49|STATUS: loops: 1
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:49|STATUS: --- Environment ---
2026-07-15 13:50:49|STATUS: MLPERF_RESULTS_DIR: [not set]
2026-07-15 13:50:49|STATUS: MLPERF_SYSTEMNAME: klpgpu
2026-07-15 13:50:49|STATUS: MPI_RUN_BIN: [not set]
2026-07-15 13:50:49|STATUS: MPI_EXEC_BIN: [not set]
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:49|STATUS: --- Object Storage (S3) ---
2026-07-15 13:50:49|STATUS: bucket: [not set]
2026-07-15 13:50:49|STATUS: storage_library: s3dlio
2026-07-15 13:50:49|STATUS: uri_scheme: s3
2026-07-15 13:50:49|STATUS: endpoint: [not set]
2026-07-15 13:50:49|STATUS: load_balance_strategy: round_robin
2026-07-15 13:50:49|STATUS: aws_region: us-east-1
2026-07-15 13:50:49|STATUS: aws_ca_bundle: [not set]
2026-07-15 13:50:49|STATUS: AWS_ACCESS_KEY_ID: [not set]
2026-07-15 13:50:49|STATUS: AWS_SECRET_ACCESS_KEY: [not set]
2026-07-15 13:50:49|STATUS:
2026-07-15 13:50:51|WARNING: Skipping environment validation (--skip-validation flag)
⠋ Capturing or verifying code image... 0:00:002026-07-15 13:50:51|STATUS: code image match found at /home/keithpij/results/keithpij/code-4be7cfbc
2026-07-15 13:50:51|STATUS: Benchmark results directory: /home/keithpij/results/open/keithpij/results/klpgpu/training/retinanet/run/20260715_135049
2026-07-15 13:50:51|INFO: Collector script staged at /home/keithpij/results/open/keithpij/results/klpgpu/training/retinanet/run/20260715_135049/collector-staging/mlps_collector.py (persisted as run artifact)
2026-07-15 13:50:51|INFO: Running MPI collection across 1 host(s)
2026-07-15 13:50:51|INFO: MPI collection completed successfully (1 hosts reported)
2026-07-15 13:50:52|INFO: --object mode: loading credentials from /home/keithpij/storage/.env
2026-07-15 13:50:52|INFO: --object mode: injected storage params (storage_type=s3, storage_root=mlcommons, library=s3dlio, uri_scheme=s3, force_path_style=True)
2026-07-15 13:50:52|INFO: skip_listing enabled: 1,170,301 train files → validation_interval=1,000 (~1,172 HEAD checks at startup)
2026-07-15 13:50:52|INFO: Created benchmark run: training_run_retinanet_20260715_135049
2026-07-15 13:50:52|STATUS: Verifying benchmark run for training_run_retinanet_20260715_135049
2026-07-15 13:50:52|RESULT: Minimum file count dictated by dataset size to memory size ratio.
2026-07-15 13:50:52|ERROR: INVALID: [INVALID] Disallowed parameter override: storage.storage_library = s3dlio (Parameter: Overrode Parameters, Expected: None, Actual: s3dlio)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_type = s3 (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_root = mlcommons (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_options.storage_library = s3dlio (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.storage_options.uri_scheme = s3 (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: storage.s3_force_path_style = true (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: dataset.skip_listing = True (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Closed: [CLOSED] Tool-injected parameter: dataset.listing_validation_interval = 1000 (Parameter: Tool-Injected Parameters)
2026-07-15 13:50:52|STATUS: Benchmark run is INVALID due to 1 issues ([RunID(program='training', command='run', model='retinanet', run_datetime='20260715_135049')])
2026-07-15 13:50:52|ERROR: Invalid configuration found. Aborting benchmark run.