orte_init failed for some reason

[momonara]

Hi.

I ran the mlpstorage training, and got some errors

/root/storage/mlpstorage training datasize --model retinanet --client-host-memory-in-gb 250 --num-client-hosts 1 --max-accelerators 4 --accelerator-type b200 --file --allow-run-as-root
/root/storage/mlpstorage training datagen --hosts 127.0.0.1 --num-processes 4 --model retinanet --data-dir /home/nvme/retinanet_data --results-dir /home/mlp_results/retinanet_results  --param dataset.num_files_train=4155900 --file --allow-run-as-root
/root/storage/mlpstorage training run --hosts 127.0.0.1 --num-client-hosts 1 --client-host-memory-in-gb 250 --num-accelerators 4 --accelerator-type b200 --model retinanet  --data-dir /home/nvme/retinanet_data --results-dir /home/mlp_results/retinanet_results --param dataset.num_files_train=4155900 reader.read_threads=4 --file --allow-run-as-root

[OUTPUT] 2026-05-12T13:15:08.757087 Running DLIO [Training] with 4 process(es)
[OUTPUT] 2026-05-12T13:15:16.778433 Max steps per epoch: 43290 = 1 * 4155900 / 24 / 4 (samples per file * num files / batch size / comm size)
[OUTPUT] 2026-05-12T13:15:39.206214 Starting epoch 1: 43290 steps expected
[OUTPUT] 2026-05-12T13:15:39.206664 Starting block 1
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  getting local rank failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost.localdomain:4185562] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to
guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost.localdomain:4185561] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to
guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost.localdomain:4185563] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to
guarantee that all other processes were killed!
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[localhost.localdomain:4185564] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to
guarantee that all other processes were killed!
[localhost.localdomain:4188348] 3 more processes have sent help message help-orte-runtime.txt / orte_init:startup:internal-failure

[momonara]

Hi.

I've tried some tests, and found out that the error occurs from "commit 39e657d - Bug fixes and performance enhancements: object storage, checkpointing, Parquet loading (#359)"

When I run the same command (/root/storage/mlpstorage training run~~) with "commit ee70bf3", it works.
Below command, which is from "commit 39e657d" also works.

mpirun -n 4 -host 127.0.0.1:4 --bind-to none --map-by socket --allow-run-as-root /root/storage/.venv/bin/dlio_benchmark workload=retinanet_b200 ++hydra.run.dir=/home/mlp_results/retinanet_results/training/retinanet/run/20260514_094047 ++hydra.output_subdir=dlio_config ++workload.dataset.num_files_train=100 ++workload.reader.read_threads=4 ++workload.dataset.data_folder=/home/nvme/retinanet_data/retinanet --config-dir=/root/storage/configs/dlio

So I think some changes in uv.lock cause this error.

[russfellows]

I just PR’d a fix into my fork of MLCommons storage. The fix now enables setting the MPI method. I’ll work on trying to get my fork merged into MLCommons tomorrow, but in the meantime feel free to test my fork: https://github.com/russfellows/mlc-storage Regards, —Russ

…

On May 13, 2026, at 7:05 PM, momonara ***@***.***> wrote: momonara left a comment (mlcommons/storage#369) <#369 (comment)> Hi. I've tried some tests, and found out that the error occurs from "commit 39e657d <39e657d> - Bug fixes and performance enhancements: object storage, checkpointing, Parquet loading (#359 <#359>)" When I run the same command (/root/storage/mlpstorage training run~~) with "commit ee70bf3 <ee70bf3>", it works. Below command, which is from "commit 39e657d <39e657d>" also works. mpirun -n 4 -host 127.0.0.1:4 --bind-to none --map-by socket --allow-run-as-root /root/storage/.venv/bin/dlio_benchmark workload=retinanet_b200 ++hydra.run.dir=/home/mlp_results/retinanet_results/training/retinanet/run/20260514_094047 ++hydra.output_subdir=dlio_config ++workload.dataset.num_files_train=100 ++workload.reader.read_threads=4 ++workload.dataset.data_folder=/home/nvme/retinanet_data/retinanet --config-dir=/root/storage/configs/dlio So I think some changes in uv.lock cause this error. — Reply to this email directly, view it on GitHub <#369 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF64UJ7TBCXIJ6CAK4TDGKL42ULWTAVCNFSM6AAAAACY2D2ODGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DINBWGQZTMNZQGM>. You are receiving this because you were assigned.