what is the ROMIO bug on polaris?

[roblatham00]

I, the ROMIO maintainer, was surprised to read https://west-code.org/doc/West/v5.4.0/installations/polaris.html and find that there's a bug in "ROMIO, Cray MPICH"

Let's track this down and fix it so you can use the Lustre optimizations.

[vyu16]

Thank you @roblatham00

We encountered a divide-by-zero error on multiple Cray systems when calling MPI_File_write_at_all. We were advised by our HPC support team to use the workaround export ROMIO_FSTYPE_FORCE="ufs:", and then decided to set this environment variable whenever Cray MPICH is used. I'm testing whether this is still necessary and will report back.

[roblatham00]

OK, thanks. That might be a bug I fixed in ROMIO a decade ago. Our HPE/Cray friends make their own modifications to ROMIO. Let me know if its still a problem and I can harass them to fix it.

[vyu16]

Thanks for maintaining ROMIO!

The test case that used to trigger this problem went through successfully. I just realized that a rewriting of the relevant piece of code in August 2024 removed all calls to MPI_File_write_at_all, therefore we shouldn't have to set ROMIO_FSTYPE_FORCE="ufs:" anymore. I'm taking it away from my job scripts and if there're indeed no problems I'll update our documentation.

[roblatham00]

Oh... did you move away from collective I/O? it is a more complicated path as you've seen, but it can greatly enhance performance.

[vyu16]

Indeed, we first moved to collective I/O for performance, then we redesigned the flow of the code such that no I/O is needed in that part, which leads to even better performance 😄