mfo_open_files error in tavg::outfld()

[ajariwala611]

Hello,

I am using NEKRS v23 for boundary layer flow simulations on TACC lonestar6 systems. I am calling time averaging routines in UDF_ExecuteStep as tavg::outfld(1,1). After successfully outputting the avg*** file the code crashes with unknown error in mfo_open_file while outputting rms*** field file. This only happen on lonestar6 while it works fine on stampede3. I asked the admin and they told me lonestar6 scratch directory is BeeGFS file system while stampede3 uses VAST system. I am not sure if this has to do with underlying bug in the code somewhere or more technical hardware configuration issue.

Error message right before the crash:

        0  2.4881E+02 Write checkpoint
       FILE:
 avgflat_plate0.f00001                                                          
                                                      
 min/max:   0.0000       20.000       0.0000       5.0000      -5.0000       5.0000    
 min/max: -0.88254       1.3307     -0.79897       1.0981     -0.98974      0.94205    
 min/max: -0.55418       1.0467    
 min/max:  -11034.       8398.7    

        0  2.4881E+02 done :: Write checkpoint
                              file size =      4.2    GB
                              avg data-throughput =     0.9GB/s
                              io-nodes =    12


        0  2.4881E+02 Write checkpoint
       FILE:
 rmsflat_plate0.f00002                                                          
                                                      
ERROR: Error opening file in mfo_open_files.   ierr=*** 
  
an error occured: dying ...

My question is has anyone encountered similar file I/O errors with NEKRS on BeeGFS (Lonestar6)? Any suggestions for diagnosing whether this is a NEKRS code issue or a hardware/filesystem configuration problem with BeeGFS would be greatly appreciated!

Thank you for your help.