ompio and MPI_MODE_SEQUENTIAL and NFS

[dshrader]

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

2.1.2 and 3.0.0

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

release tarball

Please describe the system on which you are running

• Operating system/version: RHEL 7
• Computer hardware: Intel Xeon CPUs
• Network type: Intel Omnipath

---

Details of the problem

When using MPI_MODE_SEQUENTIAL in MPI_FILE_OPEN with an NFS target and using ompio, we get the following error:

mca_sharedfp_lockedfile_file_open: Error during file open

Here is a reproducer written in fortran:

PROGRAM main
    use mpi

    integer ierr, i, myrank, BUFSIZE, thefile, c50, c22, c19, c16

    call MPI_INIT(ierr)
    call MPI_COMM_RANK(MPI_COMM_WORLD, myrank, ierr)

    call MPI_FILE_OPEN(MPI_COMM_WORLD, 'testfile', &
                       MPI_MODE_WRONLY + MPI_MODE_CREATE + MPI_MODE_SEQUENTIAL, &
                       MPI_INFO_NULL, thefile, ierr)

    call MPI_FILE_CLOSE(thefile, ierr)
    call MPI_FINALIZE(ierr)
END PROGRAM main

If the above program is compiled as a.out, here is the output from trying to run it while in a NFS directory:

shell$ mpirun --map-by ppr:1:node ./a.out
[001.localdomain:151509] mca_sharedfp_lockedfile_file_open: Error during file open
[002.localdomain:36249] mca_sharedfp_lockedfile_file_open: Error during file open
shell$

If romio is used instead, the program runs fine:

shell$ mpirun --map-by ppr:1:node -mca io romio314 ./a.out
shell$

If MPI_MODE_SEQUENTIAL is removed from the source, the program runs without an error with both ompio and romio. Running out of lustre also works for both ompio and romio.

[dshrader]

I was mistaken when I said that the reproducer ran with ompio with lustre. It turns out that the runtime chose romio over ompio in that case. When specifying ompio, it fails (the current working directory is in a lustre mount):

shell$ mpirun --map-by ppr:1:node -mca io ompio ./a.out 
App launch reported: 3 (out of 3) daemons - 1 (out of 2) procs
[001.localdomain:176690] mca_sharedfp_lockedfile_file_open: Error during file open
[002.localdomain:59194] mca_sharedfp_lockedfile_file_open: Error during file open
shell$ mpirun --map-by ppr:1:node -mca io romio314 ./a.out 
App launch reported: 3 (out of 3) daemons - 1 (out of 2) procs

Using '--mca io_base_verbose 100' without the '--mca io [ompio|romio314]' command line parameter, I can see that romio314 is selected over ompio.

[edgargabriel]

Thank you for the bug report, I will look into that. I am a bit confused on the usage of MPI_MODE_SEQUENTIAL to be honest, would it be possible to extend the example by at least one additional functional call (write) to see how you are using the file afterwards in your code?

[edgargabriel]

to maybe extend on my previous message, I understand what is going on, for the fix it would be good to have an understanding on how the file is subsequently used. Are you using shared file pointers operations afterwards?

[dshrader]

Here is a version of the reproducer that does something with the file, but it isn't too complicated. It was an earlier version of the reproducer.

PROGRAM main
    use mpi

    integer ierr, i, myrank, BUFSIZE, thefile, c50, c22, c19, c16
    parameter (BUFSIZE=100) 
    integer buf(BUFSIZE) 
    integer(kind=MPI_OFFSET_KIND) disp     

    call MPI_INIT(ierr)
    call MPI_COMM_RANK(MPI_COMM_WORLD, myrank, ierr)

    do i = 0, BUFSIZE 
        buf(i) = myrank * BUFSIZE + i 
    enddo 
    call MPI_FILE_OPEN(MPI_COMM_WORLD, 'testfile', &
                       MPI_MODE_WRONLY + MPI_MODE_CREATE + MPI_MODE_SEQUENTIAL, &
                       MPI_INFO_NULL, thefile, ierr)
    call MPI_FILE_WRITE_ORDERED(thefile,buf,BUFSIZE,MPI_INTEGER, &
                        MPI_STATUS_IGNORE, ierr)

    call MPI_FILE_CLOSE(thefile, ierr)
    call MPI_FINALIZE(ierr)

END PROGRAM main

[dshrader]

Thank you for taking a look, @edgargabriel!

[edgargabriel]

@dshrader thanks, this is a big help!

[edgargabriel]

I can confirm that with pr 4997 the code executes correctly.