New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdms2 netcdf use parallel flag fails #1593

Closed
mcenerney1 opened this Issue Oct 7, 2015 · 16 comments

Comments

Projects
None yet
3 participants
@mcenerney1
Contributor

mcenerney1 commented Oct 7, 2015

cdms2 has a flag that turns off parallel IO. I have evidence that it fails.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Oct 7, 2015

@mcenerney1 can you please put a simple script for us to reproduce?

@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 7, 2015

I did try to make one but failed. I'll continue. But, Denis looked at the code and verified the flag is unused.

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Oct 7, 2015

Here is a script that failed

import mpi4py
import cdms2

sz = mpi4py.MPI.COMM_WORLD.Get_size()
rk = mpi4py.MPI.COMM_WORLD.Get_rank()

# All flags are set to OFF for parallel writing
# ----------------------------------------------

cdms2.setNetcdfClassicFlag(0)
cdms2.setNetcdfShuffleFlag(0)
cdms2.setNetcdfDeflateFlag(0)
cdms2.setNetcdfDeflateLevelFlag(0)
cdms2.setNetcdfUseParallelFlag(0)


if rk == 0:
    print rk
    f=cdms2.open("test_mpi_write_2.nc")
    f.close
``
@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Oct 7, 2015

Here is my solution which works...

in Cdunifmodule.c we need to check for the cdms_use_parallel before opening a file with MPIIO.

#ifdef PARALLEL
      int ierr;
      int ncid;

      if(cdms_use_parallel == 1) {
          ierr = nc_open_par(controlpath,mode|NC_MPIIO,MPI_COMM_WORLD,MPI_INFO_NULL,&ncid);
      }
      if((cdms_use_parallel == 0) || (ierr != NC_NOERR )) {
        ierr = nc_open(controlpath,mode,&ncid);
      }
      return ncid;
#else
      ret
@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 7, 2015

do you mean ncmode not mode? also controlpath is undefined.

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Oct 7, 2015

mode is passed as a parameter and already contains NC_WRITE or NC_NOWRITE. Charles is creating a patch and will add this to 2.4.

@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 7, 2015

Thanks. I modified the wrong part of the code. Is it supposed to be cdopen?

@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 7, 2015

Mea Culpa!

@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 8, 2015

This works beautifully!

@mcenerney1 mcenerney1 closed this Oct 8, 2015

@mcenerney1 mcenerney1 reopened this Oct 8, 2015

@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 8, 2015

It worked for 3 processors, but not 4. The symptoms are the same as before. I suspect the fix needs to be somewhere else also.

@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 8, 2015

It now works for any number of processors for now. I lost the following mods to cdscan which must
be included in cdscan.
if name == 'main':
main(sys.argv)
from mpi4py import MPI
comm = MPI.Comm.Get_parent()
comm.send('cdscan complete')

@mcenerney1 mcenerney1 closed this Oct 8, 2015

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Oct 8, 2015

@mcenerney1 since it's not in master, let's not close it yet

@doutriaux1 doutriaux1 reopened this Oct 8, 2015

@doutriaux1 doutriaux1 added this to the 2.4 milestone Oct 8, 2015

@doutriaux1 doutriaux1 added the cdms2 label Oct 8, 2015

@mcenerney1

This comment has been minimized.

Contributor

mcenerney1 commented Oct 8, 2015

OK. We have to resolve the cdms2.setNetcdfUseParallelFlag flag in cdscan.
I confirmed that it is set to 1. For now it has no effect on the diagnostics.
But it may in the future.

@dnadeau4

This comment has been minimized.

Contributor

dnadeau4 commented Oct 8, 2015

I created a pull request to set cdms2.setNetcdfUseParallelFlag(0) in cdscan.

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 4, 2016

@dnadeau4 is this in?

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jan 4, 2016

looks like yes. @dnadeau4 please reopen if not

@doutriaux1 doutriaux1 closed this Jan 4, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment