Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdms2 netcdf use parallel flag fails #1593

Closed
mcenerney1 opened this issue Oct 7, 2015 · 16 comments
Closed

cdms2 netcdf use parallel flag fails #1593

mcenerney1 opened this issue Oct 7, 2015 · 16 comments
Assignees
Labels
Milestone

Comments

@mcenerney1
Copy link
Contributor

@mcenerney1 mcenerney1 commented Oct 7, 2015

cdms2 has a flag that turns off parallel IO. I have evidence that it fails.

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 7, 2015

@mcenerney1 can you please put a simple script for us to reproduce?

@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 7, 2015

I did try to make one but failed. I'll continue. But, Denis looked at the code and verified the flag is unused.

@dnadeau4
Copy link
Contributor

@dnadeau4 dnadeau4 commented Oct 7, 2015

Here is a script that failed

import mpi4py
import cdms2

sz = mpi4py.MPI.COMM_WORLD.Get_size()
rk = mpi4py.MPI.COMM_WORLD.Get_rank()

# All flags are set to OFF for parallel writing
# ----------------------------------------------

cdms2.setNetcdfClassicFlag(0)
cdms2.setNetcdfShuffleFlag(0)
cdms2.setNetcdfDeflateFlag(0)
cdms2.setNetcdfDeflateLevelFlag(0)
cdms2.setNetcdfUseParallelFlag(0)


if rk == 0:
    print rk
    f=cdms2.open("test_mpi_write_2.nc")
    f.close
``

@dnadeau4
Copy link
Contributor

@dnadeau4 dnadeau4 commented Oct 7, 2015

Here is my solution which works...

in Cdunifmodule.c we need to check for the cdms_use_parallel before opening a file with MPIIO.

#ifdef PARALLEL
      int ierr;
      int ncid;

      if(cdms_use_parallel == 1) {
          ierr = nc_open_par(controlpath,mode|NC_MPIIO,MPI_COMM_WORLD,MPI_INFO_NULL,&ncid);
      }
      if((cdms_use_parallel == 0) || (ierr != NC_NOERR )) {
        ierr = nc_open(controlpath,mode,&ncid);
      }
      return ncid;
#else
      ret

@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 7, 2015

do you mean ncmode not mode? also controlpath is undefined.

@dnadeau4
Copy link
Contributor

@dnadeau4 dnadeau4 commented Oct 7, 2015

mode is passed as a parameter and already contains NC_WRITE or NC_NOWRITE. Charles is creating a patch and will add this to 2.4.

@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 7, 2015

Thanks. I modified the wrong part of the code. Is it supposed to be cdopen?

@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 7, 2015

Mea Culpa!

@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 8, 2015

This works beautifully!

@mcenerney1 mcenerney1 closed this Oct 8, 2015
@mcenerney1 mcenerney1 reopened this Oct 8, 2015
@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 8, 2015

It worked for 3 processors, but not 4. The symptoms are the same as before. I suspect the fix needs to be somewhere else also.

@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 8, 2015

It now works for any number of processors for now. I lost the following mods to cdscan which must
be included in cdscan.
if name == 'main':
main(sys.argv)
from mpi4py import MPI
comm = MPI.Comm.Get_parent()
comm.send('cdscan complete')

@mcenerney1 mcenerney1 closed this Oct 8, 2015
@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 8, 2015

@mcenerney1 since it's not in master, let's not close it yet

@doutriaux1 doutriaux1 reopened this Oct 8, 2015
@doutriaux1 doutriaux1 added this to the 2.4 milestone Oct 8, 2015
@mcenerney1
Copy link
Contributor Author

@mcenerney1 mcenerney1 commented Oct 8, 2015

OK. We have to resolve the cdms2.setNetcdfUseParallelFlag flag in cdscan.
I confirmed that it is set to 1. For now it has no effect on the diagnostics.
But it may in the future.

@dnadeau4
Copy link
Contributor

@dnadeau4 dnadeau4 commented Oct 8, 2015

I created a pull request to set cdms2.setNetcdfUseParallelFlag(0) in cdscan.

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Jan 4, 2016

@dnadeau4 is this in?

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Jan 4, 2016

looks like yes. @dnadeau4 please reopen if not

@doutriaux1 doutriaux1 closed this Jan 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants