Memory error with MOC analysis on edison compute node

[xylar]

I'm running ACME PreAndPostProcessingsScripts/xylar/update_mpas_analysis_v0.2 on Edison by submitting a batch scripts. I get this error:

Plotting streamfunction of Meridional Overturning Circulation (MOC)...

  List of files for climatologies:
/scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850.ne30_oEC.edison/run/mpaso.hist.am.timeSeriesStatsMonthly.0030-12-01.nc through /scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850.ne30_oEC.edison/run/mpaso.hist.am.timeSeriesStatsMonthly.0041-01-01.nc

  List of files for timeSeries:
/scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850.ne30_oEC.edison/run/mpaso.hist.am.timeSeriesStatsMonthly.0030-12-01.nc through /scratch2/scratchdirs/golaz/ACME_simulations/20161117.beta0.A_WCYCL1850.ne30_oEC.edison/run/mpaso.hist.am.timeSeriesStatsMonthly.0041-01-01.nc

  Reading region and transect mask for Atlantic...

  Compute and/or plot post-processed MOC climatological streamfunction...
   Load data...
Traceback (most recent call last):
  File "python/MPAS-Analysis/run_analysis.py", line 243, in <module>
    analysis(config)
  File "python/MPAS-Analysis/run_analysis.py", line 182, in analysis
    moc_streamfunction(config)
  File "/global/u2/x/xylar/acme_work/analysis/xylar/update_MPAS_Analysis_v0.2/coupled_diags/python/MPAS-Analysis/mpas_analysis/ocean/meridional_overturning_circulation.py", line 134, in moc_streamfunction
    regionNames, dictClimo)
  File "/global/u2/x/xylar/acme_work/analysis/xylar/update_MPAS_Analysis_v0.2/coupled_diags/python/MPAS-Analysis/mpas_analysis/ocean/meridional_overturning_circulation.py", line 294, in _compute_moc_climo_postprocess
    horizontalVel = annualClimatology.avgNormalVelocity.values
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/xarray/core/dataarray.py", line 384, in values
    return self.variable.values
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/xarray/core/variable.py", line 309, in values
    return _as_array_or_item(self._data_cached())
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/xarray/core/variable.py", line 275, in _data_cached
    self._data = np.asarray(self._data)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/numpy/core/numeric.py", line 531, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/array/core.py", line 1045, in __array__
    x = self.compute()
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/base.py", line 94, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/base.py", line 201, in compute
    results = get(dsk, keys, **kwargs)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/threaded.py", line 76, in get
    **kwargs)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/async.py", line 500, in get_async
    raise(remote_exception(res, tb))
dask.async.MemoryError: 

Traceback
---------
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/async.py", line 266, in execute_task
    result = _execute_task(task, data)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/async.py", line 247, in _execute_task
    return func(*args2)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/toolz/functoolz.py", line 468, in __call__
    ret = f(ret)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/dask/array/reductions.py", line 229, in mean_combine
    total = sum(pair['total'], **kwargs)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/numpy/lib/nanfunctions.py", line 538, in nansum
    return np.sum(a, axis=axis, dtype=dtype, out=out, keepdims=keepdims)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 1814, in sum
    out=out, **kwargs)
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/numpy/core/_methods.py", line 32, in _sum
    return umr_sum(a, axis, dtype, out, keepdims)

It seems like MOC computations are too big for a compute node!!! We need to figure out what's going on...

[pwolfram]

@xylar, this is just for the serial MOC, right? Can you please advise me on how to reproduce this so that I can help take a closer look?

[pwolfram]

It is possible that we may need to be explicit about chunking as we move to higher and higher resolutions, where chunk sizes are specified on tasks being run on a compute node as well as the available memory on the compute node.

[xylar]

@pwolfram, I hadn't had a chance to describe the setup in detail. Here is what you should be able to do to get this error:

• check out PreAndPostProcessingScripts/xylar/update_MPAS_Analysis_v0.2
• copy run_EDISON.csh and run_EDISON.sl to whatever name you would typically use.
• In your custom version of run_EDISON.csh, set output_base_dir to your favorite scratch directory and turn atmosphere diagnostics off (setenv generate_atm_diags 0). You can turn off other things, too, for efficiency if you want.
• Edit your custom version of run_EDISON.sl to point to your custom version of run_EDISON.csh.
• sbatch run_EDISON_pwolfram.sl (or whatever).
  If the queue is being nice, you could try an interactive job instead:

salloc -N 1 -p debug -L SCRATCH,project
....
srun -n 1 ./run_EDISON_pwolfram.csh

I agree that chunking may be the solution. My guess is that the problem is in loading the MOC data files (3D velocity data) because that's where I had memory problems on my laptop as well. It doesn't seem to realize that loading 120 3D data sets is too much at once...

[xylar]

It is possible that we may need to be explicit about chunking as we move to higher and higher resolutions, where chunk sizes are specified on tasks being run on a compute node as well as the available memory on the compute node.

@pwolfram, we also need to be careful on the login nodes. As I said previously, I got in trouble at NERSC for overloading one of the login nodes by running 3 analysis jobs at once. That's why I've switched to the compute nodes.

I'm a little unclear how we are going to know how to chunk a given data set. If an analysis task supplies open_multifile_dataset with a list of variables to read in (or, worse yet, they want all variables), how do we know whether this is 2D data (and we can probably handle it) or 3D data and we should definitely chunk it in time (and maybe in space, too!)?

[pwolfram]

Thank you @xylar, I'll take a look at this issue.

[xylar]

@pwolfram, you requested an update version of xarray. I can't do that in the shared anaconda repository because it would break the current ACME analysis. But I made a new package you can try instead:

module load python/anaconda-2.7-climate_test

Just modify the appropriate line of your custom run_EDISON.csh.

[pwolfram]

@xylar, the plan for the merge with ACME is to use the soon to be released 0.9.1, right? This would imply I can use features from xarray 0.9.0 in the fix here. If not, I would say this is something we need to discuss. Thanks!

[xylar]

@pwolfram, 0.9.1 has already been released (at least on conda-forge) and I've been using it most places already. I was assuming there would be another sub-release (0.9.2?) with your recent fix.

Yes, you should be able to use features from 0.9.0.

[pwolfram]

Thanks @xylar, I decremented the version number. You are 100% correct. The next release that is needed is 0.9.2.