Extreme CPU load ~3100% during variable read from netcdf

[durack1]

I have noticed some extreme CPU usage during variable read from a netcdf file, an example below:

import cdms2 as cdm
f = '/work/durack1/Shared/obs_data/Argo/UCSD/180719_UCSD_monthly_TAndS_200401-201806_2p5-1975db.nc'
fH = cdm.open(f)
testAxes = fH('to')

And from top:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
103217 durack1   20   0 7727m 5.1g  29m R 3100.0  2.3   7:48.84 anaconda2/envs/cdat80py2/bin/python -m spyder_kernels.console

Conda env info:

(cdat80py2) duro@ocean:[8800]> ls anaconda2/envs/cdat80py2/conda-meta/cdtime*
anaconda2/envs/cdat80py2/conda-meta/cdtime-3.0-py27h6091dcd_1.json
(cdat80py2) duro@ocean:[8801]> ls anaconda2/envs/cdat80py2/conda-meta/cdms2*
anaconda2/envs/cdat80py2/conda-meta/cdms2-3.0.1-py27h6091dcd_1.json

@dnadeau4 @zshaheen @doutriaux1

[zshaheen]

@durack1 Thanks for confirming what I've been seeing. I haven't tested 3.0.1 yet, but I'll try to confirm what you have.

In the meantime, @dnadeau4 and @doutriaux1 we need a test for this. It should query ps or top and check the resources used. That's not hard, but I have no idea how that would be ran in the run_tests.py that cdms is using. Cause I think that uses multiprocessing in and of itself.

[durack1]

@zshaheen no problem, I've seen ~1200, ~1500 but the new record of 3100% today was too much not to list this issue.. Let me know if you want me to dig a little deeper into the libraries that comprise the cdat80py2 environment, it was created 1st August 2018 14:22 PDT for FYI

[dnadeau4]

@zshaheen thanks for doing this.

[zshaheen]

@durack1 @doutriaux1 @dnadeau4

I have horrible news. I just tested 3.0.1 of cdms and I'm getting the issue. I followed the instructions I had in #248.

With cdms like this, even running e3sm_diags with 4 processes spawns so many children processes that it can't even complete. The 12+ users of our (@chengzhuzhang and I) software are having similar issues.

[durack1]

@zshaheen same hash as the build I am using above?

[zshaheen]

@durack1 Yes:

(e3sm_diags_env_1.3.5a) shaheen2@acme1:~/github/acme_diags/tests/system$ ls ~/anaconda2/envs/e3sm_diags_env_1.3.5a/conda-meta/cdms2*
/export/shaheen2/anaconda2/envs/e3sm_diags_env_1.3.5a/conda-meta/cdms2-3.0.1-py27h6091dcd_1.json
(e3sm_diags_env_1.3.5a) shaheen2@acme1:~/github/acme_diags/tests/system$ ls ~/anaconda2/envs/e3sm_diags_env_1.3.5a/conda-meta/cdtime*
/export/shaheen2/anaconda2/envs/e3sm_diags_env_1.3.5a/conda-meta/cdtime-3.0-py27h6091dcd_1.json

[durack1]

Maybe a rebuild with careful compiler and library selection will fix things? I believe most machines I’ve seen issues are RHEL 6.10

[durack1]

I haven’t checked Mac

[dnadeau4]

@zshaheen do you have a test?

[dnadeau4]

@durack1 this file does not exist.

'/work/durack1/Shared/obs_data/Argo/UCSD/180719_UCSD_monthly_TAndS_200401-201806_2p5-1975db.nc'

[zshaheen]

@dnadeau4 not an automated test, but you can manually follow these instructions. This should only take 1 min or so.