Performance loss with new base image?

[fmaussion]

For #814, I switched to a new base image.

The performance loss is clearly visible, and for some tests so important that the tests now time out. See the time differences between these two:

• old image: https://travis-ci.org/OGGM/oggm/builds/567785364
• new image: https://travis-ci.org/OGGM/oggm/builds/567774052

@TimoRoth do you have an idea how to test this?

Related question: what environment do we use on asv? https://cluster.klima.uni-bremen.de/~github/asv/#/ - if we don't use our docker images yet, we might consider using them if possible.

[fmaussion]

Note that this is maybe not super important - py37 seems largely unaffected.

[TimoRoth]

asv uses conda using one of the generated environment files:
https://github.com/OGGM/oggm/blob/master/asv_oggm_plugin.py
It's been pinned to an older version for a while now, so might need an update.
ASV does not support docker. It insists on installing an environment via pip or conda on its own, since it wants to record package version in its benchmark results and has options to alter single dependencies versions.

I'm not sure why the new container would be so much slower though. It contains a lot of updates, last time we had random slowdown it was an issue with newer netCDF4 versions.
It makes some tests freeze for 10 minutes without any output even. Both on py36 and 37, so both are affected.

[fmaussion]

So if wee look at the previous image (June), the tests time look like this:

In py37, they look like this:

An with the latest images, like this:

[TimoRoth]

For the record, here is the output of pip3 list --format=freeze for the two base images.

20190613

affine==2.2.2
atomicwrites==1.3.0
attrs==19.1.0
Cartopy==0.17.0
certifi==2019.3.9
cftime==1.0.3.4
chardet==3.0.4
Click==7.0
click-plugins==1.1.1
cligj==0.5.0
configobj==5.0.6
coverage==4.5.3
cycler==0.10.0
Cython==0.29.10
decorator==4.4.0
descartes==1.1.0
filelock==3.0.12
Fiona==1.8.6
GDAL==2.2.4
geopandas==0.5.0
idna==2.8
ilock==1.0.2
imageio==2.5.0
importlib-metadata==0.18
joblib==0.13.2
kiwisolver==1.1.0
matplotlib==3.1.0
more-itertools==7.0.0
motionless==1.3.2
munch==2.3.2
netCDF4==1.5.1.2
networkx==2.3
nose==1.3.7
numpy==1.16.4
packaging==19.0
pandas==0.24.2
Pillow==6.0.0
pip==19.1.1
pluggy==0.12.0
portalocker==1.4.0
progressbar2==3.42.0
py==1.8.0
pyparsing==2.4.0
pyproj==1.9.6
pyshp==2.1.0
pytest==4.6.3
pytest-cov==2.7.1
pytest-mpl==0.111.oggm
python-dateutil==2.8.0
python-utils==2.3.0
pytz==2019.1
PyWavelets==1.0.3
rasterio==1.0.24
requests==2.22.0
salem==0.2.4-3-gab27b54
scikit-image==0.15.0
scipy==1.3.0
setuptools==41.0.1
Shapely==1.6.4.post2
six==1.12.0
snuggs==1.4.6
tifffile==2019.5.30
urllib3==1.25.3
versioneer==0.18
virtualenv==16.6.0
wcwidth==0.1.7
wheel==0.33.4
xarray==0.12.1
zipp==0.5.1

20190802

affine==2.2.2
atomicwrites==1.3.0
attrs==19.1.0
Cartopy==0.17.0
certifi==2019.6.16
cftime==1.0.3.4
chardet==3.0.4
Click==7.0
click-plugins==1.1.1
cligj==0.5.0
cloudpickle==1.2.1
configobj==5.0.6
coverage==4.5.4
cycler==0.10.0
Cython==0.29.13
dask==2.2.0
decorator==4.4.0
descartes==1.1.0
distributed==2.2.0
filelock==3.0.12
Fiona==1.8.6
GDAL==2.2.4
geopandas==0.5.1
HeapDict==1.0.0
idna==2.8
ilock==1.0.2
imageio==2.5.0
importlib-metadata==0.19
joblib==0.13.2
kiwisolver==1.1.0
matplotlib==3.0.3
more-itertools==7.2.0
motionless==1.3.2
msgpack==0.6.1
munch==2.3.2
netCDF4==1.5.1.2
networkx==2.3
nose==1.3.7
numpy==1.17.0
packaging==19.1
pandas==0.25.0
Pillow==6.1.0
pip==19.2.1
pluggy==0.12.0
portalocker==1.5.0
progressbar2==3.42.0
psutil==5.6.3
py==1.8.0
pyparsing==2.4.2
pyproj==2.2.1
pyshp==2.1.0
pytest==5.0.1
pytest-cov==2.7.1
pytest-mpl==0.111.oggm
python-dateutil==2.8.0
python-utils==2.3.0
pytz==2019.2
PyWavelets==1.0.3
PyYAML==5.1.2
rasterio==1.0.24
requests==2.22.0
salem==0.2.4-6-g69666ac
scikit-image==0.15.0
scipy==1.3.0
setuptools==41.0.1
Shapely==1.6.4.post2
six==1.12.0
snuggs==1.4.6
sortedcontainers==2.1.0
tblib==1.4.0
tifffile==2019.7.26
toolz==0.10.0
tornado==6.0.3
urllib3==1.25.3
versioneer==0.18
virtualenv==16.7.2
wcwidth==0.1.7
wheel==0.33.4
xarray==0.12.3
zict==1.0.0
zipp==0.5.2

The diff between them

 atomicwrites==1.3.0
 attrs==19.1.0
 Cartopy==0.17.0
-certifi==2019.3.9
+certifi==2019.6.16
 cftime==1.0.3.4
 chardet==3.0.4
 Click==7.0
 click-plugins==1.1.1
 cligj==0.5.0
+cloudpickle==1.2.1
 configobj==5.0.6
-coverage==4.5.3
+coverage==4.5.4
 cycler==0.10.0
-Cython==0.29.10
+Cython==0.29.13
+dask==2.2.0
 decorator==4.4.0
 descartes==1.1.0
+distributed==2.2.0
 filelock==3.0.12
 Fiona==1.8.6
 GDAL==2.2.4
-geopandas==0.5.0
+geopandas==0.5.1
+HeapDict==1.0.0
 idna==2.8
 ilock==1.0.2
 imageio==2.5.0
-importlib-metadata==0.18
+importlib-metadata==0.19
 joblib==0.13.2
 kiwisolver==1.1.0
-matplotlib==3.1.0
-more-itertools==7.0.0
+matplotlib==3.0.3
+more-itertools==7.2.0
 motionless==1.3.2
+msgpack==0.6.1
 munch==2.3.2
 netCDF4==1.5.1.2
 networkx==2.3
 nose==1.3.7
-numpy==1.16.4
-packaging==19.0
-pandas==0.24.2
-Pillow==6.0.0
-pip==19.1.1
+numpy==1.17.0
+packaging==19.1
+pandas==0.25.0
+Pillow==6.1.0
+pip==19.2.1
 pluggy==0.12.0
-portalocker==1.4.0
+portalocker==1.5.0
 progressbar2==3.42.0
+psutil==5.6.3
 py==1.8.0
-pyparsing==2.4.0
-pyproj==1.9.6
+pyparsing==2.4.2
+pyproj==2.2.1
 pyshp==2.1.0
-pytest==4.6.3
+pytest==5.0.1
 pytest-cov==2.7.1
 pytest-mpl==0.111.oggm
 python-dateutil==2.8.0
 python-utils==2.3.0
-pytz==2019.1
+pytz==2019.2
 PyWavelets==1.0.3
+PyYAML==5.1.2
 rasterio==1.0.24
 requests==2.22.0
-salem==0.2.4-3-gab27b54
+salem==0.2.4-6-g69666ac
 scikit-image==0.15.0
 scipy==1.3.0
 setuptools==41.0.1
 Shapely==1.6.4.post2
 six==1.12.0
 snuggs==1.4.6
-tifffile==2019.5.30
+sortedcontainers==2.1.0
+tblib==1.4.0
+tifffile==2019.7.26
+toolz==0.10.0
+tornado==6.0.3
 urllib3==1.25.3
 versioneer==0.18
-virtualenv==16.6.0
+virtualenv==16.7.2
 wcwidth==0.1.7
 wheel==0.33.4
-xarray==0.12.1
-zipp==0.5.1
+xarray==0.12.3
+zict==1.0.0
+zipp==0.5.2

[TimoRoth]

Interesting how the June Image has matplotlib 3.1.0, but was running our tests just fine for months.

[fmaussion]

xarray and numpy are good candidates for performance changes. But in order to know this we would need more systematic tests, e.g. with asv (our benchmarks are not yet complete enough to ensure a good performance coverage, though... should work on this...)

[TimoRoth]

numpy was the first thing I just tested, and it does not appear to change anything to downgrade it back to the old version: https://travis-ci.org/TimoRoth/oggm/builds/567823025
Gonna test xarray next.

Edit: looking at it some more, it does seem to have some impact, but it's not back to the old numbers.

[fmaussion]

This is now definitely attributed to numpy: numpy/numpy#14281

Closed in favor of #855