proxy: proxy_read_timeout config should be configurable #123
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have a performance problem with the production deployment at Ouranos so we need a longer timeout. Being a Ouranos specific need, it should not be harcoded. Timeout in Prod: ``` WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS _URL=https://pavics.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_datas et.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints HTTPError: 504 Server Error: Gateway Time-out for url: https://pavics.ouranos.ca/twitcher/ows/proxy/raven/wps ===================================================== 11 failed, 4 passed, 1 warning in 249.80s (0:04:09) =========================================== ``` Pass easily on my test VM with very modest hardware (10G ram, 2 cpu): ``` WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/raven/wps FINCH_WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/finch/wp s FLYINGPIGEON_WPS_URL=https://lvupavicsmaster.ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Runni ng_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints =========================================================== 15 passed, 1 warning in 33.84s =========================================================== ``` For comparison, a run on Prod without Twitcher: ``` WPS_URL=https://pavics.ouranos.ca/raven/wps FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://pavics .ouranos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize -with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints HTTPError: 504 Server Error: Gateway Time-out for url: https://pavics.ouranos.ca/raven/wps ================================================ 11 failed, 4 passed, 1 warning in 248.99s (0:04:08) ================================================= ``` A run on Prod without Twitcher and Nginx: ``` WPS_URL=http://pavics.ouranos.ca:8096/ FINCH_WPS_URL=https://pavics.ouranos.ca/twitcher/ows/proxy/finch/wps FLYINGPIGEON_WPS_URL=https://pavics.oura nos.ca/twitcher/ows/proxy/flyingpigeon/wps pytest --nbval-lax --verbose docs/source/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb --sanitize-with docs/source/output-sanitize.cfg --ignore docs/source/notebooks/.ipynb_checkpoints ===================================================== 15 passed, 1 warning in 218.46s (0:03:38) ====================================================== ```
huard
approved these changes
Feb 10, 2021
tlvu
added a commit
to bird-house/birdhouse-deploy-ouranos
that referenced
this pull request
Feb 10, 2021
…-timeout Sample code to bypass twitcher to debug performance raven process timeout. Just commented out sample code. Keep as a form of documentation how each organization can customize PAVICS deployment. Used in bird-house/birdhouse-deploy#123 for testing.
tlvu
added a commit
to Ouranosinc/PAVICS-e2e-workflow-tests
that referenced
this pull request
Feb 19, 2021
docker: add xESMF to Jupyter env For regridding notebook, see Ouranosinc/pavics-sdi#201 (comment). Also unpin `owslib` (broke xarray) and `cftime` (timeout bug) pinned in previous release (#60). This new Jupyter env is deployed to https://medus.ouranos.ca/jupyter so you can test out xESMF. Jenkins build with Raven notebooks enabled: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/test-nbval-lax-DO_NOT_MERGE/13/console That 12 mins 45 sec includes all Raven, Finch and Pavics-sdi notebooks. It's against my devel server lvupavicsmaster.ouranos.ca since Boreas currently have a performance problem (bird-house/birdhouse-deploy#123) and would take too long to run all Raven notebooks. Raven failures are known, compared to Ouranosinc/raven#349 (review). ``` 12:33:56 =========================== short test summary info ============================ 12:33:56 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 8 12:33:56 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 9 12:33:56 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 10 12:33:56 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 11 12:33:56 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 18 12:33:56 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 19 12:33:56 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 20 12:33:56 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 21 12:33:56 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 22 12:33:56 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 1 12:33:56 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 3 12:33:56 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 4 12:33:56 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 5 12:33:56 FAILED raven-master/docs/source/notebooks/Region_selection.ipynb::Cell 7 12:33:56 FAILED raven-master/docs/source/notebooks/Region_selection.ipynb::Cell 8 12:33:56 FAILED raven-master/docs/source/notebooks/Subset_climate_data_over_watershed.ipynb::Cell 5 12:33:56 ============ 16 failed, 234 passed, 2 skipped in 765.72s (0:12:45) ============= ``` Jenkins build against prod Boreas: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/test-nbval-lax-DO_NOT_MERGE/15/console (performance problem seems to have improved, not sure what has changed on Prod, the whole testsuite used to take around 30 mins before). ``` 16:52:37 =========================== short test summary info ============================ 16:52:37 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 8 16:52:37 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 9 16:52:37 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 10 16:52:37 FAILED raven-master/docs/source/notebooks/Bias_correcting_climate_data.ipynb::Cell 11 16:52:37 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 18 16:52:37 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 19 16:52:37 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 20 16:52:37 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 21 16:52:37 FAILED raven-master/docs/source/notebooks/Full_process_example_1.ipynb::Cell 22 16:52:37 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 1 16:52:37 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 3 16:52:37 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 4 16:52:37 FAILED raven-master/docs/source/notebooks/Multiple_watersheds_simulation.ipynb::Cell 5 16:52:37 FAILED raven-master/docs/source/notebooks/Region_selection.ipynb::Cell 7 16:52:37 FAILED raven-master/docs/source/notebooks/Region_selection.ipynb::Cell 8 16:52:37 FAILED raven-master/docs/source/notebooks/Subset_climate_data_over_watershed.ipynb::Cell 5 16:52:37 ============ 16 failed, 234 passed, 2 skipped in 1012.44s (0:16:52) ============ ``` Noticeable changes: ```diff > - xesmf=0.5.2=pyhd8ed1ab_0 < - owslib=0.21.0=pyhd8ed1ab_0 > - owslib=0.23.0=pyhd8ed1ab_0 < - cftime=1.3.1=py37h6323ea4_0 > - cftime=1.4.1=py37h902c9e0_0 < - dask=2021.1.1=pyhd8ed1ab_0 > - dask=2021.2.0=pyhd8ed1ab_0 < - rioxarray=0.1.1=pyhd8ed1ab_0 > - rioxarray=0.2.0=pyhd8ed1ab_0 ``` Full `conda env export` diff: [210201.2-210216-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6012490/210201.2-210216-conda-env-export.diff.txt) ```diff 10c10 < - alembic=1.5.3=pyhd8ed1ab_0 --- > - alembic=1.5.4=pyhd8ed1ab_0 12c12 < - anyio=2.0.2=py37h89c1867_4 --- > - anyio=2.1.0=py37h89c1867_0 21c21 < - bleach=3.2.3=pyh44b312d_0 --- > - bleach=3.3.0=pyh44b312d_0 38c38 < - cartopy=0.18.0=py37h623d01f_7 --- > - cartopy=0.18.0=py37h02548d7_10 45c45 < - cffi=1.14.4=py37hc58025e_1 --- > - cffi=1.14.5=py37hc58025e_0 48c48 < - cftime=1.3.1=py37h6323ea4_0 --- > - cftime=1.4.1=py37h902c9e0_0 60c60 < - cryptography=3.3.1=py37h7f0c10b_1 --- > - cryptography=3.4.4=py37hf1a17b8_0 69,70c69,70 < - dask=2021.1.1=pyhd8ed1ab_0 < - dask-core=2021.1.1=pyhd8ed1ab_0 --- > - dask=2021.2.0=pyhd8ed1ab_0 > - dask-core=2021.2.0=pyhd8ed1ab_0 79c79 < - distributed=2021.1.1=py37h89c1867_0 --- > - distributed=2021.2.0=py37h89c1867_0 98c98 < - gdal=3.1.4=py37h2ec2946_2 --- > - gdal=3.1.4=py37h2ec2946_5 102,103c102,103 < - geos=3.8.1=he1b5a44_0 < - geotiff=1.6.0=h5d11630_3 --- > - geos=3.9.0=h9c3ff4c_0 > - geotiff=1.6.0=h2b14fbe_4 110,112c110,112 < - gitpython=3.1.12=pyhd8ed1ab_0 < - glib=2.66.4=hc4f0c31_2 < - glib-tools=2.66.4=hc4f0c31_2 --- > - gitpython=3.1.13=pyhd8ed1ab_0 > - glib=2.66.7=h9c3ff4c_0 > - glib-tools=2.66.7=h9c3ff4c_0 117c117 < - h5netcdf=0.8.1=py_0 --- > - h5netcdf=0.10.0=pyhd8ed1ab_0 134c134 < - ipython=7.19.0=py37h888b3d9_2 --- > - ipython=7.20.0=py37h888b3d9_2 139c139 < - jedi=0.17.2=py37h89c1867_1 --- > - jedi=0.18.0=py37h89c1867_2 141c141 < - joblib=1.0.0=pyhd8ed1ab_0 --- > - joblib=1.0.1=pyhd8ed1ab_0 154c154 < - jupyter_server=1.2.3=py37h89c1867_1 --- > - jupyter_server=1.3.0=py37h89c1867_0 163c163 < - jupytext=1.9.1=pyhd3deb0d_1 --- > - jupytext=1.10.0=pyh44b312d_0 165c165 < - kealib=1.4.14=h0042707_0 --- > - kealib=1.4.14=he4dc956_1 170c170 < - lcms2=2.11=hcbb858e_1 --- > - lcms2=2.12=hddcbb42_0 180c180 < - libdap4=3.20.6=h1d1bd15_1 --- > - libdap4=3.20.6=hd7c4107_1 189c189 < - libgdal=3.1.4=h02eeb80_2 --- > - libgdal=3.1.4=h0a4f56a_5 192c192 < - libglib=2.66.4=h748fe8e_2 --- > - libglib=2.66.7=h1f3bc88_0 196c196 < - libkml=1.3.0=h74f7ee3_1012 --- > - libkml=1.3.0=h02e6976_1012 201c201 < - libnghttp2=1.41.0=h8cfc5f6_2 --- > - libnghttp2=1.43.0=h812cca2_0 205a206 > - librttopo=1.1.0=hccdd1c9_5 207,208c208,209 < - libspatialindex=1.9.3=he1b5a44_3 < - libspatialite=5.0.0=heaf302f_0 --- > - libspatialindex=1.9.3=h9c3ff4c_3 > - libspatialite=5.0.1=h04c9dda_2 228c229 < - markdown-it-py=0.6.1=pyhd8ed1ab_0 --- > - markdown-it-py=0.6.2=pyhd8ed1ab_0 232c233 < - mdit-py-plugins=0.2.4=pyhd8ed1ab_0 --- > - mdit-py-plugins=0.2.5=pyhd8ed1ab_0 236c237 < - more-itertools=8.6.0=pyhd8ed1ab_0 --- > - more-itertools=8.7.0=pyhd8ed1ab_0 242c243 < - nbclient=0.5.1=py_0 --- > - nbclient=0.5.2=pyhd8ed1ab_0 244c245 < - nbformat=5.1.2=pyhd8ed1ab_1 --- > - nbformat=5.0.8=py_0 250c251 < - netcdf4=1.5.5.1=nompi_py37hdbee05a_101 --- > - netcdf4=1.5.6=nompi_py37hdbee05a_100 258c259 < - numpy=1.20.0=py37haa41c4c_0 --- > - numpy=1.20.1=py37haa41c4c_0 265c266 < - owslib=0.21.0=pyhd8ed1ab_0 --- > - owslib=0.23.0=pyhd8ed1ab_0 267c268 < - packaging=20.8=pyhd3deb0d_0 --- > - packaging=20.9=pyh44b312d_0 269c270 < - pandas=1.2.1=py37hdc94413_0 --- > - pandas=1.2.2=py37hdc94413_0 274c275 < - parso=0.7.1=pyh9f0ad1d_0 --- > - parso=0.8.1=pyhd8ed1ab_0 280c281 < - pillow=8.1.0=py37he6b4880_1 --- > - pillow=8.1.0=py37h4600e1f_2 290c291 < - proj=7.1.1=h966b41f_3 --- > - proj=7.2.0=h277dcde_2 292,293c293,294 < - prompt-toolkit=3.0.14=pyha770c72_0 < - prompt_toolkit=3.0.14=hd8ed1ab_0 --- > - prompt-toolkit=3.0.16=pyha770c72_0 > - prompt_toolkit=3.0.16=hd8ed1ab_0 305c306 < - pygments=2.7.4=pyhd8ed1ab_0 --- > - pygments=2.8.0=pyhd8ed1ab_0 309c310 < - pyproj=2.6.1.post1=py37h6415a23_3 --- > - pyproj=3.0.0.post1=py37hf874b97_0 319c320 < - python=3.7.9=hffdb5ce_0_cpython --- > - python=3.7.9=hffdb5ce_100_cpython 328c329 < - pyzmq=22.0.1=py37h499b945_0 --- > - pyzmq=22.0.3=py37h499b945_0 337c338 < - rioxarray=0.1.1=pyhd8ed1ab_0 --- > - rioxarray=0.2.0=pyhd8ed1ab_0 347c348 < - shapely=1.7.1=py37hba0730f_1 --- > - shapely=1.7.1=py37h4c50fd9_2 356c357 < - sqlalchemy=1.3.22=py37h5e8e339_1 --- > - sqlalchemy=1.3.23=py37h5e8e339_0 358c359 < - statsmodels=0.12.1=py37h902c9e0_2 --- > - statsmodels=0.12.2=py37h902c9e0_0 364,365c365,366 < - tifffile=2021.1.14=pyhd8ed1ab_0 < - tiledb=2.1.6=h91fcb0e_1 --- > - tifffile=2021.2.1=pyhd8ed1ab_0 > - tiledb=2.2.3=h91fcb0e_0 370c371 < - tqdm=4.56.0=pyhd8ed1ab_0 --- > - tqdm=4.56.2=pyhd8ed1ab_0 391,392c392,394 < - xeus=1.0.0=h78d96c3_0 < - xeus-python=0.10.2=py37h52ea871_0 --- > - xesmf=0.5.2=pyhd8ed1ab_0 > - xeus=1.0.1=h78d96c3_0 > - xeus-python=0.11.0=py37h52ea871_0 406c408 < - zeromq=4.3.3=h58526e2_3 --- > - zeromq=4.3.4=h9c3ff4c_0 ``` Full new `conda env export`: [210216-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/6012491/210216-conda-env-export.yml.txt) ``` name: birdy channels: - cdat - conda-forge - defaults dependencies: - _libgcc_mutex=0.1=conda_forge - _openmp_mutex=4.5=1_gnu - affine=2.3.0=py_0 - alembic=1.5.4=pyhd8ed1ab_0 - ansi2html=1.6.0=py37h89c1867_1 - anyio=2.1.0=py37h89c1867_0 - appdirs=1.4.4=pyh9f0ad1d_0 - argon2-cffi=20.1.0=py37h5e8e339_2 - async_generator=1.10=py_0 - attrs=20.3.0=pyhd3deb0d_0 - backcall=0.2.0=pyh9f0ad1d_0 - backports=1.0=py_2 - backports.functools_lru_cache=1.6.1=py_0 - beautifulsoup4=4.9.3=pyhb0f4dca_0 - bleach=3.3.0=pyh44b312d_0 - blinker=1.4=py_1 - blosc=1.21.0=h9c3ff4c_0 - bokeh=2.2.3=py37h89c1867_0 - boltons=20.2.1=pyh9f0ad1d_0 - boost-cpp=1.74.0=hc6e9bd1_2 - bottleneck=1.3.2=py37h902c9e0_3 - branca=0.4.2=pyhd8ed1ab_0 - brotli=1.0.9=h9c3ff4c_4 - brotli-python=1.0.9=py37hcd2ae1e_4 - brotlipy=0.7.0=py37h5e8e339_1001 - brunsli=0.1=h9c3ff4c_0 - bzip2=1.0.8=h7f98852_4 - c-ares=1.17.1=h36c2ea0_0 - ca-certificates=2020.12.5=ha878542_0 - cached-property=1.5.1=py_0 - cairo=1.16.0=h7979940_1007 - cartopy=0.18.0=py37h02548d7_10 - cdat_info=8.2.1=pyh9f0ad1d_1 - cdtime=3.1.4=py37h5ad02f4_2 - cdutil=8.2.1=pyh9f0ad1d_0 - certifi=2020.12.5=py37h89c1867_1 - certipy=0.1.3=py_0 - cf_xarray=0.4.0=pyh44b312d_0 - cffi=1.14.5=py37hc58025e_0 - cfgrib=0.9.8.5=pyhd8ed1ab_0 - cfitsio=3.470=hb418390_7 - cftime=1.4.1=py37h902c9e0_0 - chardet=4.0.0=py37h89c1867_1 - charls=2.2.0=h9c3ff4c_0 - click=7.1.2=pyh9f0ad1d_0 - click-plugins=1.1.1=py_0 - cligj=0.7.1=pyhd8ed1ab_0 - clisops=0.5.1=pyhd3deb0d_0 - cloudpickle=1.6.0=py_0 - colorama=0.4.4=pyh9f0ad1d_0 - colorcet=2.0.6=pyhd8ed1ab_0 - configurable-http-proxy=1.3.0=0 - coverage=5.4=py37h5e8e339_0 - cryptography=3.4.4=py37hf1a17b8_0 - curl=7.71.1=he644dc0_8 - cycler=0.10.0=py_2 - cytoolz=0.11.0=py37h5e8e339_3 - dash=1.19.0=pyhd8ed1ab_0 - dash-core-components=1.15.0=pyhd8ed1ab_0 - dash-html-components=1.1.2=pyh44b312d_0 - dash-renderer=1.9.0=pyhd8ed1ab_0 - dash-table=4.11.2=pyhd8ed1ab_0 - dask=2021.2.0=pyhd8ed1ab_0 - dask-core=2021.2.0=pyhd8ed1ab_0 - datashader=0.12.0=pyhd3deb0d_0 - datashape=0.5.4=py_1 - dbus=1.13.6=hfdff14a_1 - debugpy=1.2.1=py37hcd2ae1e_1 - decorator=4.4.2=py_0 - defusedxml=0.6.0=py_0 - descartes=1.1.0=py_4 - distarray=2.12.2=py_1 - distributed=2021.2.0=py37h89c1867_0 - docopt=0.6.2=py_1 - dv3d=8.2.1=pyh9f0ad1d_0 - eccodes=2.20.0=ha0e6eb6_0 - entrypoints=0.3=pyhd8ed1ab_1003 - esgf-compute-api=2.3.7=0_h1234567_ - esmf=8.0.1=nompi_he31a43a_2 - esmpy=8.0.1=nompi_py37h59b2dc9_2 - expat=2.2.10=h9c3ff4c_0 - ffmpeg=4.2.3=h167e202_0 - fiona=1.8.18=py37h527b4ca_0 - flask=1.1.2=pyh9f0ad1d_0 - flask-compress=1.8.0=pyhd8ed1ab_0 - fontconfig=2.13.1=hba837de_1004 - freetype=2.10.4=h0708190_1 - freexl=1.0.5=h516909a_1002 - fsspec=0.8.5=pyhd8ed1ab_0 - future=0.18.2=py37h89c1867_3 - g2clib=1.6.0=hf3f1b0b_9 - gdal=3.1.4=py37h2ec2946_5 - genutil=8.2.1=py37h161383b_1 - geojson=2.5.0=py_0 - geopandas=0.8.2=pyhd8ed1ab_0 - geos=3.9.0=h9c3ff4c_0 - geotiff=1.6.0=h2b14fbe_4 - geoviews=1.8.2=py_0 - geoviews-core=1.8.2=py_0 - gettext=0.19.8.1=h0b5b191_1005 - ghostscript=9.53.3=h58526e2_2 - giflib=5.2.1=h36c2ea0_2 - gitdb=4.0.5=pyhd8ed1ab_1 - gitpython=3.1.13=pyhd8ed1ab_0 - glib=2.66.7=h9c3ff4c_0 - glib-tools=2.66.7=h9c3ff4c_0 - gmp=6.2.1=h58526e2_0 - gnutls=3.6.13=h85f3911_1 - gst-plugins-base=1.14.5=h0935bb2_2 - gstreamer=1.18.3=h3560a44_0 - h5netcdf=0.10.0=pyhd8ed1ab_0 - h5py=3.1.0=nompi_py37h1e651dc_100 - hdf4=4.2.13=h10796ff_1004 - hdf5=1.10.6=nompi_h6a2412b_1114 - heapdict=1.0.1=py_0 - holoviews=1.14.1=pyhd3deb0d_0 - hvplot=0.7.0=pyhd3deb0d_0 - icu=68.1=h58526e2_0 - idna=2.10=pyh9f0ad1d_0 - imagecodecs=2021.1.11=py37h95c7a1b_1 - imageio=2.9.0=py_0 - importlib-metadata=3.4.0=py37h89c1867_0 - importlib_metadata=3.4.0=hd8ed1ab_0 - importlib_resources=5.1.0=py37h89c1867_0 - iniconfig=1.1.1=pyh9f0ad1d_0 - ipykernel=5.4.2=py37h888b3d9_0 - ipyleaflet=0.13.6=pyhd8ed1ab_1 - ipython=7.20.0=py37h888b3d9_2 - ipython_genutils=0.2.0=py_1 - ipywidgets=7.6.3=pyhd3deb0d_0 - itsdangerous=1.1.0=py_0 - jasper=1.900.1=h07fcdf6_1006 - jedi=0.18.0=py37h89c1867_2 - jinja2=2.11.3=pyh44b312d_0 - joblib=1.0.1=pyhd8ed1ab_0 - jpeg=9d=h36c2ea0_0 - json-c=0.13.1=hbfbb72e_1002 - json5=0.9.5=pyh9f0ad1d_0 - jsonschema=3.2.0=py_2 - jupyter=1.0.0=py37h89c1867_6 - jupyter-archive=2.2.0=pyhd8ed1ab_0 - jupyter-dash=0.4.0=pyhd8ed1ab_0 - jupyter_bokeh=2.0.4=pyhd8ed1ab_0 - jupyter_client=6.1.11=pyhd8ed1ab_1 - jupyter_conda=4.1.0=hd8ed1ab_1 - jupyter_console=6.2.0=py_0 - jupyter_core=4.7.1=py37h89c1867_0 - jupyter_server=1.3.0=py37h89c1867_0 - jupyter_telemetry=0.1.0=pyhd8ed1ab_1 - jupyterhub=1.3.0=py37h89c1867_0 - jupyterhub-base=1.3.0=py37h89c1867_0 - jupyterlab=2.2.9=pyhd8ed1ab_0 - jupyterlab-git=0.23.3=pyhd8ed1ab_0 - jupyterlab_pygments=0.1.2=pyh9f0ad1d_0 - jupyterlab_server=1.2.0=py_0 - jupyterlab_widgets=1.0.0=pyhd8ed1ab_1 - jupytext=1.10.0=pyh44b312d_0 - jxrlib=1.1=h7f98852_2 - kealib=1.4.14=he4dc956_1 - kiwisolver=1.3.1=py37h2527ec5_1 - krb5=1.17.2=h926e7f8_0 - lame=3.100=h7f98852_1001 - lazy-object-proxy=1.5.2=py37h5e8e339_1 - lcms2=2.12=hddcbb42_0 - ld_impl_linux-64=2.35.1=hea4e1c9_2 - lerc=2.2.1=h9c3ff4c_0 - libaec=1.0.4=h9c3ff4c_1 - libblas=3.8.0=17_openblas - libcblas=3.8.0=17_openblas - libcdms=3.1.2=h981a4fd_113 - libcf=1.0.3=py37heda41dd_109 - libclang=11.0.1=default_ha53f305_1 - libcurl=7.71.1=hcdd3856_8 - libdap4=3.20.6=hd7c4107_1 - libdeflate=1.7=h7f98852_5 - libdrs=3.1.2=h7918d09_113 - libdrs_f=3.1.2=h5026c31_111 - libedit=3.1.20191231=he28a2e2_2 - libev=4.33=h516909a_1 - libevent=2.1.10=hcdb4288_3 - libffi=3.3=h58526e2_2 - libgcc-ng=9.3.0=h2828fa1_18 - libgdal=3.1.4=h0a4f56a_5 - libgfortran-ng=9.3.0=hff62375_18 - libgfortran5=9.3.0=hff62375_18 - libglib=2.66.7=h1f3bc88_0 - libgomp=9.3.0=h2828fa1_18 - libiconv=1.16=h516909a_0 - libidn2=2.3.0=h516909a_0 - libkml=1.3.0=h02e6976_1012 - liblapack=3.8.0=17_openblas - libllvm10=10.0.1=he513fc3_3 - libllvm11=11.0.1=hf817b99_0 - libnetcdf=4.7.4=nompi_h56d31a8_107 - libnghttp2=1.43.0=h812cca2_0 - libopenblas=0.3.10=pthreads_h4812303_5 - libpng=1.6.37=h21135ba_2 - libpq=12.3=h255efa7_3 - libprotobuf=3.14.0=h780b84a_0 - librttopo=1.1.0=hccdd1c9_5 - libsodium=1.0.18=h36c2ea0_1 - libspatialindex=1.9.3=h9c3ff4c_3 - libspatialite=5.0.1=h04c9dda_2 - libssh2=1.9.0=hab1572f_5 - libstdcxx-ng=9.3.0=h6de172a_18 - libtiff=4.2.0=hdc55705_0 - libunistring=0.9.10=h14c3975_0 - libuuid=2.32.1=h7f98852_1000 - libuv=1.40.0=h7f98852_0 - libwebp-base=1.2.0=h7f98852_0 - libxcb=1.13=h7f98852_1003 - libxkbcommon=1.0.3=he3ba5ed_0 - libxml2=2.9.10=h72842e0_3 - libxslt=1.1.33=h15afd5d_2 - libzopfli=1.0.3=h9c3ff4c_0 - llvmlite=0.35.0=py37h9d7f4d0_1 - locket=0.2.0=py_2 - lxml=4.6.2=py37h77fd288_1 - lz4-c=1.9.3=h9c3ff4c_0 - mako=1.1.4=pyh44b312d_0 - mamba_gator=4.1.0=pyhd8ed1ab_1 - markdown=3.3.3=pyh9f0ad1d_0 - markdown-it-py=0.6.2=pyhd8ed1ab_0 - markupsafe=1.1.1=py37h5e8e339_3 - matplotlib=3.3.4=py37h89c1867_0 - matplotlib-base=3.3.4=py37h0c9df89_0 - mdit-py-plugins=0.2.5=pyhd8ed1ab_0 - memory_profiler=0.58.0=py_0 - mesalib=18.3.1=h590aaf7_0 - mistune=0.8.4=py37h5e8e339_1003 - more-itertools=8.7.0=pyhd8ed1ab_0 - msgpack-python=1.0.2=py37h2527ec5_1 - multipledispatch=0.6.0=py_0 - munch=2.5.0=py_0 - mysql-common=8.0.22=ha770c72_3 - mysql-libs=8.0.22=h935591d_3 - nbclient=0.5.2=pyhd8ed1ab_0 - nbdime=2.1.0=py_0 - nbformat=5.0.8=py_0 - nbval=0.9.6=pyh9f0ad1d_0 - nc-time-axis=1.2.0=py_1 - ncurses=6.2=h58526e2_4 - nest-asyncio=1.4.3=pyhd8ed1ab_0 - netcdf-fortran=4.5.3=nompi_h1a0d97b_101 - netcdf4=1.5.6=nompi_py37hdbee05a_100 - nettle=3.6=he412f7d_0 - networkx=2.5=py_0 - nodejs=15.3.0=h92b4a50_1 - notebook=6.2.0=py37h89c1867_0 - nspr=4.29=h9c3ff4c_1 - nss=3.61=hb5efdd6_0 - numba=0.52.0=py37hdc94413_0 - numpy=1.20.1=py37haa41c4c_0 - oauthlib=3.0.1=py_0 - olefile=0.46=pyh9f0ad1d_1 - openblas=0.3.10=pthreads_h04b7a96_5 - openh264=2.1.1=h780b84a_0 - openjpeg=2.4.0=hf7af979_0 - openssl=1.1.1i=h7f98852_0 - owslib=0.23.0=pyhd8ed1ab_0 - oyaml=1.0=pyhd8ed1ab_0 - packaging=20.9=pyh44b312d_0 - pamela=1.0.0=py_0 - pandas=1.2.2=py37hdc94413_0 - pandoc=2.11.4=h7f98852_0 - pandocfilters=1.4.2=py_1 - panel=0.10.3=pyhd8ed1ab_0 - param=1.10.1=pyhd3deb0d_0 - parso=0.8.1=pyhd8ed1ab_0 - partd=1.1.0=py_0 - patsy=0.5.1=py_0 - pcre=8.44=he1b5a44_0 - pexpect=4.8.0=pyh9f0ad1d_2 - pickleshare=0.7.5=py_1003 - pillow=8.1.0=py37h4600e1f_2 - pint=0.16.1=py_0 - pip=21.0.1=pyhd8ed1ab_0 - pixman=0.40.0=h36c2ea0_0 - plotly=4.14.3=pyh44b312d_0 - pluggy=0.13.1=py37h89c1867_4 - pooch=1.3.0=pyhd8ed1ab_0 - poppler=0.89.0=h2de54a5_5 - poppler-data=0.4.10=0 - postgresql=12.3=hc2f5b80_3 - proj=7.2.0=h277dcde_2 - prometheus_client=0.9.0=pyhd3deb0d_0 - prompt-toolkit=3.0.16=pyha770c72_0 - prompt_toolkit=3.0.16=hd8ed1ab_0 - protobuf=3.14.0=py37hcd2ae1e_1 - pscript=0.7.5=pyhd3deb0d_0 - psutil=5.8.0=py37h5e8e339_1 - pthread-stubs=0.4=h36c2ea0_1001 - ptyprocess=0.7.0=pyhd3deb0d_0 - py=1.10.0=pyhd3deb0d_0 - pycparser=2.20=pyh9f0ad1d_2 - pyct=0.4.6=py_0 - pyct-core=0.4.6=py_0 - pycurl=7.43.0.6=py37h88a64d2_1 - pydap=3.2.2=pyh9f0ad1d_1001 - pygments=2.8.0=pyhd8ed1ab_0 - pyjwt=2.0.1=pyhd8ed1ab_0 - pyopenssl=20.0.1=pyhd8ed1ab_0 - pyparsing=2.4.7=pyh9f0ad1d_0 - pyproj=3.0.0.post1=py37hf874b97_0 - pyqt=5.12.3=py37h89c1867_7 - pyqt-impl=5.12.3=py37he336c9b_7 - pyqt5-sip=4.19.18=py37hcd2ae1e_7 - pyqtchart=5.12=py37he336c9b_7 - pyqtwebengine=5.12.1=py37he336c9b_7 - pyrsistent=0.17.3=py37h5e8e339_2 - pyshp=2.1.3=pyh44b312d_0 - pysocks=1.7.1=py37h89c1867_3 - pytest=6.2.2=py37h89c1867_0 - python=3.7.9=hffdb5ce_100_cpython - python-dateutil=2.8.1=py_0 - python-editor=1.0.4=py_0 - python-json-logger=2.0.1=pyh9f0ad1d_0 - python_abi=3.7=1_cp37m - pytz=2021.1=pyhd8ed1ab_0 - pyviz_comms=2.0.1=pyhd3deb0d_0 - pywavelets=1.1.1=py37h902c9e0_3 - pyyaml=5.4.1=py37h5e8e339_0 - pyzmq=22.0.3=py37h499b945_0 - qt=5.12.9=h9d6b050_2 - qtconsole=5.0.2=pyhd8ed1ab_0 - qtpy=1.9.0=py_0 - rasterio=1.2.0=py37ha549118_0 - readline=8.0=he28a2e2_2 - regionmask=0.6.2=pyhd8ed1ab_0 - requests=2.25.1=pyhd3deb0d_0 - retrying=1.3.3=py_2 - rioxarray=0.2.0=pyhd8ed1ab_0 - roocs-utils=0.1.5=pyhd3deb0d_1 - rtree=0.9.7=py37h0b55af0_1 - ruamel.yaml=0.16.12=py37h5e8e339_2 - ruamel.yaml.clib=0.2.2=py37h5e8e339_2 - scikit-image=0.18.1=py37hdc94413_0 - scikit-learn=0.24.1=py37h69acf81_0 - scipy=1.6.0=py37h14a347d_0 - send2trash=1.5.0=py_0 - setuptools=49.6.0=py37h89c1867_3 - shapely=1.7.1=py37h4c50fd9_2 - siphon=0.8.0=py37h89c1867_1004 - six=1.15.0=pyh9f0ad1d_0 - smmap=3.0.5=pyh44b312d_0 - snappy=1.1.8=he1b5a44_3 - sniffio=1.2.0=py37h89c1867_1 - snuggs=1.4.7=py_0 - sortedcontainers=2.3.0=pyhd8ed1ab_0 - soupsieve=2.0.1=py_1 - sqlalchemy=1.3.23=py37h5e8e339_0 - sqlite=3.34.0=h74cdb3f_0 - statsmodels=0.12.2=py37h902c9e0_0 - tblib=1.6.0=py_0 - terminado=0.9.2=py37h89c1867_0 - testpath=0.4.4=py_0 - threadpoolctl=2.1.0=pyh5ca1d4c_0 - threddsclient=0.4.2=py_0 - tifffile=2021.2.1=pyhd8ed1ab_0 - tiledb=2.2.3=h91fcb0e_0 - tk=8.6.10=h21135ba_1 - toml=0.10.2=pyhd8ed1ab_0 - toolz=0.11.1=py_0 - tornado=6.1=py37h5e8e339_1 - tqdm=4.56.2=pyhd8ed1ab_0 - traitlets=5.0.5=py_0 - traittypes=0.2.1=pyh9f0ad1d_2 - typing_extensions=3.7.4.3=py_0 - tzcode=2021a=h7f98852_0 - udunits2=2.2.27.27=h360fe7b_0 - urllib3=1.26.3=pyhd8ed1ab_0 - vcs=8.2.1=pyh9f0ad1d_0 - voila=0.2.6=pyhd8ed1ab_0 - vtk-cdat=8.2.0.8.2=py37_mesalibhead77ed_0 - wcwidth=0.2.5=pyh9f0ad1d_2 - webencodings=0.5.1=py_1 - webob=1.8.6=py_0 - werkzeug=1.0.1=pyh9f0ad1d_0 - wget=1.20.1=h22169c7_0 - wheel=0.36.2=pyhd3deb0d_0 - widgetsnbextension=3.5.1=py37h89c1867_4 - x264=1!152.20180806=h14c3975_0 - xarray=0.16.2=pyhd8ed1ab_0 - xclim=0.23.0=pyhd8ed1ab_0 - xerces-c=3.2.3=h9d8b166_2 - xesmf=0.5.2=pyhd8ed1ab_0 - xeus=1.0.1=h78d96c3_0 - xeus-python=0.11.0=py37h52ea871_0 - xorg-kbproto=1.0.7=h7f98852_1002 - xorg-libice=1.0.10=h516909a_0 - xorg-libsm=1.2.3=h84519dc_1000 - xorg-libx11=1.6.12=h516909a_0 - xorg-libxau=1.0.9=h7f98852_0 - xorg-libxdmcp=1.1.3=h7f98852_0 - xorg-libxext=1.3.4=h516909a_0 - xorg-libxrender=0.9.10=h516909a_1002 - xorg-renderproto=0.11.1=h14c3975_1002 - xorg-xextproto=7.3.0=h7f98852_1002 - xorg-xproto=7.0.31=h7f98852_1007 - xz=5.2.5=h516909a_1 - yaml=0.2.5=h516909a_0 - zeromq=4.3.4=h9c3ff4c_0 - zfp=0.5.5=h9c3ff4c_4 - zict=2.0.0=py_0 - zipp=3.4.0=py_0 - zlib=1.2.11=h516909a_1010 - zstd=1.4.8=ha95c52a_1 - pip: - astunparse==1.6.3 - birdhouse-birdy==0.7.0 - cdms2==3.1.5 - colour==0.1.5 - funcsigs==1.0.2 - handcalcs==1.3.0 - innerscope==0.2.0 - ipython-blocking==0.2.1 - jupyternotify==0.1.15 - nbconvert==5.6.1 - nbresuse==0.4.0 - pixiedust==1.1.19 - pytest-tornasync==0.6.0.post2 - ravenpy==0.2.3 - regrid2==3.1.5 - requests-magpie==0.1.1 - wrapt==1.12.1 prefix: /opt/conda/envs/birdy ```
tlvu
added a commit
that referenced
this pull request
Apr 16, 2021
Got intermittent Jupyter personal server spawn timeout today. First time ever on prod. It's intermittent so unable to really test which of the 3 timeout configs was the root cause so double all 3 of them. The CPU was 85% idle, RAM not that full (see `free -m` output below) so suspect IO problem for the timeout (see #123). ``` $ free -m total used free shared buff/cache available Mem: 128822 36628 1611 2550 90582 89032 Swap: 3905 3905 0 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have a performance problem with the production deployment at Ouranos so we need a longer timeout. Being a Ouranos specific need, it should not be hardcoded as in previous PR #122.
The previous increase was sometime not enough !
The value is now configurable via
env.localas most other customizations. Documentation updated.Timeout in Prod:
Pass easily on my test VM with very modest hardware (10G ram, 2 cpu):
Pass against Medus:
Pass against
hirondelle.crim.ca:For comparison, a run on Prod without Twitcher (PR bird-house/birdhouse-deploy-ouranos#5):
A run on Prod without Twitcher and Nginx (direct hit Raven):