Increase sci-wms performance to that of ncwms

[rsignell-usgs]

I understand the need to get things working first and then worry about performance.

But very few people will use sci-wms for regular/curvilinear grids if it is much slower than ncwms.

For the test here:
https://gist.github.com/rsignell-usgs/c2d112d050c42914f9b7
the getmap request is taking about 5 times longer with sci-wms compared to ncwms.

Test 1: sci-wms request

Test 2: ncwms request

[kwilcox]

Better tests (exact same bbox, projection, elevation):

sci-wms

ncwms

[kwilcox]

Your sci-wms endpoint is hitting a DAP server and ncwms would be hitting files directly...

[brianmckenna]

Is sci-wms.whoi.edu the same machine as geoport-dev.whoi.edu? Is that basically network access vs. local access?

[rsignell-usgs]

Yes, sci-wms.whoi.edu is the same machine as geoport-dev.whoi.edu. I think Kyle is right, that OPeNDAP is simply much slower that local file read, even when the OPeNDAP access is local. See cells [6],[7] and [8] here:
http://nbviewer.ipython.org/gist/rsignell-usgs/c2d112d050c42914f9b7

I added a test for CDMRemote, and it turns out it's quite a bit faster than OPeNDAP. So we could add a test to see if CDMRemote is enabled, and if so, use it. If not, fall back to OPeNDAP. It's the exact same syntax. It's just:

from siphon.cdmr import Dataset

instead of

from netCDF4 import Dataset

[rsignell-usgs]

But the sad news is that sci-wms will never be as fast as ncWMS, I guess.

[kwilcox]

We don't cache anything except a netcdf file on disk with the coordinate information. We could instead try keeping the object in memory using the newish netcdf in memory dataset functionality (haven't played with that). But yes, OPeNDAP will always be slower than disk access when requesting the actual data slice.

[rsignell-usgs]

This is a major bummer. Nobody will use sci-wms for structured grid (even if we have support for SGRID) if it's significantly slower than ncWMS. We will have support for velocity vectors, but staggered grid people have to choose between:

1. plotting velocity vectors with slower scalar data (sci-wms)
2. no velocity vectors with faster scalar data (ncWMS)

The best thing I guess would be to push for ncWMS to support SGRID conventions...

Unless we can figure out how to speed up plotting scalar fields in sci-wms...

[rsignell-usgs]

The impact of this slowness is really felt when the clients request tiles. The map below from the sci-wms demo client took 30 seconds to finish drawing, issues 12 different getMap requests. The same map from ncWMS takes about 5 seconds.

I did notice that while most of these getMap requests look okay, two look suspicious, with e-10 specified for y_min in the bounding box:

http://sci-wms.whoi.edu/wms/datasets/coawst?service=WMS&request=GetMap&version=1.1.1&layers=Hwave&styles=filledcontours_jet&format=image/png&transparent=true&height=256&width=256&colorscalerange=0,1&srs=EPSG:3857&bbox=-15028131.257091932,-7.081154551613622e-10,-12523442.714243276,2504688.5428486555

http://sci-wms.whoi.edu/wms/datasets/coawst?service=WMS&request=GetMap&version=1.1.1&layers=Hwave&styles=filledcontours_jet&format=image/png&transparent=true&height=256&width=256&colorscalerange=0,1&srs=EPSG:3857&bbox=-7514065.628545966,-7.081154551613622e-10,-5009377.085697311,2504688.5428486555

[kwilcox]

That's a tile off the bottom of the map that just happens to be close to the equator.

[rsignell-usgs]

So should we be requesting tiles that are off the bottom of the map?
These requests return completely blank images.

[PBrockmann]

As far as I know, there are only 3 open source projects to have implement WMS for netCDF files:
ncWMS2 (java servlet), adaguc server (C++ ) and sci-wms (python).

ncWMS2 works well up to curvilinear grids but the contour style is very special I would say gently. With the adaguc server, I have seen a test from a generic grid and it can also handle properly curvilinear grids (see https://dev.knmi.nl/projects/adagucserver/wiki/DataFormats).
But only sci-wms has been design to handle any type of grids and what I can see so far is very promissing from the rendering point of view because you have the quality of matplolib behind.

The main challenge is of course to have a WMS in a very responsive way both with OPeNDAP access and local netCDF files. But I would be more interested to get a very efficient local server to propose WMS from any calculs on netCDF files made from a IPython notebooks.
The Climate and Earth System Modelling communities would be very interested by that for sure.

See this capture from the IPython notebook with the use of folium (a python wrapper to Leaflet) https://gist.github.com/PBrockmann/5874c88bb61ac2b7af50

Hope this possible perspective will interest you.

[rsignell-usgs]

@PBrockmann , thanks for the vote of confidence!