Incorrect download of rasters

[stufraser1]

Some TIFs not downloading as correct format - default to wcs

See previous comments
GeoNode#129
GeoNode#1485

@ErikKBethke Please add links to some layers that this occurred on

[ErikKBethke]

Examples of layers that have recurring issue:

http://www.geonode-gfdrrlab.org/layers/hazard%3Ass_muis_rp0050m
http://www.geonode-gfdrrlab.org/layers/hazard%3Aswio_com_500_cf

Note that KML, PDF, PNG, JPEG appear to be downloading as intended.

[ErikKBethke]

WCS error message for http://54.229.171.111/layers/hazard:ss_muis_rp0050m:

  illegal bbox, minY: 187.27201915472722 is greater than maxY: 154.3123442971117

[fvanderbiest]

Problem with data, also witnessed when harvesting from TH, see GFDRR/thinkhazard#754

[cgiovando]

The Labs GeoNode is now on version 2.6, but this issue is still happening.

http://www.geonode-gfdrrlab.org/geoserver/wcs?crs=EPSG%3A4326&service=WCS&format=GeoTIFF&request=GetCoverage&height=16744&width=43200&version=1.0.0&BBox=-187.27200037926434%2C187.27201915472722%2C-126.59567035338198%2C154.3123442971117&Coverage=hazard%3Ass_muis_rp0050m&access_token=fd4fd642562311e781cc02a677ab2e43

returns the following exception:

<?xml version="1.0" encoding="UTF-8"?><ServiceExceptionReport version="1.2.0" > >
<ServiceException code="InvalidParameterValue" locator="bbox">
illegal bbox, minY: 187.27201915472722 is greater than maxY: 154.3123442971117
</ServiceException></ServiceExceptionReport>

@afabiani can you please help us debugging and fixing the issue?

The user should be able to click on Download Layer --> Data --> GeoTIFF and download the entire dataset at native resolution.

[afabiani]

Hi, the GetCoverage request is wrong. If it has been automatically generated by GeoNode, this could be a side-effect of this bug on GeoNode

GeoNode#3109

I will need to double check in order to understand why this is happening.

[rmurnane]

I am also having problems downloading data. When trying to download SWIO data (COM, MAD, MUS, SEY, Zanzibar) I get the following error message:

  illegal bbox, minX: 37.56745399999999 is greater than maxX: -25.741642000000002

[afabiani]

Dear all,
with this PR (GeoNode#3158) I should have fixed the issue on GeoNode master branch.

As soon as the PR has been revised and merged, we can envisage to do a backport to GeoNode 2.6.1 (which, if I'm not wrong, should be the one you are currently using).

Nevertheless, in your specific case the Download Links have been already written into the DB. In order to fix your existing data you would need to modify them either manually or by writing a simple Python script that does the job for you.

Long story short, for all WCS links you should either invert the coords x0,y0,x1,y2 with x0,x1,y1,y2 or, in case you need the whole dataset to be downloaded, remove the bbox params at all.

As an instance you could simplify the download link for http://www.geonode-gfdrrlab.org/layers/hazard%3Aswio_com_500_cf

from the current one

http://www.geonode-gfdrrlab.org/geoserver/wcs?crs=EPSG%3A4326&service=WCS&format=GeoTIFF&request=GetCoverage&height=655&width=812&version=1.0.0&BBox=43.17029%2C44.59557%2C-12.42715%2C-11.35819&Coverage=hazard%3Aswio_com_500_cf&access_token=ea464cc05ff611e79aa702a677ab2e43

to the following one:

http://www.geonode-gfdrrlab.org/geoserver/wcs?service=WCS&format=GeoTIFF&request=GetCoverage&version=2.0.1&CoverageId=hazard%3Aswio_com_500_cf&access_token=ea464cc05ff611e79aa702a677ab2e43

WARNING: Allowing people to download full res images without restrictions to the BBOX is dangerous. They can easily perform a DDOS attack to your server and also completely fill your HD space. In order to avoid that, you should set limits to the GeoServer OWS Services (see the following links for details)

• http://geoserver.geo-solutions.it/edu/en/adv_gsconfig/parameters.html
• http://docs.geoserver.org/latest/en/user/extensions/controlflow/index.html (for more advanced configuration)

[Vampouille]

Following query should fix download link in database :

UPDATE base_resourcebase SET csw_anytext = regexp_replace(csw_anytext,
'(b|Bb|Box)=(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)', -- pattern text
E'\\1=\\2\\3\\6\\5\\4\\7\\8', -- replacement text
'g')-- flags

[fvanderbiest]

Nice SQL query :-)
@Vampouille did you run it on both dev and prod servers ?

[Vampouille]

@afabiani I backport your PR our 2.6 branch with : https://github.com/camptocamp/labs_geonode/pull/7

I try to fix link stored in DB by updating csw_anytext on base_resourcebase table. But it does not seem to work. Do you known where those links are stored in DB ?

Additionally, I test on:
http://www.geonode-gfdrrlab.org/layers/hazard%3Ass_muis_rp0050m
download link of this layer :
https://www.geonode-gfdrrlab.org/geoserver/wcs?crs=EPSG%3A4326&service=WCS&format=GeoTIFF&request=GetCoverage&height=16744&width=43200&version=1.0.0&BBox=-187.27200037926434%2C187.27201915472722%2C-126.59567035338198%2C154.3123442971117&Coverage=hazard%3Ass_muis_rp0050m&access_token=ec45100272c911e7a34c02a677dff4dc

EPSG is set to 4326 but all parameters of bbox are above 90:

• -187.2
• 187.2
• -126.5
• 154.3
  so even if we fix bbox order we will still have issue on values

[Vampouille]

Do you known where those links are stored in DB ?

Links behind "Download layer" are stored in base_link table so following query should be applied:

UPDATE base_link SET url = regexp_replace(url,'(b|Bb|Box)=(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)',E'\\1=\\2\\3\\6\\5\\4\\7\\8','g');

Already applied on geonode int server

So third link on this layer :
http://int.geonode-gfdrrlab.org/layers/hazard%3Ass_muis_rp0050m
has a valid bbox values and order:

https://int.geonode-gfdrrlab.org/geoserver/wcs?crs=EPSG%3A4326&service=WCS&format=GeoTIFF&request=GetCoverage&height=16744&width=43200&version=1.0.0&BBox=-180.0%2C-55.90833333341696%2C180.00001877546288%2C83.62500727714671&Coverage=hazard%3Ass_muis_rp0050m&access_token=6e51dd6672cd11e7964402ffe6652b08

[fvanderbiest]

Image camptocamp/geonode_django:labs_geonode_2.6.x_20170727140636 deployed to production geonode and production db updated:

postgres=# \c geonode
psql (9.6.3, server 9.5.7)
You are now connected to database "geonode" as user "postgres".
geonode=# UPDATE base_link SET url = regexp_replace(url,'(b|Bb|Box)=(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)(,|\%2C)(-?\d+.?\d+)',E'\\1=\\2\\3\\6\\5\\4\\7\\8','g');

UPDATE 28832

The issue should be fixed now.

[fvanderbiest]

Can we close this one ?

[ErikKBethke]

When attempting to download rasters, such as this layer, there is an issue where the file download becomes much larger than the original file size.

@vdeparday believes it may be a result of the default WCS request. Thoughts?

[vdeparday]

Yes, the various return periods for this layer have the issue. It looks like the layer also does not display on the map:

[vdeparday]

actually looks like the rendering is fine (just somehow this layer does not show coastal flood in this area which seems to be wrong but the rendering is fine. However, the WCS request does not work.

[ErikKBethke]

Bounding box issue seems to have resurfaced:

Example layer
Example response:
<ows:ExceptionReport xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ows="http://www.opengis.net/ows/2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="2.0.0" xsi:schemaLocation="http://www.opengis.net/ows/2.0 http://schemas.opengis.net/ows/2.0/owsExceptionReport.xsd">
<ows:Exception exceptionCode="NoApplicableCode">
ows:ExceptionTextillegal bbox, minY: 180.00500000000002 is greater than maxY: 67.005 []
</ows:ExceptionText>
</ows:Exception>
</ows:ExceptionReport>

[fvanderbiest]

When attempting to download rasters, such as this layer, there is an issue where the file download becomes much larger than the original file size.

Can you open a different issue for this one ?
...specifying which output format you choose. Thanks.

[fvanderbiest]

Bounding box issue seems to have resurfaced: https://www.geonode-gfdrrlab.org/layers/hazard:ts_mih_rp1000

The layer was uploaded on 2017-08-15, which is 15 days after we applied the above SQL fix on the db.
The upload process is probably still buggy. We'll have a second look at it.

[afabiani]

Hi all,
first of all I see few redundant links for WCS on that layer

Are listed four while there should be only two.

Other than this, the bbox specified there seem to be wrong in both cases.

For sure on GeoNode side there's place for improvements on this. There are not consistency checks at all. We should envisage to improve the methods generating those WCS download links

It is odd anyway that those values do not match any of the ones specified for the layer spatial extent
(https://www.geonode-gfdrrlab.org/layers/hazard:ts_mih_rp100/metadata_detail)

I can actually download the GeoTIFF at full resolution by using the link here

https://www.geonode-gfdrrlab.org/geoserver/wcs?service=WCS&format=geotiff&request=GetCoverage&version=2.0.1&coverageid=hazard%3Ats_mih_rp100&access_token=a7c30912212311e8b61c02a677f5a383

However, the resolution is huge. See CoverageDescription link here

https://www.geonode-gfdrrlab.org/geoserver/wcs?service=WCS&format=geotiff&request=DescribeCoverage&version=2.0.0&coverageid=hazard%3Ats_mih_rp100&access_token=a7c30912212311e8b61c02a677f5a383

How big is the original Tiff?

[stufraser1]

This remains a priority issue to solve after the v2.8 upgrade

[vdeparday]

I thought they a direct download link had been implemented to avoid going through the WCS that hasn't been working in most cases. @afabiani and @cgiovando was it included in the 2.8 release.

[vdeparday]

@afabiani and @cgiovando could we get an answer asap ? We could also almost remove the other option almost as it seems like it is never working.

[stufraser1]

@pvalsecc Please could you look at this one, running the manage command and checking it works? Alessio advises below:
if the branch has been aligned with the latest 2.8.x code, the new link "Original Data" will appear on the new uploaded Layers, among the Download Links.
In order to update the old Layers too (the ones already ingested before the update), you should be able to run the following management command from the shell:
python manage.py set_all_layers_metadata
If everything goes well, the new Download Link will appear on old instances too.
Please let me know how it goes and if you need further assistance.
Alessio.

As a test for new upload: I uploaded a new .tif layer uploaded and this seems to work, offering download options for GeoTiif, GZIP and original layer as zip file, which contains the original .tif + .sld files. https://www.geonode-gfdrrlab.org/layers/hazard:gpw_v4_population_density_2015_ca

[pvalsecc]

I tried to run manage.py set_all_layers_metadata on integration and I don't see this link added to existing layers. New one indeed have it.

[pvalsecc]

We are based on this commit: 2ea8bc6

[stufraser1]

I see it on production version: e.g.
https://www.geonode-gfdrrlab.org/layers/hazard:wgs_hazard_viet500_1km
https://www.geonode-gfdrrlab.org/layers/hazard:wgs_phil1000_1km

It is still confusing to see the other formats, and some users may click on those initially, but at least the original data link is there.

[stufraser1]

#11 keeps the issue open that there are multiple options - to be removed.