Empty NetCDF file was created

[PaulMousset]

I tried to ingest a granule from Sentinel-2.

Configuration :

• s2-config.txt
• agdc-metadata.txt

The file creation is performed without problems. But, when I read bands containing in the NetCDF file (read each band as array), all value are -999 (due to my nodata configuration).

I have verified that the file is not corrupt and values in JP2 files aren't empty.

Thanks

[v0lat1le]

Seems to work on my end...
Are you using latest development branch? What platform are you running on?

[PaulMousset]

I tried on Windows 7 & CentOS 6.5
Data Cube, version 1.0.3+82.gb3191ab

I will retry with the lastest version.

[v0lat1le]

Some major changes have been merged into develop branch the past few days. Ingesting process is a bit different now. Check out http://agdc-v2.readthedocs.io/en/develop/. Alternatively you could get the stable branch, but the next stable release will include the changes

[PaulMousset]

Thanks. I try this new version. I have a question : When I Ioad a product definition, why some informations are not indexed ? Like measurements and there spectral_definition.

Finally, the Datacube work for my sentinel data 👍

[v0lat1le]

Good to hear you got it working!

Not sure what you mean by not all of the information being indexed... Could you add a code example so we can try to reproduce?

[PaulMousset]

I loaded product definitions using config files supplied in docs/config_samples/dataset_types.
For exemple, when I add ls5_scenes, ls5_nbar_scene measurement aren't stored.
Moreover, is it normal that some definition fields are empty ?

[v0lat1le]

That looks really strange... The definitions that are missing would be significantly larger than the ones that are there. Are you sure it's not a SQL client display issue?

What happens if you run:

from datacube import Datacube
print Datacube().list_measurements()

[PaulMousset]

You are right ! It's a bug with PgAdmin. The command display all data !

[PaulMousset]

I tried to play with the API, so I look the Notebooks.

First, I stored Landsat 5 product definition and dataset (provided here).

I loaded my DataCube fondig :

dc = datacube.Datacube(config='./datacube.conf', app='dc-example')
>>> Datacube<index=Index<db=PostgresDb<engine=Engine(postgresql://datacube:***@localhost:5432/datacube)

Then, I loaded a prduct :

dc.load(product='ls5_nbar_albers')

But an error has occurred :

/home_local/pmousset/agdc-v2-develop/datacube/storage/storage.py in wheres_my_band(self, src, time)
    176         dist = float('+inf')
    177         for i in range(1, src.count+1):
--> 178             v = float(src.tags(i)['NETCDF_DIM_time'])
    179             if abs(sec_since_1970 - v) < dist:
    180                 idx = i

KeyError: 'NETCDF_DIM_time'

I also tried with Sentinel-2 data. Issue is the same.

DataCube Version : version 0+unknown
(Version Retrieved June 9)

[v0lat1le]

Could be a gdal version issue...

could you run: python -c 'from osgeo import gdal; print gdal.__version__'

Also if you can find the created netcdf files and do:
ncdump -h <filename>
gdalinfo <filename>
gdalinfo NETCDF:<filename>:blue
on one of them and attach the output, it'd help us figure out what's wrong

[PaulMousset]

python -c 'from osgeo import gdal; print gdal.__version__' > 2.0.0

ncdump -h <filename> > ncdump.txt
gdalinfo <filename> > gdalinfo.txt
gdalinfo NETCDF:<filename>:blue > gdalinfo_blue.txt

[v0lat1le]

This file looks ok. In gdalinfo_blue.txt there's NETCDF_DIM_time field, whuch we're looking for...

If you enable logging by:

import logging
logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s', level=logging.INFO)

There should be more info printed out. It should point to the actual file that fails

[PaulMousset]

It returns this issue:

2016-06-14 10:32:28,050 ERROR Error opening source dataset: file://NetCDF:/data/LS5_TM_NBAR/LS5_TM_NBAR_3577_15_-40_19920323231413000000.nc:blue
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home_local/agdc-v2-develop/datacube/api/core.py", line 193, in load
    return self._get_dataset(**indexers)
  File "/home_local/agdc-v2-develop/datacube/api/core.py", line 216, in _get_dataset
    dataset = self.product_data(sources, geobox, measurements.values())
  File "/home_local/agdc-v2-develop/datacube/api/core.py", line 332, in product_data
    fuse_func=fuse_func)
  File "/home_local/agdc-v2-develop/datacube/storage/storage.py", line 105, in fuse_sources
    reproject(sources[0], destination)
  File "/home_local/agdc-v2-develop/datacube/storage/storage.py", line 72, in reproject
    with source.open() as src:
  File "/home_local/.conda/envs/datacube/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home_local/agdc-v2-develop/datacube/storage/storage.py", line 170, in open
    raise e
  File "/home_local/agdc-v2-develop/datacube/storage/storage.py", line 160, in open
    bandnumber = self.wheres_my_band(src, self.time)
  File "/home_local/agdc-v2-develop/datacube/storage/storage.py", line 178, in wheres_my_band
    v = float(src.tags(i)['NETCDF_DIM_time'])
KeyError: 'NETCDF_DIM_time'

[v0lat1le]

Maybe it's rasterio version...
Could you please run this:

import rasterio
print rasterio.__version__
with rasterio.open('file://NetCDF:/data/LS5_TM_NBAR/LS5_TM_NBAR_3577_15_-40_19920323231413000000.nc:blue') as ds:
    print ds.tags(1)

Any way you could upload the .nc file as well?

[PaulMousset]

print rasterio.__version__

Version : 0.36.0

with rasterio.open('file://NetCDF:/data/LS5_TM_NBAR/LS5_TM_NBAR_3577_15_-40_19920323231413000000.nc:blue') as ds:
    print ds.tags(1)
<ipython-input-83-3047d970ccb8> in <module>()
      1 import rasterio
      2 with rasterio.open('file://NetCDF:/data/LS5_TM_NBAR/LS5_TM_NBAR_3577_15_-40_19920323231413000000.nc:blue') as ds:
----> 3     print(ds.tag(1))

AttributeError: 'rasterio._io.RasterReader' object has no attribute 'tag'

The .nc file : ls5.zip

[v0lat1le]

the last line should be print(ds.tags(1))

I get this when I run it:

0.35.1
{u'_FillValue': u'-999',
u'grid_mapping': u'crs',
u'alias': u'band_1',
u'NETCDF_VARNAME': u'blue',
u'long_name': u'Nadir BRDF Adjusted Reflectance 0.45-0.52 microns (Blue)',
u'units': u'1',
u'NETCDF_DIM_time': u'701392453'}

[PaulMousset]

Oops, I read too fast ! Me, I get this :

{'NETCDF_DIMENSION_time': '701392453',
'NETCDF_VARNAME': 'blue',
'units': '1',
'grid_mapping': 'crs',
'NETCDF_time_units': 'seconds since 1970-01-01 00:00:00',
'alias': 'band_1',
'_FillValue': '-999',
'long_name': 'Nadir BRDF Adjusted Reflectance 0.45-0.52 microns (Blue)'}

[v0lat1le]

Somehow your version has NETCDF_DIMENSION_time but we're looking for NETCDF_DIM_time... I'll need to investigate what's up with that.

For now I've commited a workaround for that issue

[PaulMousset]

Thank you ! I'll try soon.

[v0lat1le]

Ok it looks like a change from gdal 1.9 to gdal 1.10.
Could you please run

import rasterio
print(rasterio.__gdal_version__)

when you get a chance.

[PaulMousset]

I have 1.9.2 version

[PaulMousset]

On the server, there are gdal 1.9.2 and gdal 2.0.0 .
I have a virtual environment for DataCube with gdal 2.0.0. But rasterio seems use gdal 1.9.2