Indexing of complex data - Cint16

[clausmichele]

I am trying to index Sentinel-1 SLC rasters, which contains data of type CInt16 (complex numbers), however opendatacube does not support this data type yet. QGIS and GDAL are able to open the GeoTiff I'm using for testing.
Are there plans to add this feature? Thank you.

datacube.utils.documents.InvalidDocException: 'cint16' is not one of ['float16', 'float32', 'float64', 'int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64']

Failed validating 'enum' in schema['properties']['measurements']['items']['properties']['dtype']:
    {'enum': ['float16',
              'float32',
              'float64',
              'int8',
              'int16',
              'int32',
              'int64',
              'uint8',
              'uint16',
              'uint32',
              'uint64']}

On instance['measurements'][0]['dtype']:
    'cint16'

Environment information

• Which datacube --version are you using?
  1.8

[Kirill888]

@clausmichele Can you share an example file.

So, dtype is about the numpy side of things and numpy does not have an integer complex type as far as I am aware.

Can you open your test file with rasterio and report if it works and what .dtype does it use. Basically we rely on rasterio to perform data loading so if it works there it should be easy enough to support on datacube side.

CInt16 is mentioned here:

https://github.com/mapbox/rasterio/blob/master/rasterio/dtypes.py

But I never encountered a file with that type, so not sure if it will work with rasterio. The dtype would probably be complex64 which a numpy name for float32,float32 complex number. And we might have to add it to this schema

https://github.com/opendatacube/datacube-core/blob/develop/datacube/model/schema/dataset-type-schema.yaml#L48

However there might be more implications as I'm not sure how well projection change is supported for those types.

[Kirill888]

After speaking with remote sensing folks, who are also working with that data, it seems that a bigger problem is lack of auto-rectification. My understanding is that Sentinel-1 is using Ground Control Points for Geo-referencing, and this is not something datacube currently supports. There a lot of assumptions throughout the code base that there is a linear mapping from pixel space to projected space, and there is currently no rectification on load.

You might have to reproject this data into some regular grid(s) outside of datacube first.

[clausmichele]

Here is some details about the data. I see that for this particular case the major problem is that it is not georeferenced.

import rasterio
ds = rasterio.open('s1a-ew1-slc.tiff')

> /home/user/anaconda3/envs/dask/lib/python3.8/site-packages/rasterio/__init__.py:219: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)

band1 = ds.read(1)
band1[0,0].dtype

> dtype('complex128')

ds.bounds

> BoundingBox(left=0.0, bottom=13944.0, right=8332.0, top=0.0)

[Kirill888]

That looks like a bug in rasterio it should really remap CInt16 to complex64, but yeah bigger problem is use of GCPs.

[clausmichele]

@Kirill888 any advance on this?

[Kirill888]

Complex dtypes are now allowed by product schema, whether this will work with your data I don't know

[clausmichele]

Is it possible to index data which is not yet georeferenced? I solved the problem of complex data splitting real and imaginary part as two bands, but I still need to index it with ODC, but I don't have coordinates. xarray doesn't complain with the data, but just uses x and y range as coordinates. Do you see a solution?

[Kirill888]

I'm closing this issue as comlpex data types are now supported. For GCP support we should open a different issue. There is currently no plan for implementing this within 1.8 series.