Modify geostreams API to support polygons

[max-zilla]

We want to support geometry/polygons in PostGIS for geostreams API, i.e. to store field of view polygons instead of a single point.

The geospatial extractor already generates the polygons, but @robkooper thought the PostGIS db under clowder is configured for "geography" instead of "geometry".

@yanliu-chn tagging you on this issue so we can figure out what needs to be updated and by who. Hopefully this won't be a big change.

[robkooper]

Created matching clowder issue : https://opensource.ncsa.illinois.edu/jira/browse/CATS-643

[max-zilla]

I am starting on this. Some discussion here: https://opensource.ncsa.illinois.edu/confluence/display/CATS/PostGIS+and+the+Geostreams+API?showComments=true

We can consider two pieces here:

• allow Clowder API to support geometries beyond Point. This is possible using either 'geometry' or 'geography' data type.
• do we want to store our data in 'geography' type as it currently does, or change to 'geometry'? geometry does not require conversion to WGS84 lat/lon and is likely faster, but would require a database migration and does not scale to state/national/global scales accurately.

We can do part 1 (and thus allow polygons for datapoints) without doing part 2, and do part 2 down the road if we need to for performance reasons. That would be my recommendation.

[dlebauer]

Makes sense to start with part 1 as long as someone knows how to convert from gantry coordinates to WGS-84. But we have planned to use the geometry data type from the beginning because of all of its features.

Geometry has more functionality and is faster. The difference in functionality is outlined in this matrix such as:

• the ability to import from and export to standard geospatial data types (kml, geojson, wkt, gdal raster)
• the ability to compute metrics of interest from polygons, 3D shapes and point clouds

The scales of relevance for our project are mm to km. We don't need to accurately align distant sites. We do need to cross-reference data from different coordinate systems on small scales: the gantry data is collected with mm accuracy and the tractor / drone data at cm scales. Cross-site (e.g. AZ, IL, KS) analyses will not depend on accurate locations of the sites.

Furthermore, we are faced with ingesting geospatial metadata from multiple coordinate systems. PostGIS already supports this, including defining new coordinate systems and translating among coordinate systems. For example:

• the gantry coordinate system with x increasing to the left
• the as-yet undefined reversed x gantry coordinate system
  • these are aligned with the sensors, so a raster will be 'square' on the coordinate systems
  • these are close to being aligned with NS/EW, but not sure if we would need to re-project a raster to this system
• Our mapping from gantry coordinates to WGS84 has errors, and will likely change.
• The precision tractors and drones in AZ use a GPS coordinate system with incorrect calibration - it is a few meters off but can't be modified because it is a commercial system used to guide tractors and drones around the region

Using geometries will allow us to define the different coordinates and ingest geospatial metadata directly into a common table without an extra translation layer. Data can be queried / exported in any coordinate system.

If the alternative is re-implementing all of this conversion functionality, it will add time and complexity to our pipeline, and will be more error prone / less stable than a mature software like PostGIS. If our assumptions / definition of one coordinate system changes, or we need a new one, we only have to define / modify in one place rather than identify and recompute all of the affected columns.

This stuff is a headache and it seems less of a headache to let postgis deal with it. Maybe the solution would be to add another geometry field rather than update all of the other Clowder users' databases. But it seems geometry supports all geography functions but not viceversa.

[max-zilla]

@yanliu-chn @robkooper we should discuss PostGIS a bit. Specifically storing multiple projections in a single geometry. If PostGIS is not sufficiently scalable, should we consider alternatives?

[yanliu-chn]

Hi Max,

Nathan Casler from our lab gave me the answer: you can store multiple geometry columns (each project as one column) in the table and when you retrieve from the table, you specify which geometry column to use.

A related discussion online is: http://gis.stackexchange.com/questions/176263/can-a-postgis-table-or-view-have-two-geometry-columns

[max-zilla]

Here is a pull request that adds support for polygons/GeoJSON for streams/sensors/datapoints:
https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/clowder/pull-requests/1051/overview

This for now will let our extractors post GeoJSON - still geography, so still needs WGS 84. @caicai89- FYI. There's an example of how it works in the pull request.

[dlebauer]

@max-zilla what is the status of this issue?

[max-zilla]

@dlebauer https://opensource.ncsa.illinois.edu/bitbucket/projects/CATS/repos/clowder/pull-requests/1051/overview

Still waiting on reviewers. I'll tag@robkooper here in case he can accelerate things, I've fixed all the known issues with the PR as of a couple weeks ago.

[robkooper]

@max-zilla can you deploy this on terra-clowder-dev. Biggest thing we have to look at with pull request that it does not break geodahboard.

[max-zilla]

@robkooper I can deploy to terra-clowder-dev but I haven't used/deployed geodashboard before, will need to track down some help for that. Will likely wait until Jan 3 given the other things I'm trying to get out this week.

[robkooper]

Don't deploy geodashboard, @lmarini is checking out to make sure your changes work with geodashboard. Just deploy this code with search in clowder so people can test it with Terra.