Making data useful in the cloud

[jonmjoyce]

What are some capabilities we should consider to make oceanographic data more usable?

Some of the answers we've considered so far and have driven this design have been:

• Analysis-Ready, Cloud Optimized (ARCO) formats like zarr
• Catalogued, discoverable metadata to select desired datasets without parsing filenames
• Web services for natively retrieving data without relying on 3rd party installs or libraries

What other ways could we consider making access to scientific data simpler and more accessible to others?

[dpsnowden]

I don't have a suggestion to answer the question posed in the first post. But, I hope that this group is considering all oceanographic data types. We tend to gravitate toward the gridded data first, since it is often easier to deal with and the standards are more mature. But our in situ data is arguably the IOOS programs most important contribution and we need to find ways to make sure it is an equal partner in our prototyping. What is the current state of the art for cloud optimized tabular or non-gridded data?

[rsignell-usgs]

• Tabular/Time Series: Parquet or Zarr
• BNDarray data (model output): Zarr datasets or "kerchunked" collections of NetCDF4, ideally with 10-200MB chunk sizes
• Raster: Cloud-Optimized Geotiff
• Vector: Soon GeoParquet? https://github.com/opengeospatial/geoparquet#geoparquet
• Point Clouds: entwine point tiles, cloud-optimized point cloud https://copc.io/

[jonmjoyce]

Thanks for supplying that list, Rich; those are definitely the main contenders. I'll add Time Series databases as another option, which would be interesting to explore with sensors and other point observations.

@dpsnowden I've been thinking about incorporating different data types a lot and that's what's led me to some of the architecture choices I'm exploring. We know that there will be many kinds of data in many formats, and making those cloud-optimized and accessible is a good goal. I think that exploring the ability to perform data mashups across these data types (e.g. forecast model vs obs at time t, or model a vs model b vs GOES obs) will lead to more added value to the data as a whole.

[MathewBiddle]

Outside of tabular and gridded data, are there any thoughts about audio (FLACC?), image (JPEG2000? could be Geotiff?), and video formats?

[mpiannucci]

Cloud Optimized GeoTIFFS are an important part of the discussion probably. JPEG2000 is the codec used for a lot of NOAAs operational GFS and Wave Models, but as far as I know its really not cloud flexible because you cant directly index into chunks very easily without decoding the full data stream.