Hydrofabric Data Handling

[robertbartel]

Finalize design and implementation of functionality related to managing and providing hydrofabric data, with respect to the associated deliverable target.

• Determine if remotely hosted hydrofabrics may be used #122
• Confirm/standardize and document internal storage for hydrofabric data #125
• Decide on whether and when to add support for user-uploaded hydrofabrics #123

[aaraney]

It seems that we may need to reconsider this or related hydrofabric design in DMOD. The "official" (still not clear to me what that means) hydrofabric now stores its linked data in in sidecar parquet files. Previously linked data was part of the hydrofabric and included as separate tables within the geopackage hydrofabric. During today's standup, @mattw-nws, suggested that NGEN will still expect that the hydrofabric will include all of its data inside of a geopackage file. Moreover, @mattw-nws suggested that another team will be tasked with writing transformation scripts that take the sidecar parquet files and merge that information into a hydrofabric geopackage for use with NGEN.

What does this mean for DMOD?

• Do we need to add support for hydrofabric sidecar parquet files?
  • If so, it would seem we need to integrate the capability to merge parquet files into a geopackage hydrofabric.
  • How does this affect DMOD's handling of a hydrofabric? Do we need a new hydrofabric linked data format? Or for simplicity, should a hydrofabric dataset with linked data (in either sidecar or embedded form) denote the presence of that information as an attribute on the dataset? Moreover, should sidecar parquet files live within a dmod hydrofabric dataset?