-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement CRUD GUI for STAC Catalogs, Collections, and Items, Including an Asset Uploader #127
Comments
🙏 just FYI there is a draft OGC feature extension for most of this https://docs.ogc.org/DRAFTS/20-002.html and there is a STAC extension https://github.com/radiantearth/stac-api-spec/tree/master/ogcapi-features#transaction-extension
IMHO, this is (and will be) really complex to implement. Doing 1-4 will already be great. Adding 5, will mean that you have to take care of the storage type and some 🕳️ |
@zacharyDez A few high level thoughts here and then some lower level thoughts 😄 The creation of STAC metadata for most datasets/formats is relatively complex process that normally requires some specialized knowledge about the dataset and its intended use cases. Like you mentioned, to avoid manual operations we've been trying to capture this specialized domain specific knowledge in easier to use modules via stactools-packages. Let's look in detail at each of the CRUD GUI proposed requirements
Additional considerations -
I like a lot of the concepts here but I wonder if we're at too early a stage to build a GUI around STAC item manipulation. Given our limited dev resources perhaps we can focus first on implementing the STAC transaction extension as outlined in the eoAPI transition roadmap so that we can have parity between our cdk and k8s implementations when it comes to ingestion APIs. Then we could consider building a common GUI on top of the API provided by the transaction extension if there is sufficient demand. One area of great interest where manual intervention is necessary and a GUI would be immediately valuable is around the STAC Catalog specification which would allow a user to potentially group any arbitrary set of items together into a cohesive unit. We had explored this concept on several projects where an item (which may only have one parent collection) can be a member of multiple catalogs. A GUI where a user could select and group items without defined criteria into a catalog would be awesome. Unfortunately, this would only be supported in static STAC implementations at the moment as the catalog concept is not modeled in pgSTAC as I understand it. |
@sharkinsspatial ; Thanks for the detailed message about the CRUD GUI for eoAPI. It was super helpful for my understanding and very detailed. You are correct that we are still at the point of working on an implementation. Your comment raised many questions for me that we can keep tabs on while focusing on other higher-priority issues. I think that for specific EO data, you are correct that we need to observe the dataset programmatically to populate the STAC items. Organizations cataloging smaller datasets, potentially vector ones, would correspond to end-users most interested in an interface to manage their catalogs. I still feel we could make that access happen via the GUI for creation. Access to the underlying granules to generate the bbox, and geometry should not be an issue. A lot of organizations have custom metadata schemas that they need to convert to STAC at the time of ingestion - I don't think it is entirely unreasonable that we could be able to map these custom schemas to the STAC specification. I see the main challenge in managing the potential collection vs. item-level metadata translation. The main update task that I can think of is updating the description which is likely to happen at the collection level and not the item level. Perhaps some manual QA and updates could be justifiable in some cases? It's hard to see what changes would not require modifying the underlying data pipeline to avoid reproducing. About the upsert logic, do you see this being a blocker to implementing any versioning, given we are overwriting items with identical IDs? I recognize we can create snapshots, but a changeset seems like a more lightweight approach. I'm interested to hear more about grouping items into cohesive catalogs; let's set some time to chat about previous implementations we've worked on. |
An example of a metadata catalog software with editing UI is GeoNetwork - I don't have experience with it and it is probably a different ball game (INSPIRE etc 😱) but just stumbled over this and maybe these references are somehow helpful... |
We have this on our roadmap for VEDA: https://github.com/NASA-IMPACT/veda-architecture/issues/336 Looks like @oliverroick actually implemented it in https://github.com/developmentseed/stac-admin? |
Some of it. The functionality of stac-admin is still quite limited. So far you can:
All other requirements outlined above would be great additions to what we have so far; happy to work with you on that. |
FYI a collection transaction extension is in WIP with the same CRUD approach as per items. |
fyi: There are also plans to add the missing Create, Update, Delete operations via the well-known Item and Collection Transaction extensions to STAC Browser. |
Background
STAC (SpatioTemporal Asset Catalog) provides a standardized way to describe geospatial information, facilitating the discovery and usage of spatial assets. In the eoAPI project, we need to manage STAC catalogs, collections, and items effectively. A few end-users have specifically requested this functionality, emphasizing the need for user-friendly interactions with STAC components. In most cases, we actually want to avoid having manual operations
Issue
We require a user-friendly interface to perform CRUD operations (Create, Read, Update, Delete) on STAC catalogs, collections, and items, and to load the underlying referenced assets of a STAC item based on the minimum use case described in issue #5.
Requirements
The initial scope discussed with @batpad only addressed the Update capabilities, but it would be great to have a more complete tool to manage STAC catalogs visually.
Create:
Read:
Update:
Delete:
Asset Uploader:
Additional Considerations
The text was updated successfully, but these errors were encountered: