From a2b837421d5093772f913737c4390146d91a0f1c Mon Sep 17 00:00:00 2001 From: anders0204 Date: Tue, 1 Apr 2025 10:03:35 +0200 Subject: [PATCH 1/7] Change intro section --- .../Data/Contributing to the EarthCODE Catalog.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md index 9cc7c29c..a22e33ed 100644 --- a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md +++ b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md @@ -3,7 +3,7 @@ order: 1 --- # Publishing Science Results -The catalog functionalities described in the previous sections are granted to any user (registered/and non-registered users). Catalog exploration and content discovery and access to the products is fully open and transparent. In this section we will describe how to make new data accessible through the catalog and make them available for the broad scientific community. +The ESA Open Science Data Catalogue provides free and open access to geoscience products for any user, without the need for authorization. By promoting FAIR and Open Science principles, this catalogue invites contributions from authorized users, enriching the scientific community with a diverse array of geospatial products. Contributions from users willing to share scientific outcomes and experiments help foster collaboration, innovation, and the continuous growth of scientific knowledge. :::tip :peanuts: STAC IN A NUTSHELL A STAC catalog is a structured, machine-readable collection of geospatial assets, described using JSON-based metadata records. It provides a standardized way to organize and link datasets, making it easy to search, access, and analyze geospatial data across different sources. A STAC catalog consists of collections (groupings of related datasets) and items (individual assets like satellite images or model outputs), with links connecting them for easy navigation and interoperability. From 0e506fdc2fe73db8d01c658ec4c1a22e3c232f22 Mon Sep 17 00:00:00 2001 From: anders0204 Date: Tue, 1 Apr 2025 10:06:25 +0200 Subject: [PATCH 2/7] Move STAC in a nutshell --- .../Data/Contributing to the EarthCODE Catalog.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md index a22e33ed..b31c5e43 100644 --- a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md +++ b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md @@ -5,9 +5,6 @@ order: 1 The ESA Open Science Data Catalogue provides free and open access to geoscience products for any user, without the need for authorization. By promoting FAIR and Open Science principles, this catalogue invites contributions from authorized users, enriching the scientific community with a diverse array of geospatial products. Contributions from users willing to share scientific outcomes and experiments help foster collaboration, innovation, and the continuous growth of scientific knowledge. -:::tip :peanuts: STAC IN A NUTSHELL -A STAC catalog is a structured, machine-readable collection of geospatial assets, described using JSON-based metadata records. It provides a standardized way to organize and link datasets, making it easy to search, access, and analyze geospatial data across different sources. A STAC catalog consists of collections (groupings of related datasets) and items (individual assets like satellite images or model outputs), with links connecting them for easy navigation and interoperability. -::: ### Who can contribute? @@ -41,6 +38,10 @@ __Typical workflow for uploading a product to OSC:__ 3. Create an OSC product entry, with a link to your Item Catalog. +:::tip :peanuts: STAC IN A NUTSHELL +A STAC catalog is a structured, machine-readable collection of geospatial assets, described using JSON-based metadata records. It provides a standardized way to organize and link datasets, making it easy to search, access, and analyze geospatial data across different sources. A STAC catalog consists of collections (groupings of related datasets) and items (individual assets like satellite images or model outputs), with links connecting them for easy navigation and interoperability. +::: + ## Step 1: Uploading your data to a remote storage To share your data, it must be __openly accessible via the cloud__. How you achieve this depends on your use case. If your data is already hosted by a cloud storage provider, there's no need to duplicate it, simply use those links. The key requirement is that the data is __publicly available and persistent__ (i.e., it won't be deleted). @@ -54,7 +55,6 @@ Feel free to [contact the EarthCODE team](https://earthcode.esa.int/contact) to ## Step 2: Creating and uploading a STAC Item Catalog - ### Description The purpose of the STAC Item Catalog is to collect metadata and references to your assets in a format that can be easily reused by other scientists and automated workflows, and displayed correctly in the Open Science Catalog. From 9d8dffea16e06e536964104571c43979beb094d9 Mon Sep 17 00:00:00 2001 From: anders0204 Date: Tue, 1 Apr 2025 10:17:52 +0200 Subject: [PATCH 3/7] Update 'Who can contribute' --- .../Contributing to the EarthCODE Catalog.md | 26 +++++++++++++------ 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md index b31c5e43..fbe0fe13 100644 --- a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md +++ b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md @@ -6,19 +6,27 @@ order: 1 The ESA Open Science Data Catalogue provides free and open access to geoscience products for any user, without the need for authorization. By promoting FAIR and Open Science principles, this catalogue invites contributions from authorized users, enriching the scientific community with a diverse array of geospatial products. Contributions from users willing to share scientific outcomes and experiments help foster collaboration, innovation, and the continuous growth of scientific knowledge. -### Who can contribute? +## Who can contribute? +Contributions to the Open Science Data Catalogue are vital for advancing FAIR Open Science Principles across ESA-funded Earth Science activities. +We would like to specifically encourage contributions from: -If you are an **ESA Project PI, Data Owner or OSC Admin** you can contribute to the content of the Open Science Catalog (OSC) in the following ways: +- Principal Investigators of ESA EO (Earth Observation Programme) funded Projects, +- Researchers, Scientists, Data Owners working on ESA-funded EO Projects, +- Principal Investigators, Researchers and Scientists from ESA Science Cluster Projects, +- ESA Technical Officers leading ESA EO Projects +- ESA-ESRIN Science Hub Members (e.g. ESA post-doctoral Research Fellows, ESA Living Planet Fellowship, ESA Visiting Scientists) +- Wider EO Research and Science Community: contact the EarthCODE team at [earth-code@esa.int](mailto:earth-code@esa.int) for more details! -- Result publishing by adding content to the OSC -- Updating content (description of Products, Projects, etc.) -- Request removal of existing entries +### You can enrich EarthCODE in several impactful ways: -**Other user requirements include:** +- **Publish Research Products:** Add new content to the Open Science Data Catalogue. +- **Update content:** Keep the descriptions and metadata of products, projects, and more up-to-date. +- **Request removals:** Ensure the catalogue remains accurate by requesting the removal of outdated or incorrect entries. -- Only users with an active GitHub account are allowed to contribute to the catalog. In case of absence of GitHub account, please [create an account first.](https://github.com/join) +To contribute, you to have an active GitHub account. If you don't have one yet, please [create an account](https://github.com/signup) to get started. -### How to Publish Results + +## How to Publish Results In order to publish your results to the OSC, you need to create valid STAC and/or OGC API Record objects (a structure of JSON files) and upload these through a Pull Request to the [open-science-catalog-metadata](https://github.com/ESA-EarthCODE/open-science-catalog-metadata/tree/main) repository. @@ -40,6 +48,8 @@ __Typical workflow for uploading a product to OSC:__ :::tip :peanuts: STAC IN A NUTSHELL A STAC catalog is a structured, machine-readable collection of geospatial assets, described using JSON-based metadata records. It provides a standardized way to organize and link datasets, making it easy to search, access, and analyze geospatial data across different sources. A STAC catalog consists of collections (groupings of related datasets) and items (individual assets like satellite images or model outputs), with links connecting them for easy navigation and interoperability. + +Learn more in the [official STAC tutorial](https://stacspec.org/en/tutorials/intro-to-stac/). ::: ## Step 1: Uploading your data to a remote storage From 2d4787a294e9b26a5b4cb2bbc14045d1f8a529f2 Mon Sep 17 00:00:00 2001 From: anders0204 Date: Tue, 1 Apr 2025 12:28:31 +0200 Subject: [PATCH 4/7] WIP: add review on how to publish results --- .../Contributing to the EarthCODE Catalog.md | 24 ++++++++----------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md index fbe0fe13..b07729d8 100644 --- a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md +++ b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md @@ -27,24 +27,20 @@ To contribute, you to have an active GitHub account. If you don't have one yet, ## How to Publish Results +Open Science Data Catalogue provides functionalities of data discovery and access to any user/ Earth Observation Scientist, with or without GitHub account. Open Science Data Catalogue offers easy and transparent way of accessing and exploitation of wide range of the latest EO project's results (research outcomes). Implementation of structured and harmonized way of describing result's metadata facilitates publication and data discovery process. +The Open Science Data Catalogue is built on the open-source STAC Browser application, that allows users to explore various inter-linked elements: themes, variables, eo-missions, projects, products, workflows and experiments described in format of JSON files structured in a STAC Catalog. -In order to publish your results to the OSC, you need to create valid STAC and/or OGC API Record objects (a structure of JSON files) and upload these through a Pull Request to the [open-science-catalog-metadata](https://github.com/ESA-EarthCODE/open-science-catalog-metadata/tree/main) repository. +**To publish your scientific outcomes you need to provide following:** -In fact, in most cases you would need to create two separate STAC or Record objects: -1. __Item catalog__ — A self-contained STAC catalog, typically hosted in your own public GitHub repository (__Accessible__). -2. __Product, Workflow, or Experiment entry__ — The metadata record you upload to the OSC, linking to related Projects, Themes, Variables, and your external Item Catalog (__Findable__). +- valid STAC and/or OGC API Record objects (JSON files) +- create Pull Request on [open-science-catalog-metadata](https://github.com/ESA-EarthCODE/open-science-catalog-metadata/tree/main) repository to ingest new entry. +[Here add a part STAC in a nutshell] -::: details Why keep them separate? -Earth observation research can generate large amounts of datasets with extensive metadata. Keeping data assets separate ensures faster __loading, less clutter, and a consistent access method__, while the OSC remains focused on structured metadata. -::: - -If you are new to using STAC, git, or both, this can be an intimidating process, but luckily there are many tools available to more or less automate this process depending on your setup. Check out the [OSC examples](../../../examples/index-1) for an overview. - -__Typical workflow for uploading a product to OSC:__ -1. Upload your data to a persistent remote storage (e.g. ESA's Project Results Repository (PRR), an object storage (S3) bucket, Zenodo, etc.). -2. Create the Item Catalog, with links to your assets in the remote storage. -3. Create an OSC product entry, with a link to your Item Catalog. +To share your outcomes with wide research world, you would need to prepare your research experiments according to the Open Science Catalogue structure and adopted strategy for data sharing: +- Create your Product Package (Research Experiment): Dataset (files), Code and Documentation should be accessible* (expalin what accessible means: (e.g. : data should be available for discovery and expoitation from remote storage (e.g. ESA's Projects Results Repository [link here to its descirption + information that it can be provided], object storage (S3 bucket), Zenodo, CEDA, dataverse or other). +- Create valid STAC Objects from your Dataset: A self-contained STAC Catalog with individual items (in JSON format) hosted in your own (or institutional) public GitHub repository (to be made accessible*) +- Describe your Research Experiment in Open Science Data Catalogue: to ensure the discoverability of your products, create and upload specific entries to describe your Dataset, Workflow, Experiment by linking them to related Projects, Themes, Variables (with access to STAC Catalog (point 2) :::tip :peanuts: STAC IN A NUTSHELL A STAC catalog is a structured, machine-readable collection of geospatial assets, described using JSON-based metadata records. It provides a standardized way to organize and link datasets, making it easy to search, access, and analyze geospatial data across different sources. A STAC catalog consists of collections (groupings of related datasets) and items (individual assets like satellite images or model outputs), with links connecting them for easy navigation and interoperability. From 642b02e4b8842871b63b70677b8e4af05f85289c Mon Sep 17 00:00:00 2001 From: anders0204 Date: Tue, 1 Apr 2025 15:25:17 +0200 Subject: [PATCH 5/7] Rewrite 'How to publish results' --- .../Contributing to the EarthCODE Catalog.md | 55 +++++++++++++++---- 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md index b07729d8..1686cd7f 100644 --- a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md +++ b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md @@ -27,20 +27,55 @@ To contribute, you to have an active GitHub account. If you don't have one yet, ## How to Publish Results -Open Science Data Catalogue provides functionalities of data discovery and access to any user/ Earth Observation Scientist, with or without GitHub account. Open Science Data Catalogue offers easy and transparent way of accessing and exploitation of wide range of the latest EO project's results (research outcomes). Implementation of structured and harmonized way of describing result's metadata facilitates publication and data discovery process. -The Open Science Data Catalogue is built on the open-source STAC Browser application, that allows users to explore various inter-linked elements: themes, variables, eo-missions, projects, products, workflows and experiments described in format of JSON files structured in a STAC Catalog. -**To publish your scientific outcomes you need to provide following:** +The **Open Science Data Catalog** is a publicly accessible platform that enables anyone—whether or not they have a GitHub account—to **discover and access Earth Observation research**. It provides a transparent and structured way to explore the latest results from EO projects by organizing metadata in a consistent and harmonized format. -- valid STAC and/or OGC API Record objects (JSON files) -- create Pull Request on [open-science-catalog-metadata](https://github.com/ESA-EarthCODE/open-science-catalog-metadata/tree/main) repository to ingest new entry. -[Here add a part STAC in a nutshell] +Built on the open-source **STAC Browser**, the catalog allows users to browse and explore interlinked elements such as **themes, variables, EO missions, projects, products, workflows, and experiments**, all described using **STAC-compliant JSON files**. -To share your outcomes with wide research world, you would need to prepare your research experiments according to the Open Science Catalogue structure and adopted strategy for data sharing: +--- + +### What You Need to Publish + +To publish your scientific results to the Open Science Catalog, you must: + +- Create valid **STAC** and/or **OGC API Record** objects (in JSON format). +- Submit a **Pull Request** to the [open-science-catalog-metadata-**staging**](https://github.com/ESA-EarthCODE/open-science-catalog-metadata-staging/tree/main) repository with your new entry. + +--- + +### Preparing Your Research for Publication + +To make your research outcomes accessible and discoverable by the broader scientific community, follow these steps: + +1. **Create Your Product Package (Research Experiment)** + This includes: + - **Dataset files** + - **Code** + - **Documentation** + + All of these should be *accessible*, meaning they are stored on **remote, persistent storage** that allows discovery and access. Examples include: + - ESA’s Project Results Repository (PRR) + - S3-compatible object storage (e.g. ESA S3 Bucket) + - Zenodo, CEDA, Dataverse, or other persistent archives + + ::: details Requesting PRR Storage + At the moment, requests to store data on ESA PRR is done by the ESA PLES engineering team. If you need to request permanent storage, contact the team at [earth-code@esa.int](mailto:earth-code@esa.int) + ::: + +2. **Generate a Self-Contained STAC Catalog** + - Use tools like [`stactools`](https://stactools.readthedocs.io/en/stable/), [`rio-stac`](https://github.com/developmentseed/rio-stac), or [`PySTAC`](https://pystac.readthedocs.io/en/stable/) to generate a STAC Catalog. + - Host the resulting JSON files (Catalog + Items) in a **public GitHub repository** (or institutional equivalent). + ::: warning Important + Make sure the Catalog uses **relative paths** and points to remote asset URLs! + ::: + + +3. **Describe Your Research in the Open Science Catalog** + - Create entries that describe your **dataset, workflow, and experiment**. + - Link them to relevant **projects, variables, themes, and EO missions**. + - Include a `related` link to your external STAC Catalog to ensure it is discoverable from the Open Science Catalog. -- Create your Product Package (Research Experiment): Dataset (files), Code and Documentation should be accessible* (expalin what accessible means: (e.g. : data should be available for discovery and expoitation from remote storage (e.g. ESA's Projects Results Repository [link here to its descirption + information that it can be provided], object storage (S3 bucket), Zenodo, CEDA, dataverse or other). -- Create valid STAC Objects from your Dataset: A self-contained STAC Catalog with individual items (in JSON format) hosted in your own (or institutional) public GitHub repository (to be made accessible*) -- Describe your Research Experiment in Open Science Data Catalogue: to ensure the discoverability of your products, create and upload specific entries to describe your Dataset, Workflow, Experiment by linking them to related Projects, Themes, Variables (with access to STAC Catalog (point 2) +By following these steps, your research becomes part of a broader ecosystem of reusable, discoverable, and connected scientific outputs. :::tip :peanuts: STAC IN A NUTSHELL A STAC catalog is a structured, machine-readable collection of geospatial assets, described using JSON-based metadata records. It provides a standardized way to organize and link datasets, making it easy to search, access, and analyze geospatial data across different sources. A STAC catalog consists of collections (groupings of related datasets) and items (individual assets like satellite images or model outputs), with links connecting them for easy navigation and interoperability. From 48de895ff1e9eebb094eee5097c966a0c2529ab2 Mon Sep 17 00:00:00 2001 From: anders0204 Date: Tue, 1 Apr 2025 15:25:38 +0200 Subject: [PATCH 6/7] Remove 'Stac in a Nutshell' --- .../Data/Contributing to the EarthCODE Catalog.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md index 1686cd7f..1c989d39 100644 --- a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md +++ b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md @@ -77,12 +77,6 @@ To make your research outcomes accessible and discoverable by the broader scient By following these steps, your research becomes part of a broader ecosystem of reusable, discoverable, and connected scientific outputs. -:::tip :peanuts: STAC IN A NUTSHELL -A STAC catalog is a structured, machine-readable collection of geospatial assets, described using JSON-based metadata records. It provides a standardized way to organize and link datasets, making it easy to search, access, and analyze geospatial data across different sources. A STAC catalog consists of collections (groupings of related datasets) and items (individual assets like satellite images or model outputs), with links connecting them for easy navigation and interoperability. - -Learn more in the [official STAC tutorial](https://stacspec.org/en/tutorials/intro-to-stac/). -::: - ## Step 1: Uploading your data to a remote storage To share your data, it must be __openly accessible via the cloud__. How you achieve this depends on your use case. If your data is already hosted by a cloud storage provider, there's no need to duplicate it, simply use those links. The key requirement is that the data is __publicly available and persistent__ (i.e., it won't be deleted). From acaa8551f91b4db65c8da1dc70b3c4801d0b76bc Mon Sep 17 00:00:00 2001 From: anders0204 Date: Tue, 1 Apr 2025 15:40:53 +0200 Subject: [PATCH 7/7] Expand 'Step 1' --- .../Contributing to the EarthCODE Catalog.md | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md index 1c989d39..15c8011c 100644 --- a/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md +++ b/pages/Technical Documentation/Data/Contributing to the EarthCODE Catalog.md @@ -58,9 +58,6 @@ To make your research outcomes accessible and discoverable by the broader scient - S3-compatible object storage (e.g. ESA S3 Bucket) - Zenodo, CEDA, Dataverse, or other persistent archives - ::: details Requesting PRR Storage - At the moment, requests to store data on ESA PRR is done by the ESA PLES engineering team. If you need to request permanent storage, contact the team at [earth-code@esa.int](mailto:earth-code@esa.int) - ::: 2. **Generate a Self-Contained STAC Catalog** - Use tools like [`stactools`](https://stactools.readthedocs.io/en/stable/), [`rio-stac`](https://github.com/developmentseed/rio-stac), or [`PySTAC`](https://pystac.readthedocs.io/en/stable/) to generate a STAC Catalog. @@ -77,17 +74,24 @@ To make your research outcomes accessible and discoverable by the broader scient By following these steps, your research becomes part of a broader ecosystem of reusable, discoverable, and connected scientific outputs. -## Step 1: Uploading your data to a remote storage +## Step 1: Make Your Data Accessible + +To contribute to the Open Science Catalog, your research data must be openly accessible and persistent. Begin by preparing your **Research Experiments**, which includes: +- **Data files** , which will be added to the Item Catalog. +- **Workflow** (e.g., Jupyter Notebooks, Python scripts, CWL records) +- **Documentation** (e.g., links to peer-reviewed publications or public product descriptions) -To share your data, it must be __openly accessible via the cloud__. How you achieve this depends on your use case. If your data is already hosted by a cloud storage provider, there's no need to duplicate it, simply use those links. The key requirement is that the data is __publicly available and persistent__ (i.e., it won't be deleted). +All of these components should be **publicly available and stored on persistent platforms**—meaning they should not be tied to your local machine or a temporary location. -If your data isn't in the cloud or its persistence is uncertain, the recommended process is to request access to the official -[ESA PRR](https://eoresults.esa.int/data_provider_guide.html#) and upload it there following the described procedure. -After requesting a data provider account you need to request a PRR collection which is used in the next steps as the STAC Item Catalog. +If your data is already hosted on a reliable cloud storage provider (e.g., Zenodo, CEDA, an institutional repository, or S3), you can use those links directly. -Alternatively there is also an object storage bucket maintained by the EarthCODE team that can be used to persistently store data. -Feel free to [contact the EarthCODE team](https://earthcode.esa.int/contact) to discuss available options. +If your data is not yet in the cloud or its persistence is uncertain, we recommend uploading it to the official **ESA Project Results Repository (PRR)**. To do this: +1. Request a data provider account. +2. Then, request a PRR collection, which will be used in later steps to define your STAC Item Catalog. +::: details Requesting PRR Storage +At the moment, requests to store data on ESA PRR is done by the ESA PLES engineering team. If you need to request permanent storage, contact the team at [earth-code@esa.int](mailto:earth-code@esa.int) +::: ## Step 2: Creating and uploading a STAC Item Catalog ### Description