Better documentation of metadata standard usage

[maurerle]

Description of the issue

Question regarding usage of the Metadata.

I, for example, have the tool https://github.com/open-energy-data-server/open-energy-data-server which creates a database and can continuously crawl different datasets.

The question here is, how such a tool should use the standard best.

1. use the standard to annotate each crawler by providing a JSON file next to the crawler
   • who can use this?
   • is there some more readable format like yaml/toml instead of json allowed?
   • should this standard file be updated continuously for continuously updated end dates as described in Definition of temporal resolution unclear - missing "now" option #282 ?
2. add a metadata table
   • this is available when having database access
   • should this be stored as jsonb directly in the database? As the layout has multiple layers

Ideas of solution

While we do have a standard with the OEMetadata, it is still a little unclear how this should or can be used best.

The main idea is to provide interoperability between tools, so that one can parse the standard file to figure out what is or is not included.

So this issue is about my specific question and how this can best provide compatibility to other data sources (and how it can be used), but also about the requirement to have more documentation on the usage of the standard.

Workflow checklist

• I am aware of the workflow in CONTRIBUTING.md

[jh-RLI]

Yes great observation, we have a similar issue with the energy databus where we're working on connecting the OEP.

Having a crawler tool would also be great. I have some ideas how we can come up with a good one, but i assume this will take some time until there is space for it.

Here is a bit of an overview of what we do and what can help:

You can take a look at omi. The documentation is absolutely limited and outdated. We recently reworked it completely. It is basically the core tooling for the oemetadata. Using it you can get started with automatically creating and validate (json-schema based) metadata. A week ago, i added an PR on a YAML based approach to create metadata. - I will try to complete the documentation in the next weeks.

This does not include storing the metadata. As we build on frictionless oemetadata mostly thinks of local file storage ... maybe there are solutions for frictionless which could easily adapted to the oemetadata. Nevertheless we store the metadata in the OEP as you guessed using jsonb column. But oep so far did not really implement datasets, which i also added a PR for.

Once that dataset implementation is merged you could crawl all datasets or each single resource (table) from the OEP. To find out what is available, it would require some check functions. Alternatively, we could use the JSON-LD variation of the metadata and query the metadata.

IF you have to store the metadata than it would currently be a bit of a specific setup. The OEP will not directly be able to make sure that each table resource is assigned to a dataset. This might affect crawling for some time. Yet as each table resource already got a full metadata object with just one resource one can start with that and then adapt to the datasets. Here datasets would manly help to find collections of resources as intended by the contributor.

One of the oemetadata elements the OEP autofills for each created table is the path it currently links the tables preview page. This might change to link the tables rest API endpoint to get data rows.

oemetadata/oemetadata/v2/v20/example.json

Line 15 in 5e435ab

"path": "http://openenergyplatform.org/dataedit/view/model_draft/oemetadata_table_template",

So far for now ;) think i will add more regarding your other questions later.