Using LinkML to drive data capture

[paulmillar]

In a specific project, we're exploring the idea of having JSON files stored in a (private) GitLab repository and allowing domain-experts to maintain those JSON files. The git repo also contains the LinkML schemata for those files, with validation taking place via a GitLab CI/CD pipeline.

To make it easier for the domain experts, we would like to "hide" the JSON files by (for example) providing a web page where the user can see the current data and have the possibility to modify the data. Modifying the data would (in turn) create a corresponding branch, commit the changes in that branch and create a corresponding merge request. Handling concurrent merge requests (that target the same file) could be challenging, but (for us) this is unlikely.

I could image using the LinkML schema (either directly or by converting to JSON-Schema) to build an interactive webpage that presents the data in an intuitive way; for example, using JSON Schema annotations (like title and description) to provide semantic meaning. More generally, the schema would guide the editing process; for example, any unused fields could be offered to the user, required and recommended fields are indicated, etc...

The overall goal is that the user doesn't hand-edit JSON and the user shouldn't need to know the schema.

I've found some examples of this kind of idea in other context; e.g., the adamant project provides a way to build an interactive webpage for data capture, based on a JSON-Schema. However, I haven't yet found an example of this for LinkML described data.

I think using LinkML directly (rather than first converting to JSON-Schema) would bring certain benefits. For example, if the data capture/editing page were to be based on the data's LinkML schema then the LinkML schema (for the data) could also be edited using the same framework. This is because of the LinkML meta-model: where LinkML is described using LinkML.

However, this discussion is really to gauge whether this approach makes sense, and whether anyone is already doing something similar.

[matentzn]

Hi @paulmillar, great question — this is definitely an area a few people in the community have been thinking about, and your approach sounds very reasonable to me.

There isn't (as far as I know) a single turnkey "LinkML → interactive data entry web app" solution yet, but there are a few paths people have taken or are exploring:

JSON Schema + form libraries: I could imagine using gen-json-schema to convert your LinkML schema to JSON Schema, and then use a generic JSON Schema form library like react-jsonschema-form (RJSF) or json-editor to generate an editing UI. These tools pick up on title, description, required, enums (rendered as dropdowns), etc. It's not perfect — you lose some LinkML-specific richness in the conversion — but it gets you a working form-based editor pretty quickly, and the JSON Schema ecosystem is mature.

DataHarmonizer: DataHarmonizer provides a spreadsheet-like data entry interface and has been adapted to work with LinkML schemas directly. It's oriented towards "flat" tabular data, so it may or may not fit your use case depending on how nested/hierarchical your data is.

Tool Implementer Guide: There's a Tool Implementer Guide in the LinkML docs that goes into detail about how to build generic schema-driven applications — including guidance on using LinkML metamodel elements (title, description, pattern, required, multivalued, range) to drive form behavior, handling enums, and even adding UI hints via annotations (using the DASH vocabulary for things like single-line vs multi-line text fields). If you do end up building something custom, this is probably a good starting point.

Working with non-Python stacks: One thing the guide also covers is that if you're building a JavaScript/TypeScript frontend, you can use the LinkML generator (gen-linkml) to materialise the full schema (resolving imports, inheritance, slot_usage etc.) into a self-contained YAML/JSON that's easier to consume from a JS client, rather than reimplementing all the schema resolution logic.

Your idea of using the LinkML meta-model to also edit schemas themselves is interesting — since LinkML is described in LinkML, in principle the same form-generation approach could work for both data and schema editing. I don't think anyone has built that yet, but it's a neat idea.

There's also been an older discussion along similar lines #191.

Maybe linkml-renderer might give some pointers for the rendering side of things but seems a bit unmaintained now: https://github.com/linkml/linkml-renderer

[sierra-moxon]

I really love this idea; many projects I work on have the same needs.
I'll point out a companion projects that get into the weeds with linkml model + user interface: https://pubmed.ncbi.nlm.nih.gov/36748616/, https://github.com/cidgoh/DataHarmonizer

[jsheunis]

I'm glad I found this discussion, thanks for starting it @paulmillar :) The problem-space you describe is exactly what we (colleagues and I) have been keeping ourselves busy with for the past two years or so, with LinkML schemas being a core component of our approach. We build open source tools to support data management for large, multi-site, research consortia, often working with large and idiosyncratic datasets that nevertheless have to be described, queried, accessed, and shared. Digital independence is also an important factor in our domain, so we build or incorporate self-hostable tools by design.

What we have put together to address these (meta)data management challenges is a system of integrated tools:

• LinkML-authored foundational schema components that allow constructing use-case specific schemas; these schema components are heavily influenced by existing ontologies/vocabs/schemas like DCAT and PROV-O in order to maintain the ability to map resulting data across schema domains
• a demonstrator "research information" schema (with the emphasis on demonstrator, because the foundational components are domain-agnostic)
• a SHACL-driven web-based editor UI (shacl-vue) that renders viewers and editors for data compliant with the source schema (we use LinkML's gen-shacl to export the SHACL that drives the UI)
• a Pydantic-class-driven backend metadata store and API (FastAPI-based), dump-things-server (again using LinkML's generators)
• Forgejo-aneksajo (forgejo with git-annex support) as a collaboration hub and data repository, and as an auth provider for information sharing between front- and backend (or other services/tools and the backend)

I have a back-pocket illustration handy :)

So, basically, with the combination of these tools it means you can model your data and then get a fit-for-purpose data entry app (with data validation on entry) as well as a storage backend and server (again with schema-driven validation) for free. Well, you have to host it of course :)

We've already got this running in production for multiple use cases, one being a described here. Some examples of deployments and resulting outputs:

• The shacl-vue-driven UI for capturing/editing metadata about our research lab and collaborators: https://pool.psychoinformatics.de/ui/
• The API docs for the backend connected to this frontend: https://pool.psychoinformatics.de/api/docs
• The forgejo deployment: https://hub.psychoinformatics.de/
• Our lab website (Hugo-based), as an example of a fully metadata-driven output (we enter/edit metadata into the frontend, it gets submitted to the backend, a tool queries the backend and exports metadata into a template that updates the website): https://www-draft.psychoinformatics.de/

All of the above is constantly being maintained and improved, as we learn more from users and from our own errors. I'm curious how this approach could overlap with others' needs in the LinkML community.

(P.S. many thanks to the LinkML creators and maintainers!)

[dalito]

I have been following your approach since a while - your combination of git-annex with composable LinkML schema and SHACL is quite unique.

Another interesting SHACL-to-web-interface project is https://github.com/ULB-Darmstadt/shacl-form from the University Library Darmstadt.

[jsheunis]

Another interesting SHACL-to-web-interface project is https://github.com/ULB-Darmstadt/shacl-form from the University Library Darmstadt.

Indeed! I interacted with them a while ago: ULB-Darmstadt/shacl-form#28. The idea was to see if shacl-vue could use some shacl-form internals as a dependency. My focus went on to other things, but I think my original question still holds and might be worth looking into again.

your combination of git-annex with composable LinkML schema and SHACL is quite unique.

Yeah, our bread and butter has been datalad and git-annex for a long time already and a while back we wanted to find a way to make git-annex repos / datalad datasets more interoperable with other tools and ecosystems, and specifically to overcome the restriction of requiring git in order to interact with a datalad dataset. So we ventured deep into the linked metadata world, trying to find a way to serialize and de-serialize datalad/git-annex datasets on-demand. That's when LinkML started being useful.