LinkML-Scala: fast, robust, and portable (JS/JVM/native) implementation of LinkML

[Ostrzyciel]

Hi all! :) I'd like to share something cool that we've been working on for the last few months: a new self-hosting LinkML implementation with generators!

We made LinkML-Scala because the current (Python) implementation did not meet the requirements for our projects. We needed better performance, more portability (we need LinkML to work in the browser, JVM/Java, and native binaries), and more consistent behaviors across generators, so that we can offer a smooth user experience for our devs.

LinkML-Scala covers:

• Scala runtime classes and loading instances from YAML
• SchemaView
• Schema validation
• (!) Generators: SHACL, JSON Schema, Frictionless Table Schema, RDFS, Scala, LinkML (class derivation)

Of course it's 100% open-source, under the Apache 2.0 license.

I will first show you some cool things it can do, and then explain more about its design below, in the Q&A section.

Cool things

It can compile to pure JavaScript, so you can run it in the browser.

We put together an in-browser playground – there is no backend, no WebAssembly, no pyodide or any other magic! Just pure JS :)

We have an npm package with TypeScript bindings, so you can use it in your application right away. We even have a GitHub Action that uses it! It can validate hundreds of schemas in a few seconds.

It also compiles to native binaries (Linux, macOS, Windows) – see the project's README for details. You don't need Python or Java to run and it starts up in milliseconds.

The linkml-scala CLI application is intended as a drop-in replacement of the linkml script. It doesn't have yet the same feature set, though! :)

Of course, LinkML-Scala also works on the JVM. We have packages available at Maven Central.

It's pretty fast:

We made a set of 11 benchmark datasets and compared SHACL and JSON Schema generation between our implementation and the Python one. On average, our implementation is 22.9–38.5x faster. Benchmark details.

And the last cool thing is the hardest to show – we made LinkML-Scala with a bit of a different internal philosophy than linkml. We put most of the complex logic in SchemaView and generators are as thin as possible. This allows us to have much more consistent behaviors across generators.

Questions and answers

• Why not contribute to linkml instead?
  • There are some things that Python fundamentally cannot do, like performant code or portability. Python is also not a very strict language, which makes maintaining a complex codebase challenging, at least in our opinion.
  • We also believe that a second full implementation is the thing the LinkML community needs to finally give the LinkML specification the attention it deserves. More on that in the next point...
  • We did contribute to the Python implementation in the past (e.g., fixes in the JSON-LD generator, the YARRRML generator), but we likely won't continue this work.
• How do you plan to work with the existing implementation?
  • We want to maintain a second, independent implementation, side-by-side with linkml. We want the LinkML specification to be the ground for deciding how all implementations should behave. To help with that, in the coming weeks and months we will be contributing fixes and improvements to the spec.
• Why Scala?
  • Scala is an amazingly portable language. Many people associate it with the Java Virtual Machine only, but modern Scala also compiles to JavaScript and native binaries. We use GraalVM for native compilation – the other option is Scala Native, but it wasn't as stable as Graal, so we discarded it.
  • It is possible to also make native libraries for languages like Rust, C++, or C. If you need that, please open an issue and let us know!
  • Scala also offers features like metaprogramming and an advanced type system that allowed LinkML-Scala to be both fast and safe – most errors are caught at compilation time.
• What's the feature coverage?
  • Generators: SHACL, JSON Schema, Frictionless Table Schema, RDFS, Scala, LinkML (class derivation). We are working on more – let us know in issues what you need. :)
  • Metamodel features: most of the "common stuff" like imports, ranges, enums, slot derivation, inlining. A notable (yet) unsupported feature are default values. We are constantly implementing more, but our main bottleneck is the spec, it's very incomplete. Please feel free to suggest in issues what should we work on next. :)
• How do you make it more consistent and robust?
  • We moved most of the logic from generators to SchemaView.
  • We also use a strongly-typed language where many wonky errors (e.g., accessing undefined fields in a class) are simply impossible.
  • Finally, we have a model catalog (dubbed: "model zoo") for testing the various LinkML edge cases. We employ it systematically across all generators and also cross-check with format-specific validators (e.g., JSON Schema, SHACL). This catches an unbelievable amount of bugs.
  • This model catalog is language-agnostic, so you can adopt it in the Python implementation as well. Maybe we could make it into a conformance test kit for the spec?
• What about JSON <> RDF and other conversions?
  • We have an internal solution for lifting JSONs and CSVs to RDF, and lowering RDF to JSON/CSV. It works great. We unfortunately cannot open source it at this moment. We plan to open source it later (can't name a date yet, sorry).
  • If you need this in your work, please get in touch with me, we may be able to help.
• How do you use it?
  • Mainly in a big ol' JVM server application. Also as a CLI tool, in GitHub Actions CI and (sometimes) in the browser.
  • We have a lot of closed-source generators that are based on LinkML-Scala, it's very easy to write them.
• Do you use the Python implementation?
  • We wrote the initial bootstrap generator of Scala classes in Python, but we stopped using it the moment we reached self-hosting. We do not plan to release it, it's very much out of date now.
  • In general we are not using the Python implementation internally at all (other than for comparison testing) and we are very unlikely to make any contributions to it. LinkML-Scala is designed to be a full replacement.
• What about the Java and Rust implementations?
  • Don't fit our portability requirements. Java doesn't have a clean & stable compilation path to JS, and Rust doesn't work in the JVM.
  • We also knew from the start that we want to reimplement generators, and to have very good infrastructure for implementing more generators (we need a lot of them!). This is a different scope than these project have.

What do you think?

Let us know what you think!

We are open to questions and feature suggestions. We want to make this project helpful for the community, so if something is missing, we would be happy to consider it.

[matentzn]

Thanks @Ostrzyciel :) Much appreciated! I am saying this from my personal perspective (not the perspective of the LinkML team): I would really like it if open sourcing this does not become an after-thought. This would also make this work a really valuable component to our Fall LinkML community conference/workshop #3482.

[Ostrzyciel]

Hey @matentzn, thanks! Already applied for the workshop.

Open-sourcing this is not an afterthought, it was planned from the very beginning. Like I mention in the post, we will contribute to the LinkML spec and we want to respond to community needs.

This is not the first time we are doing something like this, too. :) We maintain Jelly, an open-source binary serialization format for RDF. It now has more community implementations than first-party ones, and is used by a bunch of folks in different communities. We are also active in W3C and lead a TF in the RSP CG.

If you have any suggestions as to what else can we do to help the community and make this project (and LinkML as a whole) more sustainable, please let me know! :)

[Ostrzyciel]

(also, maybe I should state the obvious... everything I've shown here is open source, under Apache 2.0)

[Ostrzyciel]

We've written down the currently supported features and differences from linkml here: https://github.com/NeverBlink-OSS/linkml-scala/blob/main/docs/implementation_differences.md