5 Levels of data reusability

[joepio]

Tim Berners-Lee's 5 star Open Data is a really cool mental model of how to think about open data quality. Check out the website if you havent seen it: https://5stardata.info/en/

But I think it doesn't quite fit what most developers would consider usable data, so it might make sense to provide a different list that focuses on data reusability.

Mostly, it lacks typed data as a deliminator - whether the data has a machine readable schema. Personally, I think this is one of the most important characteristics. It's probably one of the main reasons why SQL is so incredibly popular, or why pretty much all programming languages have things like Structs or Classes with (type-safe) properties. But not all data has this, so I think it should be a distinction layer - a separate level, if you will.

Also, we can introduce verifiability of data, powered by Atomic Commits (or any other technology that does something similar).

I'm not sure whether we should call it '5 levels', it's definitely not as catchy as '5 stars'. I'm also not fully certain about 'usability', but I think it describes what I mean pretty well.

Anyways, here's a work in progress / draft. Feel free to share ideas / criticism / thoughts!

========

5 Levels of data reusability

Not all data are created equal.
There are notable differences in how much you can do with data and how much effort it takes.
The more reusable data is, the easier it will be to use it as a developer, researcher or other type of data user.
Re-useability is about being able to transform, sort, query, serialize, modify, render and audit data without requiring too much work.

This list is inspired by Tim Berners-Lee's 5-star open data.

Level 0: proprietary data

If you don't give others the rights to read, use or modify your data, it's reusability is zero.

That's why it's important to have a license that allow others to use your data.
A good choice for a permissive option is the Open Database License.
Creative Commons licenses are also good options to clearly communicate if, and if so then how, your data is permitted to be re-used.

It's also important to use open formats (such as CSV, JSON or PNG), instead of proprietary formats (tied to specific vendors, such as PSD or RAR).

Level 1: unstructured data

Examples: images, videos, plain text

Unstructured data is the least usable.
Humans can read it, and AI / Machine Learning systems can draw more conclusions from it then ever,
but it's hard to build an actual application or graphic from only unstructured data.

Hi! I'm Joep, I'm born in 1991.

Level 2: structured data

Examples: CSV, XML, JSON, TOML, EXCEL

Structured data can be read by machines, and this allows us to do all sorts of useful things.
We can query, sort and filter.
But still, this type of data often requires human input when it needs to be processed.
And we don't have guarantees about which fields will be filled, or what their datatypes are.
One time, a birthYear can be a string, and the next time it can be a number.
Data can be structured, but still unpredictable.

{
  "name": "Joep",
  "birthYear": 1991
}

If we want predictability, we need to make it type-safe.

Level 3: type-safe data

Examples: SQL + DB SCHEMA, JSON + JSON schema, XSD + XML, RDF + SHACL, In-memory data in type-safe programming languages

Type-safe data means that every value of the data has an explicit datatype.
It is strongly typed and has a clear schema that describes which properties you can expect in a Resource.
This means that someone re-using type-safe data can know for certain that it conforms to a specification, a set of rules.
The shape of the data is predictable.
This predictability means that developers can safely re-use it in their system without worrying about missing fields or datatype errors.

Lots of software has internal type safety, especially if you use type-safe programming languages like Typescript, Kotlin or Rust.
However, when the data leaves the system, a lot of type related data is lost.
Even if this schema related information is described, the schema itself is often not machine-readable.
The best way to have type-safe data, is to describe the schema in a machine-readable format.

In SQL, we can use a DB schema. In JSON, we can add a JSON Schema file. For XML, we have XSD.

In Atomic Data, the Properties themselves (the links in the keys in JSON-AD) describe the required datatypes, which helps developers when re-using data understand what they can expect from a value.

{
  "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Agent"],
  "https://atomicdata.dev/properties/name": "Joep",
  "https://atomicdata.dev/properties/birthYear": 1991,
  "https://atomicdata.dev/properties/worksOn": "Atomic Data",
}

Level 4: browsable data

Examples: Atomic Data, properly hosted RDF

If your data is connected to other pieces of machine-readable dat, is becomes browsable, similar to how websites link to each other.
This effectively creates a web of data, and allows for a whole new way to think about the internet.
This is what allows decentralized applications, true data ownership, and a new set of applications.

{
  "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Agent"],
  "https://atomicdata.dev/properties/name": "Joep",
  "https://atomicdata.dev/properties/birthYear": 1991,
  "https://atomicdata.dev/properties/worksOn": "https://atomicdata.dev",
}

Level 5: verifiable data

Examples: Atomic Data + Atomic Commits

When your data is verifiable, other people can verify who created it and modified it.
They can use cryptography to validate signatures, which proves that one person or machine created a piece of data.

{
  "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Agent"],
  "https://atomicdata.dev/properties/name": "Joep",
  "https://atomicdata.dev/properties/birthYear": 1991,
  "https://atomicdata.dev/properties/worksOn": "https://atomicdata.dev",
  "https://atomicdata.dev/properties/previousCommit": "https://atomicdata.dev/commits/EF18751AE781",
}

[jonassmedegaard]

Unfortunately, I think there are some details that I disagree with.

Firstly, it lacks typed data
[...]
Secondly, I think it could be a bit shorter.

Ha ha, so first you argue that it should be longer, and then that it should be shorter.

I think everyone can agree with you that the 5-star is a great simplification and that more qualities are relevant, with varying importance at different contexts.

That you find that the freedom quality is unimportant is to me simply a sign that you live in a spoiled age and environment where you can take that freedom for granted. I live in similar environment as you, but others are less fortunate: e.g. access to internet is not a given, and crippled alternatives are a real threat.

Copyleft licensing (e.g. GPL, as opposed not only to proprietary licensing schemes but also to liberal licensing like MIT and BSD) is more important than ever, and less popular than ever.

[joepio]

@jonassmedegaard I agree that open licenses are important (I spend a lot of time on promoting open data), but I think it's not on the same scale / dimension / axis as usability. Also, the focus of Tim's list was on Open data, whilst this is on data usability. Lots of data is closed / private, yet should still be usable for colleagues, for example. But maybe I should add a level 0 for closed data, or data with a proprietary license, similar to Tim's model. At least I should clarify the difference in focus.

For me the most important change is adding type safety / schema requirement in there. The lack of type safety is such a big issue with linked data nowadays, and a big disconnect to how most developers practically reason about data usability.

[jonassmedegaard]

Can you give any examples of gaining the 4th star without defining a schema and therefore types for your data?

[joepio]

Can you give any examples of gaining the 4th star without defining a schema and therefore types for your data?

Examples of RDF without a schema? Well, that's pretty much all of the RDF, unfortunately. Only a small subset has SHACL / SHeX descriptions, and to make things worse, these descriptions are not trivial to discover by looking at the predicates / triples itself.

Or do you mean something else?

[jonassmedegaard]

ah, so by "types" you don't mean that a FOAF:birthday must be within rdfs:range of rdf-schema#Literal, but instead (or additionally) e.g. some rules that a FOAF:Agent can have only 0 or 1 birthdays.

If I understand that correctly, then I recommend that you describe your requirement as extended types or shapes, to avoid confusion with OWL-style RDF Schema types.

[jonassmedegaard]

It is my understanding that all RDF is tied to classes and therefore all at least loosely typed.

Seems to me that you want to require strongly typed data.

[jonassmedegaard]

I suspect that your push at promoting a qualifier list will fare better if you don't disguise it as an evolution of TimBL's list - your list is fundamentally choosing to prioritize a different set of qualities, and using another set as basis will confuse rather than help see where you are going with your list.

[joepio]

ah, so by "types" you don't mean that a FOAF:birthday must be within rdfs:range of rdf-schema#Literal, but instead (or additionally) e.g. some rules that a FOAF:Agent can have only 0 or 1 birthdays.

If I understand that correctly, then I recommend that you describe your requirement as extended types or shapes, to avoid confusion with OWL-style RDF Schema types.

rdfs:range is not to be used for shape validation, but for inferencing new triples. That's part of what makes RDF schema's so confusing, and why I would not consider all RDF to be typed.

It is my understanding that all RDF is tied to classes and therefore all at least loosely typed.

Seems to me that you want to require strongly typed data.

I think we have some confusion about type-safe. You might be right that strongly typed is a more fitting term here. I mean that developers can safely make assumptions about data shapes. They know that Persons have a BirthDate, and that that BirthDate is an ISO datetime, for example. The first relates to a Class description, the second one to a property datatype.

Would renaming it to stronly typed remove your concern?

I suspect that your push at promoting a qualifier list will fare better if you don't disguise it as an evolution of TimBL's list - your list is fundamentally choosing to prioritize a different set of qualities, and using another set as basis will confuse rather than help see where you are going with your list.

Yeah, perhaps you're right. I do want to give credits to the original list, at least, but perhaps make the difference in goals clearer.

[jonassmedegaard]

rdfs:range is not to be used for shape validation

True - it is you not I that insist types must involve shape.

rdfs:range is to be used for triple validation.

You wrote in your draft at section 3:

Type-safe data means that every value of the data has an explicit datatype, and that these datatypes can be constrained.
This means that someone re-using this data can know for certain that it conforms to a certain specification, a set of rules.
The shape of the data is predictable.

Would be lovely if there could be a promise that all data is predictable, but you are overselling above: tying a triplet to an OWL class with rdfs:range defined is a way to have an explicit datatype, and have them be constrained. But does not promise predictability.

Seems you want to mandata SHACL in your quality assessment. Which is fine - just don't call it a refinement of TimBL's quality assessment because it really is something else, and trying to bootstrap yours from his will lead to confusion (at best).

Would renaming it to stronly typed remove your concern?

Renaming to SHACL and dropping the introduction tying it to TimBL's 5-star model would remove my concerns.

Possibly rewriting to strongly typed, or to only reference TimBL's 5-star model as inspiration for this, might each remove one of my concerns - depends of how exactly it is rephrased ;-)

[jonassmedegaard]

SHACL constraints are about the structure which makes the data easily validatable in the constructs you designed the SHACL definitions for.

OWL constraints are about the meaning which makes the data agile - allows it to fit into constructs you didn't imagine ahead.

You can have a SHACL structure for a match-making construct that says "men attracts women" and "women attracts men", which works well for your business but makes your data unusable for another context of a gay match-making construct.

After a quick search and reading only the first few paragraphs so far, I encourage you to (also, I will finish doing so now myself) read this: https://www.semanticarts.com/shacl-and-owl/