Ability to save "schema templates": schemas where some values are left empty

[wetneb]

Wikibase communities generally establish some data modelling conventions, which users are asked to follow when importing data.
For instance, a book on Wikidata will generally have a certain set of statements on it (title, author, date of publication…).

When importing data in a Wikibase instance, users need to have the knowledge of those modelling conventions. If they do not have this knowledge, a blank schema page can be quite daunting.

Proposed solution

We should make it possible for users to define "schema templates" which represent a prototypical Wikibase schema for an import in a certain domain. Those schema templates would be incomplete Wikibase schemas, in the sense that some statement values could be left blank, for instance. After loading such a template, the user could then drag and drop the columns from their project to the appropriate locations in the schema.

For the implementation, we would just use the same classes as the ones that define schemas, but relax the validation in the constructors to allow for empty value. We would then add methods to check that a given schema is complete. This would be the opportunity to return more precise error messages when the schema is incomplete (#4724).

Alternatives considered

Use ShEx or SHACL. Those formats are not designed for the Wikibase data model, but rather for RDF. This means that it would not be possible to take a schema represented in one of those formats, and render it as a Wikibase schema with holes. Therefore this does not seem fitting here.

Additional context

Requested by @trnstlntk @lozanaross in the context of the Wikimedia Commons integration project.

[johnsamuelwrites]

The proposal of “schema templates”, especially for the Wikimedia Commons Integration project is interesting. I understand the requirement where the users would propose “incomplete” schemas or “schemas with holes” suiting their particular or temporary requirements. We have a number of such use cases. For example, different language communities may describe a hospital in different ways, suiting their local needs. Hence, during WikiTechstorm 2019, we proposed to simplify the shape expressions (ShEx) in a way that helps users easily create schemas for their needs and validate their data with their own proposed schema (for example, the local definition of a hospital with the required properties). This is also very important for lexicographical data, where we required different schemas for lexemes of multiple languages.

Now coming to the Wikimedia commons integration project, based on my current usage/understanding of structured data on commons, I feel that the requirements for schemas for describing various data are not very complex and can be handled by simple shape expressions.

These use-cases were the inspirations behind the proposition of ShExStatements, a tool with the possibility to create simple shape expressions using CSV files. Take, for example, the QuickStatements tool that helps users easily create/modify statements on Wikidata using CSV files. Personally, I feel that this simplicity of using CSV files has helped lots of users to use OpenRefine and QuickStatements.

The proposal of “schema templates” could be useful for the community in the long run if it is based on standards. Tools like ShExStatements, EntitySchema Generator, ShExLite were proposed to simplify the ShEx creation process. These could be explored.

My suggestion is that you could use ShEx as the underlying technology of your proposal and continue proposing CSV for OpenRefine users for defining “schema templates”. A number of examples of shape expressions based on ShExStatements are given here. There are cases, where the use of certain properties was marked optional. Some of the shape expressions created by ShExStatements can be found here.

Finally, I would like to point out the cradle tool, which also proposes ShEx for creating new Wikidata items.

[wetneb]

Hi @johnsamuelwrites, thanks a lot for chiming in!

My suggestion is that you could use ShEx as the underlying technology of your proposal

How would you overcome the hurdles I mentioned on Wikidata?

Of course, if the schemas simply consist of a list of properties that should be on a given entity, that is not very hard to parse (but then ShEx is perhaps a bit overkill for such a use case).

Consider the case where the data needs to be input as qualifiers or references of statements. The corresponding ShEx would follow the RDF serialization of the data model to express this. For OpenRefine to be able to parse this and coerce it back to a schema with missing values, a rather complex heuristic needs to be implemented.

This scenario of using qualifiers is not fictional: it is used very commonly on Wikimedia Commons.

continue proposing CSV for OpenRefine users for defining “schema templates”.

I am not sure what you mean. Currently OpenRefine does not have any notion of schema template, and the one I am planning to implement does not use CSV, but JSON.

[johnsamuelwrites]

How would you overcome the hurdles I mentioned on Wikidata?

Translation to RDF and simplification are two possible approaches (also covered in the above link). For the second, we tried to handle the problem of creating Shape Expressions by proposing some of the tools/language subsets posted above. However, I haven't tried the former. I have used this tool for validation, but they are using the complete RDF of every entity to be validated. Based on my understanding, you want to handle it at the level of statements. For example, something like E273

continue proposing CSV for OpenRefine users for defining “schema templates”.

I wanted to say that OpenRefine must continue to support CSV files, even for future and new proposed use-cases (in this case, schema templates). JSON is interesting from a developer's perspective, but I am not convinced so far about its generalized use.

This scenario of using qualifiers is not fictional: it is used very commonly on Wikimedia Commons.

But lighter versions of ShEx can also handle these cases. For example, this ShEx of outbreak in CSV format uses qualifiers. You can even split ShEx to multiple files: E273, E274, and E275.

[wetneb]

I wanted to say that OpenRefine must continue to support CSV files, even for future and new proposed use-cases (in this case, schema templates). JSON is interesting from a developer's perspective, but I am not convinced so far about its generalized use.

I think this highlights a misunderstanding. The set of formats that OpenRefine supports to create a new project has nothing to do with the format chosen to represent schema templates (just like currently, the format used to represent Wikibase schemas in OpenRefine has no relationship to the data import formats supported by OpenRefine).

But lighter versions of ShEx can also handle these cases. For example, this ShEx of outbreak in CSV format uses qualifiers. You can even split ShEx to multiple files: E273, E274, and E275.

Yes - I am not saying that ShEx does not support qualifiers or references. What I am saying is that it seems difficult, in general, to parse a ShEx file and turn it into an OpenRefine schema where some values are left empty. There are plenty of constructs in ShEx that simply do not have an equivalent in our context: for instance, the OR operator, cardinality constraints, and so on.

As an example of why this is difficult, see this discussion about why ShEx's support in Cradle is experimental: https://twitter.com/MagnusManske/status/1227545532698169345

Now I do not think it is impossible to come up with some import/export from/to ShEx which works in simple cases, but this is bound to be brittle in general. So I would rather not use ShEx as primary data storage format for our schema templates.
If you (or anyone else) are interested in working on such an integration, I would be happy to advise and facilitate.

[ericprud]

Suppose Alice and Bob both want to upload CSVs about book editions. To cover more use cases, let's say that there's a constraint on ps:P629 that requires a P407 edition name and constrains a P1365 replaces to be another work.

replaces P1365

<edition_of_a_written_work> {
  # title
  wdt:P1476 LITERAL ;
  # edition or translation of
  ps:P629 {
    # edition name
    p:P407 @<language> ;
   # replaces
    p:P1365 @<written_work> *
  } ? ;
...

Bob.csv has a title and a publication date so he has rules that map column 1 to P1476. Alice.csv has detailed data bout encylopediae and maps column 5 to P1476, column 8 to ps:P629/p:P407, and column 9 to ps:P629/p:P1365.

In my example, it was worth exchanging the structure of the mapping target but not the mapping rules because the sources (CSVs) had different structures. Is that realistic?

I think the OpenRefine interpretation of ShEx depends on the what you expect OpenRefine to do. If the goal is just to drive user interface, it's easy to take the above schema (or ShExJ equivalent), and grep through the expressions for the predicates in order to drive a picklist with ~10 items rather than the 10K predicates in WD. Further, it would be easy to pick P629 and follow that up with prompts to supply P407 and P1365.

If the use case goes beyond UI prompts, it would be nice to work through a concrete example in order to see how the required info relates to the entity schema.