Creating providers

[mielvds]

Split of from #694:

     creating and importing concept schemes through the UI

Currently it's not possible to create a conceptschem through the UI since there is some programmatic configuration involed. Once created, it is possible to edit the metadata about the conceptscheme though. Importing of concepts is possible on a one by one basis, but not for a full conceptscheme. Importing entire conceptschemes from SKOS files is possible through the CLI.

Creating conceptschemes through the UI has been discussed once or twice, so there's certainly some interest in that.

Started draft PR in #725 . Help and guidance needed.

[koenedaele]

Before we dive into the code and the PR, I'd like to have a look at it from a more architectural point of view.

I think the endgoal we want to accomplish is: adding new CS without editing the code or config files. The current setup is extremely flexible, but it always requires updating the config and redeploying.

What we basically need is to have the create_registry funtion in https://github.com/OnroerendErfgoed/atramhasis/blob/develop/atramhasis/skos/__init__.py talk to the database to initialise all the SQLAlchemyProviders. This function is run on a per-request basis to initialise all providers. It doesn't only init internal providers stored in the SQL database, but can also handle any external provider such as the getty vocabs in this example, or a connection with someone else's Atramhasis through skosprovider_atramhasis. These providers are marked as "external" so they don't show up in the UI, but they are available in the edit environment for importing and matching concepts. So, we need to replace lines 29-143 and 160-167.

The pattern is basically: instantiate a SQLALchemyProvider and register it with the SkosRegistry. To instantiate a SQLAlchemyProvider the following things are absolutely necessary:

• A provider ID, such as ERFGOEDTYPES or SPECIES, this is the ID by which the provider is known to the registry.
• A conceptscheme_id, this is the ID of the CS in the database. This is only mandatory for SQLAlchemyProviders.
• A SQLALchemy db connection. This is only mandatory for SQLAlchemyProviders

Not mandatory in theory because there's a default option, but pretty unavoidable in actual use:

• A uri_generator that generates uri's for new concepts and collections if there isn't one present. This is a class that follows the UriGenerator interface. While it's very flexible, in actual use we've only ever used the UriPatternGenerator that adds an id like 5 to a URI pattern like https://id.erfgoed.net/thesauri/erfgoedtypes/%s to end up with https://id.erfgoed.net/thesauri/erfgoedtypes/5

I think these are the ones that are the most important. Apart from those there's also:

• atramhasis.force_display_language: An IANA language code that gives a certain language precedence for a certain provider, regardless of what the user's asking for. See https://atramhasis.readthedocs.io/en/latest/customisation.html#force-a-display-language-for-a-vocabulary for the full explanation
• default_language: An IANA language code that tell the provider what language to serve is none is requested, different from force_display_language since this can be overridden with the user's preferences
• subject: This is kind of like a tag that's added to the provider. There are two have meaning for Atramhasis: external (see above; I think these should always be handled by IT staff since they required adding a new provider class and might require passing things like a requests session) and hidden. That last one does get used quite a bit by us for thesauri that are useful for internal purposes but that we don't want to advertise. See https://atramhasis.readthedocs.io/en/latest/customisation.html#hiding-a-vocabulary for more info.
• dataset: This is an object that provides extra metadata that can be turned into a void dataset file such as https://thesaurus.onroerenderfgoed.be/void.ttl See https://github.com/OnroerendErfgoed/atramhasis/blob/develop/atramhasis/skos/__init__.py#L39 for a config example. There is some overlap between this data and what could be seen as CS data. In a lot of ways one conceptscheme could be one dataset, but your usecase for having several versions of a CS contradicts this a bit.

Of all of the above, the dataset stuff gives me the most concern. I do see that I never wrote docs on the void features and it never appeared in the release notes either, so it could be considered unofficial and unsupported. I was planning on replacing it with a dcat ap Vlaanderen file that could then be fed to open data portals, using a similar approach. But that required even more metadata and I never finished it. Still seems like that would be a useful feature, but that's a different story. All things considered, I'm considering dropping this feature to make our lives easier.

It basically boils down to ID's, URI generating patterns and a few config options that influence display options. The provider id could by default be equal to the conceptscheme_id, but be changeable to simplify things even more.

So, there's the data that makes up the actual CS and a few things that configure how Atramhasis deals with a certain CS. I don't think there's much point in being able to create a CS if you can't configure the corresponding provider and there also isn't much point in configuring a provider without a CS. This raises the question if these should be two endpoints or one. Currently I'm in favor of keeping them separated so it's possible to have different security roles there. A more technical admin who can configure the provider and a more business admin who can edit the CS data such a label and description, but can't change the URI config or hide providers. Probably makes it easier to maintain a close link between the CS endpoint and SKOS.

And that's enough for a Friday evening ;-)

[koenedaele]

BTW, it's possible we rearrange things when it comes to issues and branches and stuff. We have a development model with feature branches and epics and such, but that uses some third-party software not everyone has access to. We'll look into this next week.

[cedrikv]

@koenedaele Shouldn't we add an option to provide a conceptscheme label? If you create a provider now, the label is just 'null':

[koenedaele]

Good catch. How would this work?

• Option 1: When POSTing a Provider, make it possible to add a conceptscheme label
• Option 2: POST provider, GET conceptscheme, add label, PUT conceptscheme
• Option 3: Update that dropdown to show the URI of a conceptscheme if there's no label present

Options I'm missing?

[cedrikv]

regarding option 3 : you can't edit the label of a conceptscheme in the current UI

[koenedaele]

I think you can edit the label of a conceptscheme by pressing that "Edit" button next to the null label: