Getting notified when a resource's DataStore data changes

[torfsen]

As of CKAN 2.8, the DataStore API functions don't mix well with the IResourceController interface. Things that I noticed:

• Calling datastore_create on an existing resource (i.e. when passing resource_id) does not trigger IResourceController.after_update. In contrast, calling datastore_create to create a new resource (i.e. passing resource) does trigger IResourceController.after_create.

• Calling datastore_upsert does not trigger IResourceController.after_update

• Calling datastore_delete does not trigger IResourceController.after_update

This means that it's currently impossible to reliably get notified when a resource's data in the DataStore changes.

Now I'm not sure whether IResourceController would be the right way to perform these notifications, since technically, the actual resource metadata doesn't change -- only the associated DataStore data does. So a DataStore-specific interface would probably be more appropriate.

There exists the IDatastore interface (which, although documented in the code, doesn't seem to be mentioned in the official docs), but it is intended to alter DataStore queries before they are executed.
In my opinion, we should

• add the required hooks to an existing or new DataStore-specific interface
• update the documentation of IResourceController to mention that

[wardi]

Overriding the datastore_create, _upsert and _delete actions is one option. I like to use triggers on the datastore database to detect changes because this is slow and unreliable when done on the python side.

I've thought about adding a standard datastore trigger that would collect the # rows created, updated and deleted with each action and pass that back to the actions. Would that be the sort of thing you need?

[torfsen]

@wardi My interest in this area stems from stadt-karlsruhe/ckanext-extractor#13: currently, ckanext-extractor doesn't handle DataStore-backed resources very well, and one reason is that some ways of creating those don't trigger the usual IResourceController hooks that ckanext-extractor uses.

I wouldn't need any fancy features for solving this issue (e.g. statistics about the number of rows affected), a simple hook for reacting to any change would probably be enough.

However, I don't have much experience with using the DataStore, so there maybe an approach that, in addition to solving my problem, is better suited to other use cases.

In the meantime I'll see whether overriding datastore_create, etc. is a usable workaround for my issue.

[francescoVenuto]

Overriding the datastore_create, _upsert and _delete actions is one option. I like to use triggers on the datastore database to detect changes because this is slow and unreliable when done on the python side.

I've thought about adding a standard datastore trigger that would collect the # rows created, updated and deleted with each action and pass that back to the actions. Would that be the sort of thing you need?

hello wardi, I would be interested in what you describe in your speech: I would like to insert a trigger to detect when a resource is added to the ckan, have you already faced the thing? could you kindly give me some advice? thank you