Make remote_data_updated mechanism usable for connectors

[jgoizueta]

Connectors were missing access to the modified_at field of synchronizations.

For non-connector runner it worked because the modified_at date was passed through the downloader.

[jgoizueta]

Parameter-naming confusion

I've kept the last_modified name used pervasively through the code as the remote source update time, but I've changed the newly introduced modified_at connectors parameter to previous_last_modified to indicate that that is the external source last-modified date corresponding to the previous update (sync).

So, how was the change detection originally implemented for file/url sources?

The Runner class, used by both imports and syncs, implements a remote_data_updated? method which calls Downloader#modified? which compares the last-modified date of the new data (from HTTP header) to the value from the previous sync, which is stored in synchronizations.modified_at and passed to the Downloader through http_options.last_modified

The synchronizations.modified_at column is set at the end of syncs through set_success_state_from from the importer's (a Synchronization::Adapter for syncs) last_modified method that takes the value from the Runner and this in turn from the Downloader.

DatasourceDownloader has similar a mechanism (modified?) using checksums instead of dates. Note that table synchronizations also has a checksum column to store this too.

Now, I see a problem with this mechanism, which is that the first import doesn't update the synchronization modified_at.
So the first sync will always be executed even if unnecessary. I'll open a ticket to fix this which should be a matter of adding synchronization.modified_at = importer.last_modified to ::DataImport::update_synchronization.

And how has this been extended to db connectors (and fixed)?

The remote_data_updated? has been implemented in ConnectorRunner, where it calls the connector provider's instance of this method. The provider should be able to implement this method because it receives through its constructor a previous_last_import parameter, that is taken from synchronizations.modified_at.

The ConnectorRunner also implements a last_modified modified method, also relayed to the provider, which allows the update of synchronizations.modified_at from the last modified data of the data just copied.

[jgoizueta]

⚠️ This can break existing connectors ⚠️

Since this introduces a new parameter, previous_last_modified unexpected by the connector providers, this must be deployed with the corresponding db-connectors changes. See the clubhouse story for details