Systematically update CC catalog records

[kgodey]

Currently, when we pull data into the Catalog, it is stored, but never refreshed on future pulls from those sources.

We need to discuss how we want to go about maintaining/updating data in the Catalog.

For a description of the strategy for reingestion we're using (but not the implementation), see:
https://opensource.creativecommons.org/blog/entries/date-partitioned-data-reingestion/

[annatuma]

Loosely aiming for Q2 2020 to tackle this.

[kgodey]

@mathemancer has a plan for this, starting with #298

[mathemancer]

#298 has been implemented (see the PR #394 ).

[mathemancer]

There are three fundamentally different types of Provider APIs with regards to
this issue:

1. APIs that let us ingest metadata related to objects uploaded on a given date.
   • For these providers, a strategy like the one outlined in [Infrastructure] Implement Reingestion Strategy for Flickr #298 is necessary.
   • Of the scripts currently implmented, these are:
     • Flickr
     • Met Museum
     • PhyloPic
     • Wikimedia Commons
   • From that list, only Flickr and Wikimedia Commons are at a scale that
     implies difficulty. If we need to reingest the entire Met Museum
     collection, it's possible to do that at any time.
   • For The 'smaller' providers, the easiest solution would be to continue
     doing the daily ingestions, and combine them with a monthly 'complete'
     ingestion of their entire catalog.
2. APIs for which we ingest the entire collection every time we ingest from it.
   • For these providers, the problem is already solved.
3. APIs that offer a 'delta' endpoint that we use (currently none).
   • This would be tricky, since if we miss ingesting for some time, it may be
     that we'd have to reingest the entire collection to make sure we're in a
     consistent state.

[mathemancer]

I think we should focus on Wikimedia Commons next, both because it's the only one which isn't solved for which this might be challenging, and because we need more data from WMC for Regoknition analysis.

[mathemancer]

See issue #395 for the Wikimedia Commons plan.

[mathemancer]

For the following API providers, reingestion is not necessary, because we ingest all their data every time we run:

• Cleveland Museum
• RawPixel
• Smithsonian Institution

For the following API providers, reingestion is not necessary, because we pull from a delta endpoint (i.e., we're separating on date modified, rather than uploaded):

• PhyloPic

For the following API Providers, we need to create Apache Airflow DAGs implementing the Date-partitioned reingestion strategy:

• Flickr -- see [Infrastructure] Implement Reingestion Strategy for Flickr #298 and Add Date Partitioned Flickr reingestion workflow #394
• Wikimedia Commons -- see [Infrastructure] Implement Reingestion strategy for Wikimedia Commons #395 and Add Date Partitioned Wikimedia reingestion workflow #402
• Europeana -- see [Infrastructure] Implement reingestion strategy for Europeana #412
• Met Museum -- see [Infrastructure] Implement Reingestion Strategy for Met Museum #413

The only current API provider not mentioned above is Thingiverse which is BLOCKED by #391. Whenever that is implemented, the implementation may or may not need to use a reingestion strategy.

[mathemancer]

Moving forward, we should use the reingestion strategy out of the gate for any new Provider API Script, whenever it uses the date-partitioning strategy to ingest in the first place.