[Infrastructure] Implement reingestion strategy for Europeana

[mathemancer]

Current Situation

Once #368 is merged, we will be ingesting the previous day's uploads' metadata
each day for Europeana. But, we won't be reingesting the metadata for a given
image at any point (at least not automatically).

Suggested Improvement

We need to use a date-partitioned reingestion Apache Airflow DAG to keep the
data up to date over time. For examples, see:

• src/cc_catalog_airflow/dags/flickr_ingestion_workflow.py, and
• src/cc_catalog_airflow/dags/wikimedia_ingestion_workflow.py.

For Europeana, it's not clear yet how many days' worth of uploads we can ingest
per day. We need to:

1. Determine (on average) how many days' worth of uploaded metadata we can
   ingest from their API per day.
2. Come up with an ingestion schedule similar to the ones described in [Infrastructure] Implement Reingestion Strategy for Flickr #298 and
   [Infrastructure] Implement Reingestion strategy for Wikimedia Commons #395 .
3. Use the same general structure of the Ingestion workflows listed above to
   implement reingestion on that schedule.

Because Europeana is relatively low volume, we should try to reingest all of
their data at least once every month or two.

Benefit

Implementing this will allow us to manage data that needs to be refreshed over
time for Europeana, and catch any additions or deletions when they happen. In
the future, this would also allow us to ingest popularity data that changes over
time.

Additional context

• The implementer is advised to use the dag_factory implemented in Add Date Partitioned Flickr reingestion workflow #394, as
  well as the helper function to calculate the list lists of day shifts. It
  would be a good idea to follow the general pattern used in the implementation
  of Add Date Partitioned Flickr reingestion workflow #394, just with different parameters. Doing so will make this a quick and
  easy implementation.

[kss682]

Strategy

Distribution of item over the years

start_time                end_time          count
2013-01-01T00:00:00Z | 2014-01-01T00:00:00Z | 201172 |  
2014-01-01T00:00:00Z | 2015-01-01T00:00:00Z | 3575843 |  
2015-01-01T00:00:00Z | 2016-01-01T00:00:00Z | 1531808 |  
2016-01-01T00:00:00Z | 2016-12-31T00:00:00Z | 2515575 |  
2017-01-01T00:00:00Z | 2018-01-01T00:00:00Z | 944019 |  
2018-01-01T00:00:00Z | 2019-01-01T00:00:00Z | 5033245 |  
2019-01-01T00:00:00Z | 2020-01-01T00:00:00Z | 4562373 |  
                                                                     Total   |  | 18364035 |  

No items where uploaded before 2013.

Calculation of days of ingestion per day

Let x be the maximum number of items created on a day .
Maximum 10,000 requests per day and at a max of 100 items per request. (Based on the research in europeana ticket #279 )

Requests needed for getting one day's data : x / 100

Number of days that can ingested per day : (10000 * 100 ) / x

From the above distribution we can see that the year 2018-2019 had the maximum amount of items created , 5033245.
This means an average of 13,790 items were created on a single day.
Assuming this average across all years and adding certain buffer items lets keep 20,000 items per day as a base.

Number of days ingested per day - ( 10000 * 100 ) / 20000 - 50 days

I have doubts regarding this approach since there is possibility of bulk upload during the initial stages of uploading 2013, 2014, etc and there might be days when the created items total greater than the above estimated average.

@mathemancer

[kss682]

So, Europeana have their first uploaded date on Nov 21 2013, running script to get the daily item created count from the start date to end date (Dec 31 , 2019) gave some interesting results. They do not create or upload data on a daily basis . So the script ran to check the data for 2231 days and they have created data only in 561 days of it.
Other days item count is 0.

The strategy in the previous comment was based on the assumption that there would be some kind of uniform distribution of data with a maximum upload per day at 20,000. This assumption goes wrong as we can see that the max upload per day goes above hundred thousand .