[Infrastructure] Implement Reingestion Strategy for Met Museum

[mathemancer]

Current Situation

Currently, we generally only get information about a given image just after it
is uploaded to the Met Museum's website. This is insufficient for the reasons
outlined in #298. To summarize, some relevant metadata goes stale over time.

Suggested Improvement

We need to use a date-partitioned reingestion Apache Airflow DAG to keep the
data up to date over time. For examples, see:

• src/cc_catalog_airflow/dags/flickr_ingestion_workflow.py, and
• src/cc_catalog_airflow/dags/wikimedia_ingestion_workflow.py.

For the Met Museum, it's not clear yet how many days' worth of uploads we can
ingest per day. We need to:

1. Determine (on average) how many days' worth of uploaded metadata we can
   ingest from their API per day.
2. Come up with an ingestion schedule similar to the ones described in [Infrastructure] Implement Reingestion Strategy for Flickr #298 and
   [Infrastructure] Implement Reingestion strategy for Wikimedia Commons #395 .
3. Use the same general structure of the Ingestion workflows listed above to
   implement reingestion on that schedule.

Benefit

Implementing this will allow us to manage data that needs to be refreshed over
time for the Met Museum and catch any additions or deletions when they happen.
In the future, this would also allow us to ingest popularity data that changes
over time.

Additional context

The implementer is advised to use the dag_factory implemented in #394, as well
as the helper function to calculate the list lists of day shifts. It would be a
good idea to follow the general pattern used in the implementation of #394, just
with different parameters. Doing so will make this a quick and easy
implementation.

[kss682]

@mathemancer @annatuma
The objects updated over the last 6 months for met museum .

Date 	         Object count
2020-01-01 	 474441
2020-03-01 	 474441
2020-03-31 	 119361
2020-04-30 	 62519
2020-05-15 	 55163
2020-05-30 	 44238
2020-06-14 	 30850
2020-06-29 	 20343

In Met Museum, the date in metadataDate query param returns all objects updated after that date . So as we go back in date it results in cumulatively increasing set of records.

In that case, once we get all the data , we don't need to go back in date to refresh it as the daily crawl would automatically refresh it. (Is this correct ? )

[mathemancer]

That's correct. Your research shows this ticket is unneeded. Thanks for pointing that out!