Extend Europeana timeout #3450
Labels
💻 aspect: code
Concerns the software code in the repository
🛠 goal: fix
Bug fix
🟩 priority: low
Low priority and doesn't need to be rushed
🧱 stack: catalog
Related to the catalog and Airflow DAGs
Description
The Europeana DAG runs daily. Most days finish very quickly, but periodically we encounter a day that has a very large number of records and takes considerably longer. The current 3-day timeout is meant to accommodate this by allowing large ingestion days a significant amount of time to ingest, with the expectation that subsequent days will be very small so we can always catch back up to the present day.
Three days does not appear to be sufficient. We've had a few times where a long day has timed out, and we've needed to manually restart the timed out day from the point of failure. The most recent such example would have taken 6 days to complete. We should consider extending the timeout to 6 or 7 days.
Reproduction
Observe production logs from Europeana.
The text was updated successfully, but these errors were encountered: