Stocksnap 403 forbidden error during ingestion #4101
Labels
💻 aspect: code
Concerns the software code in the repository
🛠 goal: fix
Bug fix
🟧 priority: high
Stalls work on the project or its dependents
🧱 stack: catalog
Related to the catalog and Airflow DAGs
⛔ status: blocked
Blocked & therefore, not ready for work
🔧 tech: airflow
Involves Apache Airflow
Airflow log link
https://airflow.openverse.org/log?execution_date=2024-03-01T00%3A00%3A00%2B00%3A00&task_id=ingest_data.pull_image_data&dag_id=stocksnap_workflow&map_index=-1
Description
The Stocksnap DAG encountered an error during ingestion:
On top of this, Stocksnap uses a page counter instead of normal query params, so it's difficult to determine which page it failed on:
openverse/catalog/dags/providers/provider_api_scripts/stocksnap.py
Line 45 in 852e6b7
In addition to resolving this issue, we should try and alter the DAGs that don't normally use query parameters so they still have something to report when they fail.
DAG status
Unchanged for now since this is a monthly DAG
The text was updated successfully, but these errors were encountered: