feat(dags): Add DAP Collector dag for PPA Dev #1976
Conversation
bbirdsong2
force-pushed
the
add-dap-collector-ppa-dev-dag
branch
from
38e85a9 to
9bcd194
Compare
bbirdsong2
changed the title
| default_args=default_args, | ||
| doc_md=DOCS, | ||
| schedule_interval="*/15 * * * *", | ||
| tags=tags, |
There was a problem hiding this comment.
You might want to explicitly set catchup=False so it doesn't trigger all runs since the start_date/last run if it's not necessary. False might be the default value but I can't remember
| tags=tags, | |
| tags=tags, | |
| catchup=False, |
There was a problem hiding this comment.
Yeah, I can add that.
dags/dap_collector_ppa_dev.py
Outdated
| "retries": 1, | ||
| "retry_delay": timedelta(minutes=15), |
There was a problem hiding this comment.
Retry delay is the same as schedule interval so you'll have two running concurrently if it fails. Would it make sense to wait for the next run instead of retrying? Or is the continuity important?
There was a problem hiding this comment.
That makes sense. I copied this from another job and did not evaluate the retry logic.
If a job fails, I do not actually think a retry or subsequent run would work, because the job depends on the {{ ts }} variable of the dag run, and only goes back every 15 minutes.
I think what I can do is remove the retry and run the script manually if a job fails.
5 participants
Description
This PR adds a new PPA Dev DAP Collector DAG to run the new docker-etl job here: mozilla/docker-etl#189. This DAG will be used to test out the ads use case in the DAP infrastructure.
Related Tickets & Documents