Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add retries to job heartbeat #37541

Merged
merged 6 commits into from
Mar 8, 2024
Merged

Add retries to job heartbeat #37541

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cf154f3
Job heartbeat retries
awdavidson Feb 19, 2024
68bd08c
Tweak format
awdavidson Feb 19, 2024
87086e7
Merge branch 'main' into heartbeat-retries
awdavidson Feb 20, 2024
76fb98a
Retry on individual db operations
awdavidson Feb 26, 2024
3acd169
Merge branch 'main' into heartbeat-retries
awdavidson Feb 26, 2024
f301d03
Merge branch 'main' into heartbeat-retries
awdavidson Feb 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions airflow/jobs/job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.net import get_hostname
from airflow.utils.platform import getuser
from airflow.utils.retries import retry_db_transaction
from airflow.utils.session import NEW_SESSION, provide_session
from airflow.utils.sqlalchemy import UtcDateTime
from airflow.utils.state import JobState
Expand Down Expand Up @@ -302,6 +303,7 @@ def _kill(job_id: str, session: Session = NEW_SESSION) -> Job | JobPydantic:
@staticmethod
@internal_api_call
@provide_session
@retry_db_transaction
def _fetch_from_db(job: Job | JobPydantic, session: Session = NEW_SESSION) -> Job | JobPydantic | None:
if isinstance(job, Job):
# not Internal API
Expand Down Expand Up @@ -342,6 +344,7 @@ def _update_in_db(job: Job | JobPydantic, session: Session = NEW_SESSION):
@staticmethod
@internal_api_call
@provide_session
@retry_db_transaction
def _update_heartbeat(job: Job | JobPydantic, session: Session = NEW_SESSION) -> Job | JobPydantic:
orm_job: Job | None = session.scalar(select(Job).where(Job.id == job.id).limit(1))
if orm_job is None:
Expand Down