Scheduler throwing error when schedule is set to @hourly and catchup_by_default is set to False #37869

sudarshan2906 · 2024-03-02T00:29:51Z

sudarshan2906
Mar 2, 2024

Apache Airflow version

2.8.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I am using a very simple dag which has schedule set has @hourly. Scheduler is not scheduling it and giving error in logs.

import datetime
from airflow import DAG
from airflow.operators.empty import EmptyOperator

my_dag = DAG(
    dag_id="my_dag_name",
    start_date=datetime.datetime(2021, 1, 1),
    schedule="@hourly",
)
EmptyOperator(task_id="task", dag=my_dag)

Error in log:

ERROR [airflow.models.dag.DAG] Failed to fetch run info after data interval DataInterval(start=DateTime(2024, 3, 2, 0, 0, 0, tzinfo=Timezone('UTC')), end=DateTime(2024, 3, 2, 1, 0, 0, tzinfo=Timezone('UTC'))) for DAG 'my_dag_name'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/airflow/models/dag.py", line 1032, in next_dagrun_info
    info = self.timetable.next_dagrun_info(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/timetables/interval.py", line 90, in next_dagrun_info
    earliest = self._skip_to_latest(earliest)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/airflow/timetables/interval.py", line 157, in _skip_to_latest
    raise AssertionError("next schedule shouldn't be earlier")

If I set catchup_by_default to True or change the schedule to more than an hour it works fine.

What you think should happen instead?

No response

How to reproduce

NA

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

Using a custom docker image. Airflow configuration:

Executor: Celery
Database: Postgresql 14
Docker base image: python:3.11-bullseye

Anything else?

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Answered by Taragolis

Mar 3, 2024

Well that is more like troubleshooting rather than actual problem in airflow codebase or dependencies.

I've added just this variables when start official Apache Airflow image, and it works without any issues

    AIRFLOW__LOGGING__COLORED_CONSOLE_LOG: 'false
    AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT: 'false'
    TZ: 'CET'
    AIRFLOW__CORE__DEFAULT_TIMEZONE: 'CET'
    AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: 'CET'

What is wrong with your cause, it is hard to guess but my assumptions

No TZ environment variable
Or maybe because you overwrite symlink link by a regular file

You could try to replace

ENV TZ="CET"
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \ 
    && echo $TZ > /etc/tim…

View full answer

Taragolis · 2024-03-02T12:20:34Z

Taragolis
Mar 2, 2024
Collaborator

Unable to reproduce

  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2024-03-02T12:16:34.195+0000] {task_context_logger.py:63} INFO - Task context logging is enabled
[2024-03-02T12:16:34.196+0000] {executor_loader.py:115} INFO - Loaded executor: CeleryExecutor
[2024-03-02T12:16:34.367+0000] {scheduler_job_runner.py:808} INFO - Starting the scheduler
[2024-03-02T12:16:34.368+0000] {scheduler_job_runner.py:815} INFO - Processing each file at most -1 times
[2024-03-02T12:16:34.372+0000] {manager.py:169} INFO - Launched DagFileProcessorManager with pid: 32
[2024-03-02T12:16:34.373+0000] {scheduler_job_runner.py:1608} INFO - Adopting or resetting orphaned tasks for active dag runs
[2024-03-02T12:16:34.375+0000] {settings.py:60} INFO - Configured default timezone UTC
127.0.0.1 - - [02/Mar/2024 12:16:56] "GET /health HTTP/1.1" 200 -
[2024-03-02T12:17:22.183+0000] {dag.py:3834} INFO - Setting next_dagrun for my_dag_name to 2024-03-02 12:00:00+00:00, run_after=2024-03-02 13:00:00+00:00
[2024-03-02T12:17:23.260+0000] {dagrun.py:795} INFO - Marking run <DagRun my_dag_name @ 2024-03-02 11:00:00+00:00: scheduled__2024-03-02T11:00:00+00:00, state:running, queued_at: 2024-03-02 12:17:22.169354+00:00. externally triggered: False> successful
[2024-03-02T12:17:23.261+0000] {dagrun.py:846} INFO - DagRun Finished: dag_id=my_dag_name, execution_date=2024-03-02 11:00:00+00:00, run_id=scheduled__2024-03-02T11:00:00+00:00, run_start_date=2024-03-02 12:17:22.188603+00:00, run_end_date=2024-03-02 12:17:23.261163+00:00, run_duration=1.07256, state=success, external_trigger=False, run_type=scheduled, data_interval_start=2024-03-02 11:00:00+00:00, data_interval_end=2024-03-02 12:00:00+00:00, dag_hash=44b4fe2798b492ea8d535bd13c7385cf
[2024-03-02T12:17:23.266+0000] {dag.py:3834} INFO - Setting next_dagrun for my_dag_name to 2024-03-02 12:00:00+00:00, run_after=2024-03-02 13:00:00+00:00
127.0.0.1 - - [02/Mar/2024 12:17:26] "GET /health HTTP/1.1" 200 -
127.0.0.1 - - [02/Mar/2024 12:17:56] "GET /health HTTP/1.1" 200 -

Maybe you miss any other details?

0 replies

sudarshan2906 · 2024-03-03T13:41:57Z

sudarshan2906
Mar 3, 2024
Author

Hi @Taragolis ,

I debugged more on this. catchup_by_default doesn't seems to be the only reason. The default timezone + catchup_by_default seems to be an issue here. I am using CET timezone. And after that I was able to reproduce this.

Here is a Dockerfile and docker-compose.yml to reproduce this issue.

Dockerfile:

FROM python:3.11-bullseye

ENV AIRFLOW_HOME=/home/airflow \
    AIRFLOW_VERSION=2.8.1

RUN pip install --no-cache-dir \
    --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.11.txt" \
    apache-airflow[postgres,celery]==${AIRFLOW_VERSION}

# Time zone configuration
RUN cp /usr/share/zoneinfo/CET /etc/localtime && \
    echo "CET" > /etc/timezone

EXPOSE 8080

docker-compose.yml

version: '3'
services:
    webserver:
        platform: linux/amd64
        depends_on:
            - postgres
        build:
            context: .
        ports:
            - "8080:8080"
        environment:
            - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
            - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
            - AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT=False
            - AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE=CET
            - AIRFLOW__CORE__DEFAULT_TIMEZONE=CET
        volumes:
            - ./dags:/home/airflow/dags
        entrypoint: airflow webserver

    scheduler:
        platform: linux/amd64
        depends_on:
            - postgres
        build:
            context: .
        environment:
            - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
            - AIRFLOW__CORE__EXECUTOR=CeleryExecutor
            - AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT=False
            - AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE=CET
            - AIRFLOW__CORE__DEFAULT_TIMEZONE=CET
        volumes:
            - ./dags:/home/airflow/dags
        entrypoint: airflow scheduler

    postgres:
        image: postgres:14
        ports:
            - "5432:5432"
        environment:
            - POSTGRES_USER=airflow
            - POSTGRES_PASSWORD=airflow
            - POSTGRES_DB=airflow
        hostname: postgres

2 replies

Taragolis Mar 3, 2024
Collaborator

Well that is more like troubleshooting rather than actual problem in airflow codebase or dependencies.

I've added just this variables when start official Apache Airflow image, and it works without any issues

    AIRFLOW__LOGGING__COLORED_CONSOLE_LOG: 'false
    AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT: 'false'
    TZ: 'CET'
    AIRFLOW__CORE__DEFAULT_TIMEZONE: 'CET'
    AIRFLOW__WEBSERVER__DEFAULT_UI_TIMEZONE: 'CET'

What is wrong with your cause, it is hard to guess but my assumptions

No TZ environment variable
Or maybe because you overwrite symlink link by a regular file

You could try to replace

ENV TZ="CET"
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime \ 
    && echo $TZ > /etc/timezone

Answer selected by sudarshan2906

sudarshan2906 Mar 3, 2024
Author

Thanks. Adding ENV TZ="CET" Worked.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scheduler throwing error when schedule is set to @hourly and catchup_by_default is set to False #37869

{{title}}

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Scheduler throwing error when schedule is set to @hourly and catchup_by_default is set to False #37869

sudarshan2906 Mar 2, 2024

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Replies: 2 comments · 2 replies

Taragolis Mar 2, 2024 Collaborator

sudarshan2906 Mar 3, 2024 Author

Taragolis Mar 3, 2024 Collaborator

sudarshan2906 Mar 3, 2024 Author

sudarshan2906
Mar 2, 2024

Replies: 2 comments 2 replies

Taragolis
Mar 2, 2024
Collaborator

sudarshan2906
Mar 3, 2024
Author

Taragolis Mar 3, 2024
Collaborator

sudarshan2906 Mar 3, 2024
Author