Missing DAG runs

[aki263]

We have more than 150 DAG running.
Randomly scheduler will skip a dag runs, I cant find any pattern, nothing in logs as well, it appears like scheduler forgot these dags to scheduler all togther.

Airflow Version: v2.7.3 Git Version: .release:f1243537838516b8bb8156130bc001595bfbeb01
EKS 1.25
RDS pgsql 14.7
Kubernetes executer with 2 scheduler pod running

Instance 1, Missed run on 28th 11.30 PM UTC

args = {
    "retries": 3,
    "retry_delay": timedelta(minutes=2),
    "start_date": datetime.now() - timedelta(hours=1),
}

dag = DAG(
    dag_id="matrix_snowflake_update_kyt_hourly",
    default_args=args,
    schedule_interval="30 * * * *",
    max_active_runs=1,
)

Instance 2: Missed run at 28th on 2PM UTC

args = {
    "catchup": False,
    "start_date": datetime.now() - timedelta(hours=1),
}

dag = DAG(
    dag_id="matrix_snowflake_update_liquidity_pool_stats_hourly",
    default_args=args,
    schedule_interval="0 * * * *",
    max_active_runs=1,
)

Instance 3: Daily DAG

args = {
    "catchup": False,
    "start_date": datetime.now() - timedelta(days=1),
    'executor_config': {
        'KubernetesExecutor': {
            'limit_memory': '3.0Gi',
        }
    },
}

dag = DAG(
    dag_id="platform_producer_coingecko_info",
    default_args=args,
    schedule_interval="0 6 * * *",
    max_active_runs=1,
)

airflow.cfg directly from pod.

[api]
auth_backends = airflow.api.auth.backend.basic_auth

[celery]
flower_url_prefix = /
worker_concurrency = 16

[celery_kubernetes_executor]
kubernetes_queue = kubernetes

[core]
colored_console_log = True
dags_folder = /opt/airflow/dags
default_pool_task_slot_count = 512
executor = KubernetesExecutor
load_examples = False
remote_logging = True

[elasticsearch]
json_format = True
log_id_template = {dag_id}_{task_id}_{execution_date}_{try_number}

[elasticsearch_configs]
max_retries = 3
retry_timeout = True
timeout = 30

[kerberos]
ccache = /var/kerberos-ccache/cache
keytab = /etc/airflow.keytab
principal = airflow@FOO.COM
reinit_frequency = 3600

[kubernetes]
airflow_configmap = airflow-airflow-config
airflow_local_settings_configmap = airflow-airflow-config
multi_namespace_mode = False
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = 123123123.dkr.ecr.us-east-1.amazonaws.com/airflow-prod
worker_container_tag = a6a136ee

[kubernetes_executor]
multi_namespace_mode = False
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = 123123123.dkr.ecr.us-east-1.amazonaws.com/airflow-prod
worker_container_tag = a6a136ee

[logging]
colored_console_log = True
delete_worker_pods = False
encrypt_s3_logs = False
logging_level = DEBUG
remote_base_log_folder = s3://s3-prod/airflow-prod-logs/
remote_log_conn_id = aws_conn_s3
remote_logging = True

[metrics]
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow

[scheduler]
logging_level = DEBUG
run_duration = 41460
standalone_dag_processor = False
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow

[smtp]
smtp_host = smtp.gmail.com
smtp_port = 587
smtp_ssl = true
smtp_starttls = true

[webserver]
enable_proxy_fix = True

[enchant3dmango]

I'm pretty sure that the dynamic start_date is the cause of the skipped DAG runs.

We recommend against using dynamic values as start_date, especially datetime.now() as it can be quite confusing. The task is triggered once the period closes, and in theory an @hourly DAG would never get to an hour after now as now() moves along.
Reference: https://airflow.apache.org/docs/apache-airflow/stable/faq.html#what-s-the-deal-with-start-date.

[aki263]

Thank you, we are going to modify dags and test it.

[aki263]

It looks like its working but we will keep on testing more, a question that arises with this is why this never broke in AWS managed airflow as we were running same dags over there, we started doing self hosted recently.

cc @potiuk

[brs-fcoo]

My company is experiencing something very similar, so I'm interested whether you found out why this behavior was different on AWS compared to self hosted?