In one DAG one task to next task took a lot of time

[wxm245]

task1 2020-07-24 22:43 end, but task2 2020-07-24 22:46 just started. so ,I want to know why there is a delay of 3 minutes

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jhtimmins]

This could be caused by a few things. One likely answer is that the heartbeat was increased to ~180 seconds. Are you using default values in airflow.cfg?

[wxm245]

This could be caused by a few things. One likely answer is that the heartbeat was increased to ~180 seconds. Are you using default values in airflow.cfg?

this is my airflow.cfg
[core]
dags_folder = /data/worker/airflow/dags
base_log_folder = /data/worker/airflow/logs
remote_logging = False
remote_log_conn_id =
remote_base_log_folder =
encrypt_s3_logs = False
logging_level = DEBUG
fab_logging_level = WARN
logging_config_class =
colored_console_log = True
colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] {%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} %%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s
colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location = /data/worker/airflow/logs/dag_processor_manager/dag_processor_manager.log
executor = CeleryExecutor
sql_engine_encoding = utf-8
sql_alchemy_pool_enabled = True
sql_alchemy_pool_size = 5
sql_alchemy_max_overflow = 10
sql_alchemy_pool_recycle = 1800
sql_alchemy_pool_pre_ping = True
sql_alchemy_schema =
parallelism = 32
dag_concurrency = 16
dags_are_paused_at_creation = True
max_active_runs_per_dag = 16
load_examples = False
load_default_connections = True
plugins_folder = /data/worker/airflow/plugins
donot_pickle = False
dagbag_import_timeout = 30
dag_file_processor_timeout = 50
task_runner = StandardTaskRunner
default_impersonation =
security =
secure_mode = False
unit_test_mode = False
enable_xcom_pickling = True
killed_task_cleanup_time = 60
dag_run_conf_overrides_params = False
worker_precheck = False
dag_discovery_safe_mode = True
default_task_retries = 0
store_serialized_dags = False
min_serialized_dag_update_interval = 30
store_dag_code = %(store_serialized_dags)s
max_num_rendered_ti_fields_per_task = 30
check_slas = True

[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
run_duration = -1
num_runs = -1
processor_poll_interval = 1
min_file_process_interval = 0
dag_dir_list_interval = 300
print_stats_interval = 30
scheduler_health_check_threshold = 30
child_process_log_directory = /data/worker/airflow/logs/scheduler
scheduler_zombie_task_threshold = 300
catchup_by_default = True
max_tis_per_query = 512
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
statsd_allow_list =
max_threads = 2
authenticate = False
use_job_schedule = True
allow_trigger_in_future = False

[ashb]

Likely improved massive in Airflow 2.0.0 alpha by the AIP-15 work

[kaxil]

Can you test it with Airflow >= 2.0.1 and report back please @wxm245

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[ashb]

2.0 (and actually 2.1) is out now and that fixes this problem. Please upgrade :)