Dag makes all tasks as failed and retries when only one task in dag updated

[AwaishK]

I am using a docker container to run airflow webserver and scheduler, Everything else is working except the fact that any change in dag stops all currently running tasks and marks them failed.

E.g: There are two tasks unders dags/tasks.py named task1 and task2. Both the tasks belong to same dag name new_tasks and I change name task1 to task11 while task2 is running, then airflow renames the tasks1 to task11 and executes this task but task2 which was running when change was made, airflow will stop it and mark it as failed.

Failed tasks show below message in logs.

[2021-03-30 16:30:13,389] {local_task_job.py:170} WARNING - State of this instance has been externally set to failed. Terminating instance.

[2021-03-30 16:30:13,421] {process_utils.py:95} INFO - Sending Signals.SIGTERM to GPID 19049

[2021-03-30 16:30:13,422] {taskinstance.py:1214} ERROR - Received SIGTERM. Terminating subprocesses.

my docker-compose.yml file

version: '3.7'
services:
    postgres:
        image: postgres:12.3
        environment:
            - POSTGRES_USER=airflow
            - POSTGRES_PASSWORD=airflow
            - POSTGRES_DB=airflow
        logging:
            options:
                max-size: 10m
                max-file: "3"

    webserver:
        image: oyo-airflow:1.0
        build: .
        restart: always
        depends_on:
            - postgres
        logging:
            options:
                max-size: 10m
                max-file: "3"
        environment:
            AIRFLOW__CORE__EXECUTOR: LocalExecutor
            AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql://airflow:airflow@postgres:5432/airflow
            AIRFLOW__CORE__FERNET_KEY: ''
            AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
            AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
        volumes:
            - ./dags:/usr/local/airflow/dags
        ports:
            - "8080:8080"
        command: webserver
        healthcheck:
            test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
            interval: 30s
            timeout: 30s
            retries: 3

And my Dockerfile

FROM python:3.7-slim-buster

# Never prompt the user for choices on installation/configuration of packages
ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux

# Airflow
ARG AIRFLOW_USER_HOME=/usr/local/airflow
ARG CONFIG_FILE=config.ini
ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}

# Disable noisy "Handling signal" log messages:
# ENV GUNICORN_CMD_ARGS --log-level WARNING
COPY requirements.txt ${AIRFLOW_USER_HOME}/requirements.txt
COPY unittests.cfg ${AIRFLOW_USER_HOME}/unittests.cfg 

# Make ssh dir
# RUN mkdir /root/.ssh/

# Create known_hosts
# RUN touch /root/.ssh/known_hosts

# yqq requires to locate git package and other packages installed under build-essentials

# build-essential needed for installing python packages e.g: psutils have dependency on system 
# packages

# netcat is needed to make a network to connect to database

# openssh-server needed to install git repo using ssh

RUN    apt-get update -yqq \
    && apt-get upgrade -yqq \
    && apt-get install -yqq --no-install-recommends \
        git \
        build-essential \
        netcat \
        openssh-server \
        apt-utils \
    && useradd -ms /bin/bash -d ${AIRFLOW_USER_HOME} airflow 
 
# Add bitbuckets key and install danamica_utils
RUN python -m pip install --upgrade pip==20.2.4
RUN pip install -r ${AIRFLOW_USER_HOME}/requirements.txt 

# using secret for private key as it is encrypted form
# RUN mkdir -p ${AIRFLOW_USER_HOME}/.ssh && ln -s /run/secrets/user_ssh_key ${AIRFLOW_USER_HOME}/.ssh/id_rsa

RUN mkdir -p ${AIRFLOW_USER_HOME}/.ssh
COPY id_rsa ${AIRFLOW_USER_HOME}/.ssh/id_rsa
COPY known_hosts ${AIRFLOW_USER_HOME}/.ssh/known_hosts

# RUN ln -s /run/secrets/user_known_hosts ${AIRFLOW_USER_HOME}/.ssh/known_hosts
RUN chown -R airflow:airflow ${AIRFLOW_USER_HOME}/.ssh

COPY entrypoint.sh /entrypoint.sh

# This file is being used to transfer the local airflow configuration to airflow 
COPY airflow.cfg ${AIRFLOW_USER_HOME}/airflow.cfg
COPY webserver_config.py ${AIRFLOW_USER_HOME}/webserver_config.py

# Config file needed in danamica_utils
COPY ${CONFIG_FILE} ${AIRFLOW_USER_HOME}/config.ini

# changing the owner to airflow for /use/local/airflow and /home to it can map local folders into the container
RUN mkdir -p ${AIRFLOW_USER_HOME}/logs

RUN chown -R airflow: ${AIRFLOW_USER_HOME}
RUN chown -R airflow: ${AIRFLOW_USER_HOME}/logs
# RUN chmod 777 -R  ${AIRFLOW_USER_HOME}/logs
RUN chown -R airflow: /home

# setting the environment variable DANAMICA_CONFIG and pythonpath
ENV DANAMICA_CONFIG=${AIRFLOW_USER_HOME}/config.ini
ENV PYTHONPATH=${AIRFLOW_USER_HOME}

# tell the docker below ports are useful
EXPOSE 8080 5555 8793 6379 5672

# sets the user name to use when running the image and for any RUN, CMD and ENTRYPOINT 
# instructions that follow it in the Dockerfile.
USER airflow
# Sets the working directory for any RUN, CMD, ENTRYPOINT, COPY, and ADD instructions that follow it
WORKDIR ${AIRFLOW_USER_HOME}
# Allows to configure a container that will run as an executable.

ENV AIRFLOW__LOGGING__LOGGING_LEVEL=INFO
ENV AIRFLOW__LOGGING__BASE_LOG_FOLDER=/usr/local/airflow/logs
ENV AIRFLOW__SCHEDULER__CHILD_PROCESS_LOG_DIRECTORY=/usr/local/airflow/logs/scheduler
ENV AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE='{{ ti.dag_id }}/{{ ti.task_id }}/{{ execution_date.strftime("%%Y-%%m-%%dT%%H:%%M") }}/{{ try_number }}.log'

ENV AIRFLOW__LOGGING__REMOTE_LOGGING=False
ENV AIRFLOW__LOGGING__TASK_LOG_READER=task
ENV AIRFLOW__METRICS__STATSD_ON=False
ENV AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=False

# provide defaults to an executing container, it should be only one in dockerfile
ENTRYPOINT ["sh", "/entrypoint.sh"]

# CMD ["webserver"]

entrypoint.sh file

#!/usr/bin/env bash

# Global defaults and back-compat
: "${AIRFLOW_HOME:="/usr/local/airflow"}"

export AIRFLOW_HOME 

# Initialize the metadata database 
airflow db init
airflow db upgrade


sleep 10

# import parameters -p is password, -r is role and it is case sensitive and -u is username
airflow users create -e admin@admin.com -f admin -l admin -p airflow -r Admin  -u airflow

# execute the airflow
airflow scheduler & exec airflow webserver

And here is my requirements.txt file

apache-airflow[postgres,ssh,google_auth, celery, google]==2.0.0
pre-commit
dataclasses
eventlet
apache-airflow-upgrade-check
sqlalchemy==1.3.23
redis
flower
gevent