scheduler does not appear to be running, cannot run a simple Dag

[cguermi]

Official Helm Chart version

1.5.0 (latest released)

Apache Airflow version

2.2.4

Kubernetes Version

1.22.2

Helm Chart configuration

airflow:
  executor: "CeleryExecutor"
  config:
    core: 
      default_timezone:  Europe/Paris
    api:
      auth_backend: airflow.api.auth.backend.basic_auth
    webserver:
      default_ui_timezone: Europe/Paris
      base_url: http://ep-prod-regions-webserver.ep-prod-regions.svc.cluster.local:8080
  registry:
    secretName: harbor-secret-password
  scheduler:
    resources:
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
      limits:
        memory: 1G
      requests:
        memory: 1G
    securityContext: 
      runAsUser: 0
      fsGroup: 0
      runAsGroup: 0
    nodeSelector:
      environment: ep-prod-regions
  statsd:
    nodeSelector:
      environment: ep-prod-regions
  webserver:
    resources:
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
      limits:
        memory: 1G
      requests:
        memory: 1G
    nodeSelector:
      environment: ep-prod-regions
    extraVolumes:
      - name: configuration
        persistentVolumeClaim:
            claimName: configuration-claim
    extraVolumeMounts:
      - name: configuration
        mountPath: /opt/airflow/crystal
        subPath: CrystalEnergyConf/client/data
    defaultUser:
      enabled: true
      role: Admin
      username: ####
      email: admin@example.com
      firstName: ####
      lastName: ####
      password: ####
  webserverSecretKeySecretName: regions-airflow-webserver-secret
  triggerer:
    nodeSelector:
      environment: ep-prod-regions
  flower:
    nodeSelector:
      environment: ep-prod-regions
  redis:
    nodeSelector:
      environment: ep-prod-regions
  workers:
    extraVolumes:
      - name: configuration
        persistentVolumeClaim:
            claimName: configuration-claim
    extraVolumeMounts:
      - name: configuration
        mountPath: /opt/airflow/crystal
        subPath: CrystalEnergyConf/client/data
    nodeSelector:
      environment: ep-prod-regions
    securityContext:
      runAsUser: 0
      fsGroup: 0
      runAsGroup: 0
  data:
    metadataSecretName: airflow-metadata-connection-secret
    resultBackendSecretName: airflow-result-backend-secret
  persistence: 
    fixPermissions: true
  enabled: true
  fernetKeySecretName: fernet-key-secret

  postgresql:
    enabled: true
    existingSecret: regions-database-secret
    volumePermissions:
      enabled: true
    persistence:
      enabled: true
      size: 20Gi
    service:
      port: 5432
    primary:
      nodeSelector:
        environment: ep-prod-regions
    resources:
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
      limits:
        memory: 1G
      requests:
        memory: 1G

Docker Image customisations

FROM apache/airflow:2.2.4

USER root

COPY --chown=airflow:root ./dags/ ${AIRFLOW_HOME}/dags/

USER root

What happened

Hi,

I just installed the helm chart, everything runs well (migrations jobs , pods initialization etc ), but when i connect to the UI and tried to launch a simple dag i received the message :

"The scheduler does not appear to be running. Last heartbeat was received 22 minutes ago.

The DAGs list may not update, and new tasks will not be scheduled."

also in the scheduler logs i got this error message:

“Executor reports task instance finished (failed) although the task says its queued. (Info: None) Was the task killed externally?”

maybe it's not relevant but i noticed that in the scheduler container i got "module not found" when i ran “airflow config list” command .

What you think should happen instead

i should be able to run my simple dag tasks with no error :

import time
from datetime import datetime

from airflow.decorators import dag, task

sleep_time = 5


@task()
def bidding():
    print('Start of bidding')
    time.sleep(sleep_time)
    print('End of bidding')


@task()
def market_clearing(bids):
    print('Start of market_clearing')
    time.sleep(sleep_time)
    print('End of market_clearing')


@task()
def post_processing(accepted_bids):
    print('Start of post_processing')
    time.sleep(sleep_time)
    print('End of post_processing')


@dag(
    schedule_interval=None,
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['simulation'],
    params={
    }
)
def market_simulation():


    bids = bidding()
    accepted_bids = market_clearing(bids)
    post_processing(accepted_bids)


market_simulation_dag = market_simulation()

How to reproduce

Install the chart version mentioned above and run the dag.

Anything else

i tried to run the chart with all the executors (LocalEcecutor, KubernetesExecutor, CeleryExecutor) with no success.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Clearly somethign wrong that you've modified and you need to investigate. You need to find out if your scheduler is running and what error scheduler prints if you need some help with the investigation.

An obvious wrong thing is that you have USER root specified twice in your Dockerfile - not sure if this is the reason (airflow shoudl be able to run with root user) but this is likely wrong. There is a possibilty you have not enough resources - memory likely - or other reasons but only your kubernetes logs might tell you what's wrong. You need to take a look at the failures and try to figure out what the problem is.

Without you getting the logs there is not much help we can do to help you to debug your deployment.