Unable to init/reset db with Airflow 2.1 with docker-compose

[JavierLopezT]

I have been trying to migrate to Airflow 2 for a long time with several versions and I haven't been able to do so. Now, with 2.1 I can't init nor reset the db. This is the log of the init command:

airflow@76c689dceb01:/opt/airflow$ airflow db init
DB: postgresql+psycopg2://airflow:***@postgres:5432/airflow
[2021-06-08 13:23:24,809] {db.py:695} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
usage: airflow [-h] [--dag_route DAG_ROUTE] [--dag_name DAG_NAME] [--ts TS] [--installation_code INSTALLATION_CODE] [--extra_arguments [EXTRA_ARGUMENTS [EXTRA_ARGUMENTS ...]]]
airflow: error: unrecognized arguments: db init

And this is for the reset db:

airflow@76c689dceb01:/opt/airflow$ airflow db reset
DB: postgresql+psycopg2://airflow:***@postgres:5432/airflow
This will drop existing tables if they exist. Proceed? (y/n)y
[2021-06-08 13:23:54,835] {db.py:711} INFO - Dropping tables that exist
[2021-06-08 13:23:55,010] {migration.py:164} INFO - Context impl PostgresqlImpl.
[2021-06-08 13:23:55,010] {migration.py:167} INFO - Will assume transactional DDL.
[2021-06-08 13:23:55,145] {db.py:695} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
INFO  [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
INFO  [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
usage: airflow [-h] [--dag_route DAG_ROUTE] [--dag_name DAG_NAME] [--ts TS] [--installation_code INSTALLATION_CODE] [--extra_arguments [EXTRA_ARGUMENTS [EXTRA_ARGUMENTS ...]]]
airflow: error: unrecognized arguments: db reset

I have a function with those arguments in the utilities subfolder within dags folder. The code of the function is the following:

def get_arguments_python_script():
    import argparse

    parser = argparse.ArgumentParser()
    arguments = ['dag_route', 'dag_name', 'ts', 'installation_code']
    for argument in arguments:
        parser.add_argument(f'--{argument}')
    parser.add_argument('--extra_arguments', nargs='*')

    args = parser.parse_args()

    dag_route = args.dag_route
    dag_name = args.dag_name
    ts = args.ts
    installation_code = args.installation_code
    extra_arguments = args.extra_arguments

    if installation_code != 'None' and extra_arguments:
        return dag_route, dag_name, ts, extra_arguments, installation_code
    elif installation_code != 'None':
        return dag_route, dag_name, ts, installation_code
    elif extra_arguments:
        return dag_route, dag_name, ts, extra_arguments
    else:
        return dag_route, dag_name, ts

The entire folder utilities is in the file .airflowignore. I call this function from python scripts (called from bash operators) that are in dags subfolders. So my directory looks like:

dags
    import_google
        dag_file.py
            BashOperator -> python3 google.py dag_route dag_name
        google.py
            from utilities.files import get_arguments_python_script()
    utilities
         files.py
    .airflowignore
        utilities
plugins
    snowflake_hook.py
    snowflake_operator.py
logs
Dockerfile
docker-compose.yaml
requirements.txt
variables_secret.json

My docker files seem very much the same the ones in the official example of the docs, and work perfectly for 1.10.15. My Dockerfile is:

FROM apache/airflow:2.1.0-python3.8
USER root
# ------------------------------------------------------------
# install manually all the missing libraries to use chrome 
RUN sudo apt-get update && sudo apt-get install -y build-essential gconf-service libasound2 libatk1.0-0 libcairo2 libcups2 libfontconfig1 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libxss1 fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget libappindicator3-1 libgbm1
USER airflow

COPY plugins/aws_secrets_manager_backend.py /home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/secrets/secrets_manager.py

COPY aws_config /home/airflow/.aws/config
COPY aws_credentials /home/airflow/.aws/credentials
COPY requirements.txt requirements.txt
COPY service_account.json service_account.json

RUN pip install -r requirements.txt --user

The docker-compose is:

version: "3.7"
x-airflow-environment: &airflow-environment
  AIRFLOW__CORE__EXECUTOR: LocalExecutor
  AIRFLOW__CORE__LOAD_EXAMPLES: "False"
  AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS: "False"
  AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
  AIRFLOW__WEBSERVER__DAG_DEFAULT_VIEW: graph
  AIRFLOW__SECRETS__BACKEND: airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
  AIRFLOW__OPERATORS__DEFAULT_RAM: 2048
  AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: 300
services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
  init:
    build: .
    environment:
      <<: *airflow-environment
    depends_on:
      - postgres
    entrypoint: /bin/bash
    command: >
      -c "airflow users list || (airflow db init && airflow users create --role Admin --username airflow --password airflow -e airflow@airflow.com -f airflow -l airflow)"
    restart: on-failure
  webserver:
    build: .
    ports:
      - 8080:8080
    environment:
      <<: *airflow-environment
    depends_on:
      - init
    volumes:
      - ./dags/${WORKING_DAG}:/opt/airflow/dags/${WORKING_DAG}
      - ./plugins:/opt/airflow/plugins
      - ./variables_secret.json:/opt/airflow/variables_secret.json
      - ./dags/utilities:/opt/airflow/dags/utilities
      - ./logs:/opt/airflow/logs:delegated
    entrypoint: /bin/bash
    command: -c "airflow variables import /opt/airflow/variables_secret.json && airflow webserver"
    restart: always
  scheduler:
    build: .
    environment:
      <<: *airflow-environment
    depends_on:
      - webserver
    volumes:
      - ./dags/${WORKING_DAG}:/opt/airflow/dags/${WORKING_DAG}
      - ./plugins:/opt/airflow/plugins
      - ./variables_secret.json:/opt/airflow/variables_secret.json
      - ./dags/utilities:/opt/airflow/dags/utilities
      - ./logs:/opt/airflow/logs:delegated
    entrypoint: /bin/bash
    command: -c "airflow variables import /opt/airflow/variables_secret.json && airflow scheduler"
    restart: always

I have tried in my Mac and in a fresh Ubuntu EC2 and in both cases, I get the same error. Is there any way I could solve this or is it some kind of bug?

Thanks

[potiuk]

You probably call directly the get_arguments_python_script inside some of your DAGs that get imported. (My guess google.py or dag_file.py).

If you intend to run the script manually you should run this method in the main if:

if __name__ == "__main__":
   ...

[JavierLopezT]

Thank you very much for your answer @potiuk.

This has solved this problem (and created another). However...

In any DAG is called the function directly, only in Python scripts, so I tried first with .airflowignore file. I renamed google.py to py_google.py and after moving the airflowignore file to the root of airflow directory, I typed in it:

dags/**/py_*.py
dags/utilities

Which didn't have any effect. I am wondering if it is something wrong in the code/file or just it is a wrong approach.

Then, in all my dags I have this:
dag_globals = Variable.get('variables_secret', deserialize_json=True)
Because of this, I have had to include the DAG code also in a main statement. But I am wondering if there is any better way to solve the problem, because ALL (around 100) of my DAGs have this line.

Furthermore, this line is also in my utilities files and I have had to include a main in them also, leading to an error importing the function from other files.

I am surprised that I haven't seen any mention of this in any upgrade document. The scheduler behavior has changed a lot and I have spent a lot of time trying to solve scheduler-parsing-related errors. Maybe it is very uncommon to import variables in DAG files or even execute python scripts from files, I don't know.

Anyway, if this is what it is, and it's not some weird error related to some part of my code/deployment, I'd be happy to open a PR adding documentation.

So as I said, this solves the initial error. Now it seems that it executes the command correctly:

airflow@076e96d1e16d:/opt/airflow$ airflow db reset
DB: postgresql+psycopg2://airflow:***@postgres:5432/airflow
This will drop existing tables if they exist. Proceed? (y/n)y
[2021-06-10 12:25:48,085] {db.py:711} INFO - Dropping tables that exist
[2021-06-10 12:25:48,271] {migration.py:164} INFO - Context impl PostgresqlImpl.
[2021-06-10 12:25:48,271] {migration.py:167} INFO - Will assume transactional DDL.
[2021-06-10 12:25:48,382] {db.py:695} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
INFO  [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
INFO  [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
INFO  [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary
INFO  [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, add ti job_id index
INFO  [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing
INFO  [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing
INFO  [alembic.runtime.migration] Running upgrade 27c6a30d7c24 -> 86770d1215c0, add kubernetes scheduler uniqueness
INFO  [alembic.runtime.migration] Running upgrade 86770d1215c0, 0e2a74e0fc9f -> 05f30312d566, merge heads
INFO  [alembic.runtime.migration] Running upgrade 05f30312d566 -> f23433877c24, fix mysql not null constraint
INFO  [alembic.runtime.migration] Running upgrade f23433877c24 -> 856955da8476, fix sqlite foreign key
INFO  [alembic.runtime.migration] Running upgrade 856955da8476 -> 9635ae0956e7, index-faskfail
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> dd25f486b8ea, add idx_log_dag
INFO  [alembic.runtime.migration] Running upgrade dd25f486b8ea -> bf00311e1990, add index to taskinstance
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table
INFO  [alembic.runtime.migration] Running upgrade 0a2a5b66e19d, bf00311e1990 -> 03bc53e68815, merge_heads_2
INFO  [alembic.runtime.migration] Running upgrade 03bc53e68815 -> 41f5f12752f8, add superuser field
INFO  [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> c8ffec048a3b, add fields to dag
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag
INFO  [alembic.runtime.migration] Running upgrade dd4ecb8fbee3 -> 939bb1e647c8, task reschedule fk on cascade delete
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 6e96a59344a4, Make TaskInstance.pool not nullable
INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> d38e04c12aa2, add serialized_dag table
INFO  [alembic.runtime.migration] Running upgrade d38e04c12aa2 -> b3b105409875, add root_dag_id to DAG
INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 74effc47d867, change datetime to datetime2(6) on MSSQL tables
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> a56c9515abdc, Remove dag_stat table
INFO  [alembic.runtime.migration] Running upgrade a56c9515abdc, 004c1210f153, 74effc47d867, b3b105409875 -> 08364691d074, Merge the four heads back together
INFO  [alembic.runtime.migration] Running upgrade 08364691d074 -> fe461863935f, increase_length_for_connection_password
INFO  [alembic.runtime.migration] Running upgrade fe461863935f -> 7939bcff74ba, Add DagTags table
INFO  [alembic.runtime.migration] Running upgrade 7939bcff74ba -> a4c2fd67d16b, add pool_slots field to task_instance
INFO  [alembic.runtime.migration] Running upgrade a4c2fd67d16b -> 852ae6c715af, Add RenderedTaskInstanceFields table
INFO  [alembic.runtime.migration] Running upgrade 852ae6c715af -> 952da73b5eff, add dag_code table
INFO  [alembic.runtime.migration] Running upgrade 952da73b5eff -> a66efa278eea, Add Precision to execution_date in RenderedTaskInstanceFields table
INFO  [alembic.runtime.migration] Running upgrade a66efa278eea -> da3f683c3a5a, Add dag_hash Column to serialized_dag table
INFO  [alembic.runtime.migration] Running upgrade da3f683c3a5a -> 92c57b58940d, Create FAB Tables
INFO  [alembic.runtime.migration] Running upgrade 92c57b58940d -> 03afc6b6f902, Increase length of FAB ab_view_menu.name column
INFO  [alembic.runtime.migration] Running upgrade 03afc6b6f902 -> cf5dc11e79ad, drop_user_and_chart
INFO  [alembic.runtime.migration] Running upgrade cf5dc11e79ad -> bbf4a7ad0465, Remove id column from xcom
INFO  [alembic.runtime.migration] Running upgrade bbf4a7ad0465 -> b25a55525161, Increase length of pool name
INFO  [alembic.runtime.migration] Running upgrade b25a55525161 -> 3c20cacc0044, Add DagRun run_type
INFO  [alembic.runtime.migration] Running upgrade 3c20cacc0044 -> 8f966b9c467a, Set conn_type as non-nullable
INFO  [alembic.runtime.migration] Running upgrade 8f966b9c467a -> 8d48763f6d53, add unique constraint to conn_id
INFO  [alembic.runtime.migration] Running upgrade 8d48763f6d53 -> e38be357a868, Add sensor_instance table
INFO  [alembic.runtime.migration] Running upgrade e38be357a868 -> b247b1e3d1ed, Add queued by Job ID to TI
INFO  [alembic.runtime.migration] Running upgrade b247b1e3d1ed -> e1a11ece99cc, Add external executor ID to TI
INFO  [alembic.runtime.migration] Running upgrade e1a11ece99cc -> bef4f3d11e8b, Drop KubeResourceVersion and KubeWorkerId
INFO  [alembic.runtime.migration] Running upgrade bef4f3d11e8b -> 98271e7606e2, Add scheduling_decision to DagRun and DAG
INFO  [alembic.runtime.migration] Running upgrade 98271e7606e2 -> 52d53670a240, fix_mssql_exec_date_rendered_task_instance_fields_for_MSSQL
INFO  [alembic.runtime.migration] Running upgrade 52d53670a240 -> 364159666cbd, Add creating_job_id to DagRun table
INFO  [alembic.runtime.migration] Running upgrade 364159666cbd -> 45ba3f1493b9, add-k8s-yaml-to-rendered-templates
INFO  [alembic.runtime.migration] Running upgrade 45ba3f1493b9 -> 849da589634d, Prefix DAG permissions.
INFO  [alembic.runtime.migration] Running upgrade 849da589634d -> 2c6edca13270, Resource based permissions.
INFO  [alembic.runtime.migration] Running upgrade 2c6edca13270 -> 61ec73d9401f, Add description field to connection
INFO  [alembic.runtime.migration] Running upgrade 61ec73d9401f -> 64a7d6477aae, fix description field in connection to be text
INFO  [alembic.runtime.migration] Running upgrade 64a7d6477aae -> e959f08ac86c, Change field in DagCode to MEDIUMTEXT for MySql
INFO  [alembic.runtime.migration] Running upgrade e959f08ac86c -> 82b7c48c147f, Remove can_read permission on config resource for User and Viewer role
INFO  [alembic.runtime.migration] Running upgrade 82b7c48c147f -> 449b4072c2da, Increase size of connection.extra field to handle multiple RSA keys
INFO  [alembic.runtime.migration] Running upgrade 449b4072c2da -> 8646922c8a04, Change default pool_slots to 1
INFO  [alembic.runtime.migration] Running upgrade 8646922c8a04 -> 2e42bb497a22, rename last_scheduler_run column
INFO  [alembic.runtime.migration] Running upgrade 2e42bb497a22 -> 90d1635d7b86, Increase pool name size in TaskInstance
INFO  [alembic.runtime.migration] Running upgrade 90d1635d7b86 -> e165e7455d70, add description field to variable
INFO  [alembic.runtime.migration] Running upgrade e165e7455d70 -> a13f7613ad25, Resource based permissions for default FAB views.
INFO  [airflow.models.dagbag.DagBag] Filling up the DagBag from /opt/airflow/dags

Nevertheless, when I enter in the scheduler container logs I get:

[2021-06-10 12:20:51,125] {_internal.py:113} INFO - * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)

Traceback (most recent call last):

File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "job" does not exist
LINE 1: INSERT INTO job (dag_id, state, job_type, start_date, end_da...
^

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 64, in scheduler
job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 233, in run
session.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
_emit_insert_statements(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1135, in _emit_insert_statements
result = cached_connections[connection].execute(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
util.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "job" does not exist
LINE 1: INSERT INTO job (dag_id, state, job_type, start_date, end_da...

^

[SQL: INSERT INTO job (dag_id, state, job_type, start_date, end_date, latest_heartbeat, executor_class, hostname, unixname) VALUES (%(dag_id)s, %(state)s, %(job_type)s, %(start_date)s, %(end_date)s, %(latest_heartbeat)s, %(executor_class)s, %(hostname)s, %(unixname)s) RETURNING job.id]

[parameters: {'dag_id': None, 'state': 'running', 'job_type': 'SchedulerJob', 'start_date': datetime.datetime(2021, 6, 10, 12, 20, 51, 104091, tzinfo=Timezone('UTC')), 'end_date': None, 'latest_heartbeat': datetime.datetime(2021, 6, 10, 12, 20, 51, 104113, tzinfo=Timezone('UTC')), 'executor_class': 'LocalExecutor', 'hostname': '076e96d1e16d', 'unixname': 'airflow'}]

(Background on this error at: http://sqlalche.me/e/13/f405)

I have tried also with db init and db upgrade with no luck. So eventually I am still not able to run airflow 2 =(

[potiuk]

I think you might want to drop the volumes and remove orphans when you bring compose Down. This is the most sure way that you are running things 'from the scratch'. I believe postgres uses named volume to keep the data in our docker-compose.

If that works maybe you could also contribute PR to our docs to explain this cleanup command and add some troubleshooting section?

docker-compose down --volumes --remove-orphans

[potiuk]

Also I saw one more potential problem - if 'logs','dags'','plugins' are owned by root (i. E you have not created them first following the instructions) the whole compose might hung for minutes.

Maybe we should add some kind of self-healing to fix the ownership in this case. I have some idea how to do it but I am on vacations now so I will come back to it after I return (or @mik-laj maybe you can take a look) my idea is that the 'init' might mount the 'current' directory as well and run always as root and change the ownership of those directories to be AIRFLOW_UID/GID owned.

This is BTW only problem for Linux - for MacOS this is not a problem as mounting volumes work differently there (mapping the host user to the use running container) at the expense of being very slow.

[potiuk]

I found some time and I am adding better diagnostics and self-healing in such cases #17484

[JavierLopezT]

Thanks for your answer @potiuk.

It seems your suggestion about docker-compose down might have worked (I am facing other issues right now). Of course, when I have everything solved I'll modify the docs. Thanks!

[potiuk]

We also improved diagnostics of docker compose (we had many similar issues like yours) to guide people in case they have problems. See #17484 - maybe you can take that docker compose and try it (also the 'cleanup' instructions have been added).