Airflow v3 user creation command fails

[markhc]

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Unsure if this is my mistake, but the createUserJob fails on Kubernetes using the Helm chart.

Here's the output of the airflow users command that displays the same error as the createUserJob.

airflow@airflow-api-server-6865b4769f-cvl2c:/opt/airflow$ airflow users
Usage: airflow [-h] GROUP_OR_COMMAND ...

Positional Arguments:
  GROUP_OR_COMMAND

    Groups
      assets            Manage assets
      backfill          Manage backfills
      config            View configuration
      connections       Manage connections
      dags              Manage DAGs
      db                Database operations
      jobs              Manage jobs
      kubernetes        Tools to help run the KubernetesExecutor
      pools             Manage pools
      providers         Display providers
      tasks             Manage tasks
      variables         Manage variables

    Commands:
      api-server        Start an Airflow API server instance
      cheat-sheet       Display cheat sheet
      dag-processor     Start a dag processor instance
      info              Show information about current Airflow and environment
      kerberos          Start a kerberos ticket renewer
      plugins           Dump information about loaded plugins
      rotate-fernet-key
                        Rotate encrypted connection credentials and variables
      scheduler         Start a scheduler instance
      standalone        Run an all-in-one copy of Airflow
      triggerer         Start a triggerer instance
      version           Show the version

Options:
  -h, --help            show this help message and exit

airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'users' (choose from api-server, assets, backfill, cheat-sheet, config, connections, dag-processor, dags, db, info, jobs, kerberos, kubernetes, plugins, p
ools, providers, rotate-fernet-key, scheduler, standalone, tasks, triggerer, variables, version), see help above.

What you think should happen instead?

No response

How to reproduce

Run airflow users -h on v3.0.1 Airflow Docker image.

Operating System

Kubernetes

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployment on GKE using ArgoCD.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tirkarthi]

Do you have Airflow fab provider installed?

https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/cli-ref.html

https://github.com/apache/airflow/blob/main/providers/fab/src/airflow/providers/fab/auth_manager/cli_commands/user_command.py

[markhc]

This is a standard deploy using the provided helm chart. I have not modified any settings related to providers, so if it doesn't come pre-installed I do not have it.

My point is that, in my opinion, a standard installation should work out of the box. As it stands I cannot login to the deployed Airflow instance as no default user has been created.

[markhc]

Also, even after installing apache-airflow-providers-fab the error persists

airflow@airflow-api-server-6865b4769f-cvl2c:/opt/airflow$ pip freeze | grep fab
apache-airflow-providers-fab==2.0.2

airflow@airflow-api-server-6865b4769f-cvl2c:/opt/airflow$ airflow users create
Usage: airflow [-h] GROUP_OR_COMMAND ...

Positional Arguments:
  GROUP_OR_COMMAND

[...] omitted for brevity

Options:
  -h, --help            show this help message and exit

airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'users' (choose from api-server, assets, backfill, cheat-sheet, config, connections, dag-processor, dags, db, info, jobs, kerberos, kubernetes, plugins, p
ools, providers, rotate-fernet-key, scheduler, standalone, tasks, triggerer, variables, version), see help above.

[potiuk]

You need to have FabAuthManager conifgured. The users command is part of the Fab Auth Manaager https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/cli-ref.html.

"Standard" installation now uses SimpleAuthManeager https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/auth-manager/simple/index.html

[markhc]

You need to have FabAuthManager conifgured. The users command is part of the Fab Auth Manaager https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/cli-ref.html.

"Standard" installation now uses SimpleAuthManeager https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/auth-manager/simple/index.html

Do you have an example on how to configure it then? Because I have the package installed and I've set auth_manager = airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager in the [core] section of the config file but both airflow users create and flask fab create-user still fail.

[potiuk]

You need to have FabAuthManager conifgured. The users command is part of the Fab Auth Manaager https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/cli-ref.html.
"Standard" installation now uses SimpleAuthManeager https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/auth-manager/simple/index.html

Do you have an example on how to configure it then? Because I have the package installed and I've set auth_manager = airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager in the [core] section of the config file but both airflow users create and flask fab create-user still fail.

No idea - works for me when I just set it - i guess some settings on your side are not applied in the container:

[markhc]

Somehow reinstalling fixed it. Really not sure what was wrong in my first install. The only thing I changed was setting FAB as the auth manager in the Helm chart from the beginning (my first install was using SimpleAuth at first).

Thanks!

[mrbungie]

It also failed for me when using the helm chart, in fact, it does not show up in airflow subcommands but fab is installed and used for Oauth against Azure. Very weird.

[potiuk]

It also failed for me when using the helm chart, in fact, it does not show up in airflow subcommands but fab is installed and used for Oauth against Azure. Very weird.

I think it depends on how you are running the airflow command and whether the container you are running it in has the FAB provider installed and FAB Auth manager configured. Can you check it ?

[mrbungie]

First of all, and as always thank you so much for Apache Airflow :), I've depended on it for years for many projects.

And yes, as far as I can tell we do have it installed. Some "evidence" that may help, short of showing the running system.

1. I'm using a custom docker image (but based on apache/airflow:3.0.0-python3.12) that includes this in its requirements: apache-airflow-providers-fab==2.0.1. This is a gist with its definition and related files: https://gist.github.com/mrbungie/ab8e2a2104e56c7d6dc8872d01d44f9a
2. I'm setting the following relevant values in my helm chart: https://gist.github.com/mrbungie/eaae9aed11d04b04d960376788d3e91e (here im showing it with webserver.defaultUser = false, but when I set it to true, the createUser fails)
3. We implemented a webserver_config.py (shown in [1] gist integrated with EntraID). And it works, so FAB is working.

When I get into most containers, they do have the users subcommand:

But when checking the create-user container (after setting createUser to true), it fails.

I'm pretty sure it has to do with the fact that the env vars overriding Auth are not reaching the createUser container.
create-user envvars:

Meanwhile other containers (like api-server)

I guess that means that that setting (i.e. airflow.core.auth_manager / AIRFLOW__CORE__AUTH_MANAGER) should be set exclusively via airflow.cfg, but I think docs should be clearer about that.

If that's not the hoped behaviour, I would guess AIRFLOW__CORE__AUTH_MANAGER, if set, should be propagated down into the create-user container (after all, if its going to run create-user in Airflow 3, it should have that set to FAB or it is always going to fail). Maybe the create-user container should check if FAB is on (via env var + airflow.cfg), and if not, it should just continue gracefully and maybe logging that so users are advised to set createUser = false if they are just using Simple Auth.

If anything I'm saying makes sense, I can make a PR fixing this in some direction you propose.

More info:
Helm chart commit SHA256: 3bd9746