Airflow API kerberos authentication error

[BMFH]

Apache Airflow version

2.5.0

What happened

Configured AUTH_DB authentication for web server and Kerberos authentication for API. Web server works well.
Try to get any API endpoint and get an error 500. I see Kerberos authentication step is done, but authorization step fails.
'User' object (now it is just a string) doesn't have such parameter.

Request error

янв 13 13:54:14 nginx-test airflow[238738]: [2023-01-13 13:54:14,923] {app.py:1741} ERROR - Exception on /api/v1/dags [GET]
янв 13 13:54:14 nginx-test airflow[238738]: Traceback (most recent call last):
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 2525, in wsgi_app
янв 13 13:54:14 nginx-test airflow[238738]:     response = self.full_dispatch_request()
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1822, in full_dispatch_request
янв 13 13:54:14 nginx-test airflow[238738]:     rv = self.handle_user_exception(e)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request
янв 13 13:54:14 nginx-test airflow[238738]:     rv = self.dispatch_request()
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request
янв 13 13:54:14 nginx-test airflow[238738]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/connexion/decorators/decorator.py", line 68, in wrapper
янв 13 13:54:14 nginx-test airflow[238738]:     response = function(request)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
янв 13 13:54:14 nginx-test airflow[238738]:     response = function(request)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/connexion/decorators/validation.py", line 399, in wrapper
янв 13 13:54:14 nginx-test airflow[238738]:     return function(request)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/connexion/decorators/response.py", line 112, in wrapper
янв 13 13:54:14 nginx-test airflow[238738]:     response = function(request)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/connexion/decorators/parameter.py", line 120, in wrapper
янв 13 13:54:14 nginx-test airflow[238738]:     return function(**kwargs)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/airflow/api_connexion/security.py", line 50, in decorated
янв 13 13:54:14 nginx-test airflow[238738]:     if appbuilder.sm.check_authorization(permissions, kwargs.get("dag_id")):
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/airflow/www/security.py", line 715, in check_authorization
янв 13 13:54:14 nginx-test airflow[238738]:     can_access_all_dags = self.has_access(*perm)
янв 13 13:54:14 nginx-test airflow[238738]:   File "/usr/local/lib/python3.8/dist-packages/airflow/www/security.py", line 419, in has_access
янв 13 13:54:14 nginx-test airflow[238738]:     if (action_name, resource_name) in user.perms:
янв 13 13:54:14 nginx-test airflow[238738]: AttributeError: 'str' object has no attribute 'perms'
янв 13 13:54:14 nginx-test airflow[238738]: 127.0.0.1 - - [13/Jan/2023:13:54:14 +0300] "GET /api/v1/dags HTTP/1.1" 500 1561 "-" "curl/7.68.0"

Starting airflow-webserver log (no errors)

янв 13 13:38:51 nginx-test airflow[238502]:   ____________       _____________
янв 13 13:38:51 nginx-test airflow[238502]:  ____    |__( )_________  __/__  /________      __
янв 13 13:38:51 nginx-test airflow[238502]: ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
янв 13 13:38:51 nginx-test airflow[238502]: ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
янв 13 13:38:51 nginx-test airflow[238502]:  _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
янв 13 13:38:51 nginx-test airflow[238502]: Running the Gunicorn Server with:
янв 13 13:38:51 nginx-test airflow[238502]: Workers: 4 sync
янв 13 13:38:51 nginx-test airflow[238502]: Host: 0.0.0.0:10000
янв 13 13:38:51 nginx-test airflow[238502]: Timeout: 120
янв 13 13:38:51 nginx-test airflow[238502]: Logfiles: - -
янв 13 13:38:51 nginx-test airflow[238502]: Access Logformat:
янв 13 13:38:51 nginx-test airflow[238502]: =================================================================
янв 13 13:38:51 nginx-test airflow[238502]: [2023-01-13 13:38:51,209] {webserver_command.py:431} INFO - Received signal: 15. Closing gunicorn.
янв 13 13:38:51 nginx-test airflow[238519]: [2023-01-13 13:38:51 +0300] [238519] [WARNING] Worker with pid 238525 was terminated due to signal 15
янв 13 13:38:51 nginx-test airflow[238519]: [2023-01-13 13:38:51 +0300] [238519] [WARNING] Worker with pid 238523 was terminated due to signal 15
янв 13 13:38:51 nginx-test airflow[238519]: [2023-01-13 13:38:51 +0300] [238519] [WARNING] Worker with pid 238526 was terminated due to signal 15
янв 13 13:38:51 nginx-test airflow[238519]: [2023-01-13 13:38:51 +0300] [238519] [WARNING] Worker with pid 238524 was terminated due to signal 15
янв 13 13:38:51 nginx-test airflow[238519]: [2023-01-13 13:38:51 +0300] [238519] [INFO] Shutting down: Master
янв 13 13:38:52 nginx-test systemd[1]: airflow-webserver.service: Succeeded.
янв 13 13:38:52 nginx-test systemd[1]: Stopped Airflow webserver daemon.
янв 13 13:38:52 nginx-test systemd[1]: Started Airflow webserver daemon.
янв 13 13:38:54 nginx-test airflow[238732]: /usr/local/lib/python3.8/dist-packages/airflow/api/auth/backend/kerberos_auth.py:50 DeprecationWarning: '_request_ctx_stack' is dep>
янв 13 13:38:54 nginx-test airflow[238732]: [2023-01-13 13:38:54,393] {kerberos_auth.py:78} INFO - Kerberos: hostname nginx-test.mycompany
янв 13 13:38:54 nginx-test airflow[238732]: [2023-01-13 13:38:54,393] {kerberos_auth.py:88} INFO - Kerberos init: airflow nginx-test.mycompany
янв 13 13:38:54 nginx-test airflow[238732]: [2023-01-13 13:38:54,394] {kerberos_auth.py:93} INFO - Kerberos API: server is airflow/nginx-test.mycompany@MYCOMPANY>
янв 13 13:38:56 nginx-test airflow[238732]: [2023-01-13 13:38:56 +0300] [238732] [INFO] Starting gunicorn 20.1.0
янв 13 13:38:56 nginx-test airflow[238732]: [2023-01-13 13:38:56 +0300] [238732] [INFO] Listening at: http://0.0.0.0:10000 (238732)
янв 13 13:38:56 nginx-test airflow[238732]: [2023-01-13 13:38:56 +0300] [238732] [INFO] Using worker: sync
янв 13 13:38:56 nginx-test airflow[238735]: [2023-01-13 13:38:56 +0300] [238735] [INFO] Booting worker with pid: 238735
янв 13 13:38:57 nginx-test airflow[238736]: [2023-01-13 13:38:57 +0300] [238736] [INFO] Booting worker with pid: 238736
янв 13 13:38:57 nginx-test airflow[238737]: [2023-01-13 13:38:57 +0300] [238737] [INFO] Booting worker with pid: 238737
янв 13 13:38:57 nginx-test airflow[238738]: [2023-01-13 13:38:57 +0300] [238738] [INFO] Booting worker with pid: 238738

I tried to skip rights check, commenting problem lines and returning True from has_access function and if I remember it right in one more function from security.py. And I got it working. But it has been just a hack to check where is the problem.

What you think should happen instead

It should return right json answer with code 200.

How to reproduce

1. webserver_config.py: default

2. airflow.cfg changed lines:

[core]
security = kerberos
[api]
auth_backends = airflow.api.auth.backend.kerberos_auth,airflow.api.auth.backend.session

[kerberos]
ccache = /tmp/airflow_krb5_ccache
principal = airflow/nginx-test.mycompany
reinit_frequency = 3600
kinit_path = kinit
keytab = /root/airflow/airflow2.keytab
forwardable = True
include_ip = True

[webserver]
base_url = http://localhost:10000
web_server_port = 10000

3. Create keytab file with airflow principal

4. Log in as domain user, make request (for example):
   curl --verbose --negotiate -u : http://nginx-test.mycompany:10000/api/v1/dags

Operating System

Ubuntu. VERSION="20.04.5 LTS (Focal Fossa)"

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

How do you expect your kerberos user to map to Airflow @BMFH? I believe indeed the current way kerberos user is retrieved is wrong. From what I see. the user that is set by kerberos_auth is kerberos_id string, but what we expect there is the user object from FAB.

If - for example you have email address then likely this one should fix it #29054 - maybe you can apply the same change and see if that fixes the problem (or explore my hypothesis a bit more?)

[BMFH]

@potiuk Thank you for answer! I guessed if we do everything exactly as described in the documentation it would work. But it seems I was wrong.
I'm not a developer, I'm just a DevOps engineer and I have not idea how it should work exactly, but I try to understand the logic.

1. What we have before:
   I added some logging in the security.py and I can see that user variable = "domain_user_name@KERBEROS-REALM".
   In my case it is
   янв 20 11:27:29 nginx-test airflow[677767]: [2023-01-20 11:27:29,292] {security.py:418} INFO - !user is Dmitriy.Kondratyev@CORP.mycompany.DIGITAL
   I tried to add Airflow user with this username, but it doesn't work.

2. I added your fix and create user with email = domain_user_name@KERBEROS-REALM
   It works. User was authenticated.
   And here we have little problem. Email domain is not always have the same name as the kerberos realm.

3. I changed your code for looking username parameter.
   g.user = get_airflow_app().appbuilder.sm.find_user(username=ctx.kerberos_user)
   Created user with username "domain_user_name@KERBEROS-REALM" and it works!

So, to make kerberos auth work with API interface we need to apply this fix (with searching by username) and create an airflow user with username=domain_user@KERBEROS-REALM.

[potiuk]

Ok. Cool Thanks for cofirming :). this sounds cool, providing that the username match :). I will adapt the fix and make some documentation updates to explain it :)

[potiuk]

I made you co-author of the fix @BMFH - see if what I got is good please :)