Airflow API authentication using Kerberos is not working from the version 2.8.0

[jijoj-hmetrix]

We have set up Airflow API authentication using Kerberos. It was working until version 2.7.3 of Airflow.

(airflow_virtual) [airflow1@airflowtest airflow]$ airflow version
2.7.3
(airflow_virtual) [airflow1@airflowtest airflow]$

[user@server ~]$ curl -k -u : --negotiate https://airflowtest.xxxxxx.com:8081/api/v1/pools --service-name airflow
{
  "pools": [
    {
      "deferred_slots": 0,
      "description": "Default pool",
      "include_deferred": false,
      "name": "default_pool",
      "occupied_slots": 0,
      "open_slots": 128,
      "queued_slots": 0,
      "running_slots": 0,
      "scheduled_slots": 0,
      "slots": 128
    }
  ],
  "total_entries": 1
}
[user@server ~]$

Recently, we upgraded to version 2.8.0. After the update, the API stopped working with the following error. There are no specific entries to identify the issue in the debug log. Can someone help me resolve this?

(airflow_virtual) [airflow1@airflowtest airflow]$ airflow version
2.8.0
(airflow_virtual) [airflow1@airflowtest airflow]$

[user@server ~]$ curl -k -u : --negotiate https://airflowtest.xxxxxx.com:8081/api/v1/pools --service-name airflow
{
  "detail": null,
  "status": 403,
  "title": "Forbidden",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.8.0/stable-rest-api-ref.html#section/Errors/PermissionDenied"
}
[user@server ~]$

[brouberol]

I stumbled upon the same issue, but I realized that it wasn't a Kerberos issue. I think it has to due with /api/v1/pools not being exposed anymore, or at least not supporting the Kerberos auth backend?

I'm running Airflow 2.10.2, and I was getting the same 403 response even when running un-authenticated. However, the following works (after having re-enabled API kerberos authentication):

$ curl --negotiate -u: -s --service-name airflow https://airflow-test.xxxx.com/api/experimental/pools  | jq .
[
  {
    "description": "Default pool",
    "id": 1,
    "include_deferred": false,
    "pool": "default_pool",
    "slots": 128
  }
]

However, with the same deployment, hitting /api/v1/pools returns the same 403 that you're getting:

$ curl --negotiate -u: -s --service-name airflow https://airflow-test.xxxx.com/api/v1/pools  | jq .
{
  "detail": null,
  "status": 403,
  "title": "Forbidden",
  "type": "https://airflow.apache.org/docs/apache-airflow/2.10.2/stable-rest-api-ref.html#section/Errors/PermissionDenied"
}

[potiuk]

I think I'd recommend you to upgrade to much newer version of airflow. Upgrading to 2.8.0 (or any .0) makes very little sense in general, because almost by definition it has bugs (All the bugs - and only bugs) that were fixed in 2.8.1, 2.8.2, 2.8.3 and 2.8.4 - and since there were some teething problems with migration to Auth Managers in 2.8.0, it's very likely you are hitting those bugs that were fixed in one of the later versions of 2.8.*

So first thing first. Upgrade to latest patchlevel of newer version of airflow (ideally latest patchlevel of 2.10 - that's always my recommendation to upgrade to latest version).

Then - I think you should open an issue if it does not work for you - there were some changes in how configuration of APIs are recommended, so I recommend to read the appropriate docs and check if your configuration is properly done. For example Kerberos auth is now part of the FAB provider and ideally you should update your documentation to follow that https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/api-authentication.html

This is all described in release notes - so you can also read those - between the version you started from and migrated to.

Once you check it all and upgrade and update the configuration if you find things not working, open an issue please with some details - there shoud be some log indicating errors in case you see the authentication not working - and people who will look at the issue need to know all the details to be able to help you.

BTW:

airflow https://airflow-test.xxxx.com/api/experimental/pools

is an experimental API that is going to be removed in Airflow 3, so you should rather investigate the authentication issues you have with the REST API

[brouberol]

Thanks for writing back @potiuk! I was reading the documentation last evening, and I realized I misunderstood something. I believe the issue to be a permission / ACL issue, as:

• the stable API requires a "minimum" role to be able to interact with some resources (https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/access-control.html)
• the experimental API does not rely on ACLs. If you're authenticated, you have full access to all experimental API handlers (https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#enable-experimental-api)
• authentication cannot be disabled for the stable API (this is why I was getting 403 errors even with the default auth backend)

In my case, my user is Admin, so I need to figure out why it's getting a PermissionDenied error, but I believe this to be the root difference between the 2 cases.

[brouberol]

Ok. so here's what I found. The default auth_manager is airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager, which has a get_user method relying on flask-login's current_user to get the currently logged in user from the session.

As it stands, it seems that Kerberos authentication cannot work with the stable API, as the kerberos_auth authentication backend stores the user associated with the Kerberos token in g, not in the session.

So, what I've done to make the airflow stable API work with Kerberos authentication is define a custom auth manager:

# custom_manager.py
from airflow.providers.fab.auth_manager.fab_auth_manager import FabAuthManager
from airflow.providers.fab.auth_manager.models import User


class CustomAuthManager(FabAuthManager):
    """An authentication manager supporting both session and Keberos authentication for the Airflow stable API."""

    def get_user(self) -> User:
        """Attempt to find the current user in g.user, as defined by the kerberos authentication backend.

        If no such user is found, return the `current_user` local proxy object, linked to the user session.

        """
        from flask_login import current_user
        from flask import g

        # If a user has gone through the Kerberos dance, the kerberos authentication manager
        # has linked it with a User model, stored in g.user, and not the session.
        if (
            current_user.is_anonymous
            and getattr(g, "user", None) is not None
            and not g.user.is_anonymous
        ):
            return g.user
        return super().get_user()

and defined the following airflow.cfg config:

[core]
auth_manager = path.to.custom_auth_manager.CustomAuthManager

And with that, I was able to query the stable API with a kerberos token:

$ curl --negotiate -u: -s --service-name airflow https://airflow-test.xxx.com/api/v1/pools
{
  "pools": [
    {
      "deferred_slots": 0,
      "description": "Default pool",
      "include_deferred": false,
      "name": "default_pool",
      "occupied_slots": 0,
      "open_slots": 128,
      "queued_slots": 0,
      "running_slots": 0,
      "scheduled_slots": 0,
      "slots": 128
    }
  ],
  "total_entries": 1
}

[potiuk]

Aha.. Maybe submitting it back to Airflow would be a good idea ? cc: @vincbeck

[brouberol]

Absolutely! I'm still wrapping my head around whether this is a good fix or not, but if it is, I will do so.