override simple auth manager for airflow standalone to use provided config

[bennage]

Description

Our instance of Airflow uses Microsoft Entra of authentication. Our dev team has been testing locally using airflow standalone. We are no longer able to do this because it forces the simple auth manager:

airflow/airflow-core/src/airflow/cli/commands/standalone_command.py

Lines 182 to 185 in 8bee736

	if conf.get("core", "auth_manager") != simple_auth_manager_classpath:
	self.print_output("standalone", "Forcing auth manager to SimpleAuthManager")
	env["AIRFLOW__CORE__AUTH_MANAGER"] = simple_auth_manager_classpath
	os.environ["AIRFLOW__CORE__AUTH_MANAGER"] = simple_auth_manager_classpath # also in this process!

We would like a way to have standalone honor the provided config and not force assumed preferences.

Use case/motivation

We have custom extensions to Airflow that depend on integration with Microsoft Entra. Our dev team often needs to test locally functionality related to this integration. Using airflow standalone has been a convenient method, but it is no longer useful if we can't control the configuration.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

The airflow standalone is designed to be something small for testing. I don't think we can/should support the full capabilities of Airflow.
cc @vincbeck WDYT?

[vincbeck]

I agree with Elad but on the other end if this is something that used to work in Airflow 2, we should continue doing it to avoid friction. And I dont think this is a big change. @bennage , can you confirme this is something that works in Airflow 2?

[bennage]

I can confirm that we were able to use custom auth with airflow standalone in Airflow 2.

[potiuk]

Actually. I think maybe that's the right thing to start discussion on the devlist @bennage. I think this is more of e community decision what "standalone" airflow is targetted for.

So far, standalone airflow was really used to have "first time user experience" in a an easy way - https://airflow.apache.org/docs/apache-airflow/stable/start.html -> this is the primary (and only) use case for now.

However we already had discussions about it - whether we should consider "standalone" to be something more than that.

Example here: https://lists.apache.org/thread/cc7wfz9w8d1npopj72nlxnh52dpzbls9

We did not get to final conclusions there, what was really result of that discussion is that we got rid of Sequential Executor in Airflow 3.

But.. Maybe it's time to revisit it and discuss what we want to do with standalone.

Note that for you @bennage you are raising a problem that it "should" work like it worked in Airflow 2, but this was not really an intended use that you should rely on and not something you should expect to work. Every time someone makes such a claim, they add more maintenance / maintainers work - because effectively what we need to do is to not only "fix" the problem (even if this is not a fix - it's just something you relied on implicitly without any promises from our side), but also we would need to document the behaviour you "assume" is intended, make it works in the future - and in order to do that - add test harness that prevents us from breaking it in the future.

And actually what we usually do in such case we don't do it just "like that" - ideally we ask the person who has incentive to use the "feature" - to actually turn it into the feature (so in this case we will ask you to make all the necessary changes - document the intention, explain how it works in documentation so that people like you would know they can rely on it, fix it, and also add test harness so that it won't get broken in the future.

But all this - we do only after we agree in the community that we want to do it , and that we want to serve that use case and that we want to maintain it in the future. Which - given the intended usage of standalone - is not at all certain.

My suggestion is then @bennage that you start a discussion in the devlist and explain how you can imagine turning standalone into something else that it currently is - in the way that it will make it generic enough for others to use it (and you to use it for your case). Ideally, that should be a complete use case - with actors (who is using standalone, goals - what they want to achieve - and proposal of changes - for example adding new flags to standalone or changing the behaviour. Eventually - if the community agreess (either by lazy consensus or vote), there should be an implementation of all that done by someone (you?) - documenting the intention, usage and tests.

That would be my proposall how you should proceed @bennage

[bennage]

I don't have a strong opinion about what standalone ought to be. Maybe there's a better way to do what we're trying to do.
Here's more context.

We have a custom plugin that extends the native API. I don't think the functionality of the plugin is important, but happy to share if you think it matters. We're using a devcontainer that includes Airflow. Our devs are frequently spinning up Airflow from the devcontainer to test functionality in the plugin or to test a dag. As a convenience, we have been doing that with standalone.

What's the recommended approach?

[potiuk]

Is there any better/more clean way of tackling it?

There is no recommended approach. - it's more of a deployment and dev env decision. You can do a range of things from local venvs, to building docker-compose containrs, or running k8s. Every user has different expectations for those and it's up to you to choose a proper method of deployment that involves the way how you usually share development environment with your teams. We do not have a single recommended approach - but variety of ways you can deploy airflow described in https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html

I think the question is why Standalone with SImpleAuthManager is not something you can adjust to. But if you explain your use case with all the constraints and expectations on the devlist - maybe SimpleAuthManager is somethign that would indeed serve your case. So I - again - propose you start a devlist discussion where you explain how you used it, what is your workflow, and how you adapted it to your case and how you see it can be useful for others in similar way.

Then it might turn into one of the "recommended ways" especially if user like you will contribute back the description, use case, documentation written from the perspective of such user. Big part of our documentation has been created by users turned into contributors who either implemented some changes to fit their use case or simply contributed documentation describing "recommendation" and approach they have taken and generalized it so that others could find it useful. We have > 3300 contributors and many of them are those kind of contibutions are very much welocme.

[potiuk]

And ... I convert it to a discussion - it's not really "issue", it's more of a discussion on the intended use of standalone