-
Notifications
You must be signed in to change notification settings - Fork 13.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BROKER_URL_SECRET Not working as of Airflow 2.7 #34612
Comments
Playing around with it I discovered calling import os
# Quick hack for the issue instead of using a proper fixture
os.environ.pop("AIRFLOW__CELERY__BROKER_URL")
os.environ["AIRFLOW__CELERY__BROKER_URL_SECRET"] = "broker_url_east"
def test_airflow_cli_uses_broker_url_secret(capsys):
"""
Shows that executing through the CLI changes
the behavior of the conf.get command
in the config_command::get_value function
"""
from airflow.cli import cli_parser
from airflow.configuration import conf
default_redis = "redis://redis:6379/0"
assert default_redis == conf.get("celery", "broker_url")
test_args = ["config", "get-value", "celery", "broker_url"]
parser = cli_parser.get_parser()
args = parser.parse_args(test_args)
args.func(args)
sout = capsys.readouterr()
secret_redis_url = None
for line in sout.out.splitlines():
line: str = line.strip()
if line.startswith("redis"):
secret_redis_url = line
break
if secret_redis_url is None:
raise AssertionError(
"Could not get secret from backend!"
)
assert secret_redis_url and secret_redis_url != default_redis |
Now I am very confused because in my debugger I can see that cli Regardless, now I think it has something to do with the timing of |
I confirm that there is a problem on the configurations loading in the CLI. |
Hi! Looks like this issue also breaks airflow helm chart installation when using |
I think I have a fix for this ready to go but I cannot think of a way to properly test it since the patch fixes import time side effects. I should have a PR soon and someone wiser than me can help add a test if necessary. |
While I understand why the liveness probe mentioned by @Aakcht (because it runs the app directly) in Helm chart could fail in this case, I do not think the "worker" should have this problem. I have no idea how this would not work for
The @providers_configuration_loaded
def worker(args): Is there any special scenario where Celery App is run as separately started Python interpreter when worker is started? Do you have any other specific configuration ? @hussein-awala - can you show how you reproduced it ? |
Apache Airflow version
2.7.1
What happened
Hi team,
In the past you can use
AIRFLOW__CELERY__BROKER_URL_SECRET
as a way to retrieve the credentials from aSecretBackend
at runtime. However, as of Airflow 2.7, this technique appears to be broken. I believe this related to the discussion 34030 - Celery configuration elements not shown to be fetched with _CMD pattern. The irony is the pattern works when using theconfig get-value
command, but does not work when using the actualairflow celery command
. I suspect this has something to do with when the wrapper callsProvidersManager().initialize_providers_configuration()
.This correct prints the secret from the backend!
However actually executing celery with the same methodolgy results in the default Redis
Relevant output
Notice the redis/redis and default port with no password.
What you think should happen instead
I would expect the airflow celery command to be able to leverage the
_secret
API similar to theconfig
command.How to reproduce
You must use a secret back end to reproduce as described above. You can also do
AIRFLOW__CELERY__BROKER_URL_CMD='/usr/bin/env bash -c "echo -n ZZZ"' airflow celery worker
And you will see the ZZZ is disregarded
It appears neither historical _CMD or _SECRET APIs work after the refactor to move celery to the providers.
Operating System
ubuntu20.04
Versions of Apache Airflow Providers
Relevant ones
apache-airflow-providers-celery 3.3.3
celery 5.3.4
Deployment
Docker-Compose
Deployment details
No response
Anything else
I know this has something to do with when
ProvidersManager().initialize_providers_configuration()
is executed but I don't know the right place to put the fix.Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: