`celery --loader` option ignored

[0x2b3bfa0]

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the contribution guide on reporting bugs.
• I have checked the issues list for similar or identical bug reports.
• I have checked the pull requests list for existing proposed fixes.
• I have checked the commit log to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).
• I have tried to reproduce the issue with pytest-celery and added the reproduction script below.
  • I haven't found any elegant and concise way of using pytest to reproduce a command-line option parsing bug, and if I have to choose how to spend my scarce time, I prefer to write a meaningful issue rather than struggle shoehorning my test case into a test suite I'm not familiar with.

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required to reproduce this bug.

Related Issues and Possible Duplicates

Related Issues

• --loader option ignored #1066

Possible Duplicates

• None

Environment & Settings

celery --version output

$ celery --version
5.5.0rc1 (immunity)

celery report output

$ celery --app example --loader example.ExampleAppLoader report

software -> celery:5.5.0rc1 (immunity) kombu:5.4.0 py:3.12.3
            billiard:4.2.0 memory:N/A
platform -> system:Linux arch:64bit, ELF
            kernel version:6.5.0-1025-azure imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:memory results:db+sqlite:///results.sqlite

deprecated_settings: None

Note

The output of the loader field on celery ... report already suffices to reproduce this issue: it displays celery.loaders.app.AppLoader instead of example.AppLoader as it should.

Python Packages

pip freeze output

$ pip freeze
amqp==5.2.0
billiard==4.2.1
celery @ file:///celery # main branch
click==8.1.7
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
kombu==5.4.2
prompt_toolkit==3.0.48
python-dateutil==2.9.0.post0
six==1.16.0
tzdata==2024.2
vine==5.1.0
wcwidth==0.2.13

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A
• Minimal Celery Version: Unknown
• Minimal Kombu Version: N/A
• Minimal Broker Version: N/A
• Minimal Result Backend Version: N/A
• Minimal OS and/or Kernel Version: N/A
• Minimal Broker Client Version: N/A
• Minimal Result Backend Client Version: N/A

Minimally Reproducible Test Case

example.py

from celery import Celery
from celery.loaders.app import AppLoader


class ExampleAppLoader(AppLoader):
    def on_worker_init(self):
        print("ExampleAppLoader.on_worker_init() called")


app = Celery("example")    

example.sh

celery --app example --loader example.ExampleAppLoader worker

Expected Behavior

The command prints ExampleAppLoader.on_worker_init() called among other things.

Actual Behavior

The celery --loader option is ignored, although manually setting the CELERY_LOADER environment variable is still a viable workaround.

Explanation

It turns out the Celery application determines the default loader from the CELERY_LOADER environment variable:

celery/celery/app/base.py

Lines 391 to 397 in 03e3359

	def _get_default_loader(self):
	# the --loader command-line argument sets the environment variable.
	return (
	os.environ.get('CELERY_LOADER') or
	self.loader_cls or
	'celery.loaders.app:AppLoader'
	)

This environment variable is being set from the click function in charge of parsing the celery top level command-line options:

celery/celery/bin/celery.py

Lines 153 to 154 in 03e3359

	# Default app takes loader from this env (Issue #1066).
	os.environ['CELERY_LOADER'] = loader

Unfortunately, by the time we set that environment variable, the application has already been imported and the Celery object has already been initialized:

celery/celery/bin/celery.py

Lines 51 to 76 in 03e3359

	class App(ParamType):
	"""Application option."""
	name = "application"
	def convert(self, value, param, ctx):
	try:
	return find_app(value)
	except ModuleNotFoundError as e:
	if e.name != value:
	exc = traceback.format_exc()
	self.fail(
	UNABLE_TO_LOAD_APP_ERROR_OCCURRED.format(value, exc)
	)
	self.fail(UNABLE_TO_LOAD_APP_MODULE_NOT_FOUND.format(e.name))
	except AttributeError as e:
	attribute_name = e.args[0].capitalize()
	self.fail(UNABLE_TO_LOAD_APP_APP_MISSING.format(attribute_name))
	except Exception:
	exc = traceback.format_exc()
	self.fail(
	UNABLE_TO_LOAD_APP_ERROR_OCCURRED.format(value, exc)
	)
	APP = App()

celery/celery/bin/celery.py

Lines 90 to 95 in 03e3359

	@click.option('-A',
	'--app',
	envvar='APP',
	cls=CeleryOption,
	type=APP,
	help_group="Global Options")