Remove configuration compatibility shims from 3.x

[clokep]

Celery 4.0 introduced new configuration setting names (see the documentation), but maintained compatibility with the old names. This seems to be a source of confusion (and I've witnessed numerous issues filed that are due to configuration problems). 5.0 seems like a good time to simplify this and have only a single standard form for these configuration variables.

I believe the code lives around [here]

celery/celery/app/defaults.py

Lines 76 to 341 in 5b86b35

	NAMESPACES = Namespace(
	accept_content=Option(DEFAULT_ACCEPT_CONTENT, type='list', old=OLD_NS),
	result_accept_content=Option(None, type='list'),
	enable_utc=Option(True, type='bool'),
	imports=Option((), type='tuple', old=OLD_NS),
	include=Option((), type='tuple', old=OLD_NS),
	timezone=Option(type='string', old=OLD_NS),
	beat=Namespace(
	__old__=OLD_NS_BEAT,
	max_loop_interval=Option(0, type='float'),
	schedule=Option({}, type='dict'),
	scheduler=Option('celery.beat:PersistentScheduler'),
	schedule_filename=Option('celerybeat-schedule'),
	sync_every=Option(0, type='int'),
	),
	broker=Namespace(
	url=Option(None, type='string'),
	read_url=Option(None, type='string'),
	write_url=Option(None, type='string'),
	transport=Option(type='string'),
	transport_options=Option({}, type='dict'),
	connection_timeout=Option(4, type='float'),
	connection_retry=Option(True, type='bool'),
	connection_max_retries=Option(100, type='int'),
	failover_strategy=Option(None, type='string'),
	heartbeat=Option(120, type='int'),
	heartbeat_checkrate=Option(3.0, type='int'),
	login_method=Option(None, type='string'),
	pool_limit=Option(10, type='int'),
	use_ssl=Option(False, type='bool'),
	host=Option(type='string'),
	port=Option(type='int'),
	user=Option(type='string'),
	password=Option(type='string'),
	vhost=Option(type='string'),
	),
	cache=Namespace(
	__old__=old_ns('celery_cache'),
	backend=Option(),
	backend_options=Option({}, type='dict'),
	),
	cassandra=Namespace(
	entry_ttl=Option(type='float'),
	keyspace=Option(type='string'),
	port=Option(type='string'),
	read_consistency=Option(type='string'),
	servers=Option(type='list'),
	table=Option(type='string'),
	write_consistency=Option(type='string'),
	auth_provider=Option(type='string'),
	auth_kwargs=Option(type='string'),
	options=Option({}, type='dict'),
	),
	s3=Namespace(
	access_key_id=Option(type='string'),
	secret_access_key=Option(type='string'),
	bucket=Option(type='string'),
	base_path=Option(type='string'),
	endpoint_url=Option(type='string'),
	region=Option(type='string'),
	),
	azureblockblob=Namespace(
	container_name=Option('celery', type='string'),
	retry_initial_backoff_sec=Option(2, type='int'),
	retry_increment_base=Option(2, type='int'),
	retry_max_attempts=Option(3, type='int'),
	),
	control=Namespace(
	queue_ttl=Option(300.0, type='float'),
	queue_expires=Option(10.0, type='float'),
	exchange=Option('celery', type='string'),
	),
	couchbase=Namespace(
	__old__=old_ns('celery_couchbase'),
	backend_settings=Option(None, type='dict'),
	),
	arangodb=Namespace(
	__old__=old_ns('celery_arangodb'),
	backend_settings=Option(None, type='dict')
	),
	mongodb=Namespace(
	__old__=old_ns('celery_mongodb'),
	backend_settings=Option(type='dict'),
	),
	cosmosdbsql=Namespace(
	database_name=Option('celerydb', type='string'),
	collection_name=Option('celerycol', type='string'),
	consistency_level=Option('Session', type='string'),
	max_retry_attempts=Option(9, type='int'),
	max_retry_wait_time=Option(30, type='int'),
	),
	event=Namespace(
	__old__=old_ns('celery_event'),
	queue_expires=Option(60.0, type='float'),
	queue_ttl=Option(5.0, type='float'),
	queue_prefix=Option('celeryev'),
	serializer=Option('json'),
	exchange=Option('celeryev', type='string'),
	),
	redis=Namespace(
	__old__=old_ns('celery_redis'),
	backend_use_ssl=Option(type='dict'),
	db=Option(type='int'),
	host=Option(type='string'),
	max_connections=Option(type='int'),
	password=Option(type='string'),
	port=Option(type='int'),
	socket_timeout=Option(120.0, type='float'),
	socket_connect_timeout=Option(None, type='float'),
	retry_on_timeout=Option(False, type='bool'),
	socket_keepalive=Option(False, type='bool'),
	),
	result=Namespace(
	__old__=old_ns('celery_result'),
	backend=Option(type='string'),
	cache_max=Option(
	-1,
	type='int', old={'celery_max_cached_results'},
	),
	compression=Option(type='str'),
	exchange=Option('celeryresults'),
	exchange_type=Option('direct'),
	expires=Option(
	timedelta(days=1),
	type='float', old={'celery_task_result_expires'},
	),
	persistent=Option(None, type='bool'),
	extended=Option(False, type='bool'),
	serializer=Option('json'),
	backend_transport_options=Option({}, type='dict'),
	chord_retry_interval=Option(1.0, type='float'),
	chord_join_timeout=Option(3.0, type='float'),
	backend_max_sleep_between_retries_ms=Option(10000, type='int'),
	backend_max_retries=Option(float("inf"), type='float'),
	backend_base_sleep_between_retries_ms=Option(10, type='int'),
	backend_always_retry=Option(False, type='bool'),
	),
	elasticsearch=Namespace(
	__old__=old_ns('celery_elasticsearch'),
	retry_on_timeout=Option(type='bool'),
	max_retries=Option(type='int'),
	timeout=Option(type='float'),
	save_meta_as_text=Option(True, type='bool'),
	),
	riak=Namespace(
	__old__=old_ns('celery_riak'),
	backend_settings=Option(type='dict'),
	),
	security=Namespace(
	__old__=old_ns('celery_security'),
	certificate=Option(type='string'),
	cert_store=Option(type='string'),
	key=Option(type='string'),
	digest=Option(DEFAULT_SECURITY_DIGEST, type='string'),
	),
	database=Namespace(
	url=Option(old={'celery_result_dburi'}),
	engine_options=Option(
	type='dict', old={'celery_result_engine_options'},
	),
	short_lived_sessions=Option(
	False, type='bool', old={'celery_result_db_short_lived_sessions'},
	),
	table_schemas=Option(type='dict'),
	table_names=Option(type='dict', old={'celery_result_db_tablenames'}),
	),
	task=Namespace(
	__old__=OLD_NS,
	acks_late=Option(False, type='bool'),
	acks_on_failure_or_timeout=Option(True, type='bool'),
	always_eager=Option(False, type='bool'),
	annotations=Option(type='any'),
	compression=Option(type='string', old={'celery_message_compression'}),
	create_missing_queues=Option(True, type='bool'),
	inherit_parent_priority=Option(False, type='bool'),
	default_delivery_mode=Option(2, type='string'),
	default_queue=Option('celery'),
	default_exchange=Option(None, type='string'), # taken from queue
	default_exchange_type=Option('direct'),
	default_routing_key=Option(None, type='string'), # taken from queue
	default_rate_limit=Option(type='string'),
	default_priority=Option(None, type='string'),
	eager_propagates=Option(
	False, type='bool', old={'celery_eager_propagates_exceptions'},
	),
	ignore_result=Option(False, type='bool'),
	protocol=Option(2, type='int', old={'celery_task_protocol'}),
	publish_retry=Option(
	True, type='bool', old={'celery_task_publish_retry'},
	),
	publish_retry_policy=Option(
	{'max_retries': 3,
	'interval_start': 0,
	'interval_max': 1,
	'interval_step': 0.2},
	type='dict', old={'celery_task_publish_retry_policy'},
	),
	queues=Option(type='dict'),
	queue_ha_policy=Option(None, type='string'),
	queue_max_priority=Option(None, type='int'),
	reject_on_worker_lost=Option(type='bool'),
	remote_tracebacks=Option(False, type='bool'),
	routes=Option(type='any'),
	send_sent_event=Option(
	False, type='bool', old={'celery_send_task_sent_event'},
	),
	serializer=Option('json', old={'celery_task_serializer'}),
	soft_time_limit=Option(
	type='float', old={'celeryd_task_soft_time_limit'},
	),
	time_limit=Option(
	type='float', old={'celeryd_task_time_limit'},
	),
	store_errors_even_if_ignored=Option(False, type='bool'),
	track_started=Option(False, type='bool'),
	),
	worker=Namespace(
	__old__=OLD_NS_WORKER,
	agent=Option(None, type='string'),
	autoscaler=Option('celery.worker.autoscale:Autoscaler'),
	concurrency=Option(0, type='int'),
	consumer=Option('celery.worker.consumer:Consumer', type='string'),
	direct=Option(False, type='bool', old={'celery_worker_direct'}),
	disable_rate_limits=Option(
	False, type='bool', old={'celery_disable_rate_limits'},
	),
	enable_remote_control=Option(
	True, type='bool', old={'celery_enable_remote_control'},
	),
	hijack_root_logger=Option(True, type='bool'),
	log_color=Option(type='bool'),
	log_format=Option(DEFAULT_PROCESS_LOG_FMT),
	lost_wait=Option(10.0, type='float', old={'celeryd_worker_lost_wait'}),
	max_memory_per_child=Option(type='int'),
	max_tasks_per_child=Option(type='int'),
	pool=Option(DEFAULT_POOL),
	pool_putlocks=Option(True, type='bool'),
	pool_restarts=Option(False, type='bool'),
	proc_alive_timeout=Option(4.0, type='float'),
	prefetch_multiplier=Option(4, type='int'),
	redirect_stdouts=Option(
	True, type='bool', old={'celery_redirect_stdouts'},
	),
	redirect_stdouts_level=Option(
	'WARNING', old={'celery_redirect_stdouts_level'},
	),
	send_task_events=Option(
	False, type='bool', old={'celery_send_events'},
	),
	state_db=Option(),
	task_log_format=Option(DEFAULT_TASK_LOG_FMT),
	timer=Option(type='string'),
	timer_precision=Option(1.0, type='float'),
	),
	)

).

[thedrow]

I gave this a shot.
I'll ping you on the PR on Sunday since I have a test that fails who looks like it shouldn't.

[thedrow]

I have a branch that removes support for the old setting names.
However, we have a couple of issues with implementing this, both technical and operational issues.

• While we have declared these settings deprecated in the documentation and in the release notes of 4.0, we have not prompted our users with a deprecation warning.
• The settings are set up before the logging subsystem is set up, which makes it harder to notify the user that the settings he is using are deprecated.
• Django users, a significant audience for Celery, still use the all-caps settings. Should we force them to upgrade? Should we support uppercase settings too?

I'm not so sure anymore we should do this in 5.0.
Instead we should notify the user that he is using deprecated setting names.

What do you think?

[auvipy]

I have a branch that removes support for the old setting names.
However, we have a couple of issues with implementing this, both technical and operational issues.

* While we have declared these settings deprecated in the documentation and in the release notes of 4.0, we have not prompted our users with a deprecation warning.

* The settings are set up before the logging subsystem is set up, which makes it harder to notify the user that the settings he is using are deprecated.

* Django users, a significant audience for Celery, still use the all-caps settings. Should we force them to upgrade? Should we support uppercase settings too?

I'm not so sure anymore we should do this in 5.0.
Instead we should notify the user that he is using deprecated setting names.

What do you think?

we should not remove them in v5 for easier migration

[clokep]

Django users, a significant audience for Celery, still use the all-caps settings. Should we force them to upgrade? Should we support uppercase settings too?

I don't believe accepting the all-caps version was deprecated? The documentation around Django seems to say that the all-caps version (and prefixing them) is still allowed.

[thedrow]

Here's what I think (and what I implemented):
We should produce a warning and instruct users to run celery update settings.
In 6.0 we'll completely remove support.