More flexible configuration options like INI or YAML

[tvoinarovskyi]

Celery REALLY expects you to have the configuration in python in one or another way. The problem is that in deployment it is sometimes reasonable to locate the configuration in a quite specific location (not relative to the project. Somewhere like /etc/celery.conf).

I really tried implementing some extension that would operate on the --config option to allow custom configurators, but I failed, cause did not find a more or less reliable hook for this. Using the read_configuration hook does not let you specify the file locating from command line, and I think that's the right way to do it. (We could use an env variable, but personally I don't like this way.)

My vision of a solution is to provide a new option (or enhance --config) which would be passed to some DOCUMENTED hook in app or loader(there is something already there like cmdline_config_parser, but I really could not understand when those are called.). Plus it would be great if we could pass some custom configuration readers right into the Celery instance.
Something like:

./bin/celery --app myapp --configfile /etc/celery.yaml

celery = Celery('myapp', configurators={'yaml':YAMLConfigReader}) 

[aaronharnly]

+1 for this. We have written custom code to read Celery config from an .ini file, but it requires a fair amount of special handling, since the celery.conf.update() method expects many settings to be native booleans or ints.

Separating configuration from code is a very common operational requirement – for example to point to different BROKER_URLs in different deployment environments (dev, CI, staging, prod, etc.).

[igudym]

There is new API in development Celery version which adds user options. Look at https://gist.github.com/4295009 - how to load INI config specified in command line.

[tvoinarovskyi]

That is quite the thing I want. I will look more when I have a few free minutes and if that completely solves my tasks I will close the ticket.

[ask]

The .INI format is not automatically suitable for Celery configuration since it usually expects values to be in the proper type.

Luckily the celery.app.defaults module provides introspection for all of the configuration keys used, and you should be able to use that to coerce the values into the correct types

https://github.com/celery/celery/blob/master/celery/app/defaults.py#L45

The cmdline_config_parser is there to support quick reconfiguration added at the end of the command line (after a -- separator), examples include:

celery worker -- celery.prefetch_multiplier=10
celery worker -l info -- celery.prefetch_multiplier=(int)10
celery worker -l info -- broker.transport_options='(json){"visibility_timeout": 10}'

It's not used much in the documentation because it's not exactly user-friendly, but very useful when you want to try something quickly

[ask]

I think the tools to support other configuration formats are present. If there is anything specific that can be improved then please open up a new issue describing that :)

[deviantony]

Is there any documentation on this subject? I would also like to configure Celery with a configuration file outside of the project.

Seems that it needs a good amount of custom code to handle this atm.

[malinoff]

@deviantony you can load from whatever format you want - just make sure you have a dictionary in the end which you can pass to the app instance.

[deviantony]

@malinoff thanks !
Any existing documentation/example on this?

[malinoff]

@deviantony you should search through the docs of your format parser. Celery itself cannot document all possible formats. But generally, all you need to do is

import someparser
with open('path/to/config') as f:
    config = someparser.load(f)

[deviantony]

@malinoff I was not talking about the parser but about the expected dictionary structure for example.

I've managed to create a solution based on @igudym example.

For those interested (configuration loaded from a YML file): https://gist.github.com/deviantony/2dabeb7c2eb0ae6db32a

[gusutabopb]

+1 for this.

I really think the --config flag should be able to take more than just plain Python files. Having a Python-module-with-a-bunch-of-global-variables as the canonical way to pass configuration doesn't feel quite like good practice. The API suggested by @tvoinarovskyi seems pretty neat IMHO.

I am new to Celery but it seems the YAML solution provided by @deviantony is a bit outdated. On version 4.2 (and Python 3.7), this is the solution I came up with (based on this section of the docs):

import logging
from pathlib import Path

import ruamel.yaml as yaml
from celery import Celery, bootsteps


def load_yaml_config(parser):
    parser.add_argument('--config_path', action='store', help='Read YAML config')


class YamlConfigLoader(bootsteps.Step):
    def __init__(self, worker, config_path, **options):
        config_path = Path(config_path or '').expanduser()
        if config_path.is_file():
            with config_path.open() as f:
                worker.app.conf.update(yaml.safe_load(f))

app = Celery('project')
# Optionally add default config values here
# app.conf.foo = 'bar'
app.user_options['worker'].add(load_yaml_config)
app.steps['worker'].add(YamlConfigLoader)