Add an option to load configuration from a file, rather than an importable module.

[dougthor42]

Checklist

• I have checked the issues list for similar or identical feature requests.
  • I searched for the following terms and found nothing: "config from_pyfile", "config_from_pyfile", "flask-like config loading from file not module", "config from pyfile not module"
• I have checked the commit log to find out if a feature was already implemented in master.
  • OK, I lied: I didn't check the git history, but I did check celery/app/base.py and celery/loaders and I didn't see any additional config_from_ methods.

Brief Summary

I would like to load configuration from a python-like file rather than a module. Similar to flask's from_pyfile.

The key here is that the config file is not on python's path, nor is it part of any package.

Why do I want this? Well primarily it stems from the fact that I've gotten used to doing it this way in Flask and I want to put the Celery config in the same file as the Flask config...

Design

Architectural Considerations

None that I can tell.

Proposed Behavior

This would simply be an addition to the current "config_from_X" methods. The current methods require a Python module, which I find to be somewhat restrictive.

Easiest explained with an example:

# ./config/localhost.cfg
# Flask stuff
DEBUG = True
TESTING = False

# Celery stuff
broker_url = "redis://redis"

Usage:

from celery import Celery
celery = Celery("foo")
celery.config_from_object("project.default_config")
celery.config_from_pyfile("./config/localhost.cfg")

Note: It's yet to be determined if lazy loading would be possible with this option.

Proposed UI/UX

New API:

celery.Celery.config_from_pyfile

Updated API:

• celery.Celery.config_from_envvar also accepts non-module values, eg: "/var/www/app/config.cfg"

Diagrams

N/A

Alternatives

My current workaround is below. The only disadvantage that can see to this workaround is that there is no option for lazy loading. I don't know how to implement that, nor do I have the need for it.

def config_from_envvar(celery_app, var_name, silent=False):
    """
    Load celery config from an envvar that points to a python config file.

    Basically this:

        config_from_pyfile(os.environ['YOUR_APP_SETTINGS'])

    Example:
        >>> os.environ['CELERY_CONFIG_FILE'] = './some_dir/config_file.cfg'
        >>> config_from_envvar(celery, 'CELERY_CONFIG_FILE')

    Arguments:
        celery_app (Celery app instance): The celery app to update
        var_name (str): The env var to use.
        silent (bool): If true then import errors will be ignored.

    Shamelessly taken from Flask. Like, almost exactly. Thanks!
    https://github.com/pallets/flask/blob/74691fbe0192de1134c93e9821d5f8ef65405670/flask/config.py#L88
    """
    rv = os.environ.get(var_name)
    if not rv:
        if silent:
            return False
        raise RuntimeError('The environment variable %r is not set'
                           ' and as such configuration could not be'
                           ' loaded. Set this variable to make it'
                           ' point to a configuration file.' % var_name)
    return config_from_pyfile(celery_app, rv, silent=silent)


def config_from_pyfile(celery_app, filename, silent=False):
    """
    Mimics Flask's config.from_pyfile()

    Allows loading a separate, perhaps non `.py`, file into Celery.

    Example:
        >>> config_from_pyfile(celery, './some_dir/config_file.cfg')

    Arguments:
        celery_app (Celery app instance): The celery app to update
        filename (str): The file to load.
        silent (bool): If true then import errors will be ignored.

    Also shamelessly taken from Flask:
    https://github.com/pallets/flask/blob/74691fbe0192de1134c93e9821d5f8ef65405670/flask/config.py#L111
    """
    filename = str(Path(filename).resolve())
    d = types.ModuleType('config')
    d.__file__ = filename

    try:
        with open(filename, 'rb') as config_file:
            exec(compile(config_file.read(), filename, 'exec'), d.__dict__)
    except IOError as e:
        if silent and e.errno in(errno.ENOENT, errno.EISDIR, errno.ENOTDIR):
            return False
        e.strerror = "Unable to load config file (%s)" % e.strerror
        raise

    # Remove any "hidden" attributes: __ and _
    for k in list(d.__dict__.keys()):
        if k.startswith("_"):
            del d.__dict__[k]

    celery_app.conf.update(d.__dict__)
    return True

# Example of using the above
CFG_VAR = "MYAPP_CONFIG_FILE"
os.environ[CFG_VAR] = "./config/localhost.cfg"
celery = Celery(__name__, broker='redis://localhost')
# normal celery config loading:
celery.config_from_object('trendlines.default_config')
# followed by my workaround. Normally I'd wrap this in try..except but I'm keeping it simple here.
config_from_envvar(celery, CFG_VAR)

[auvipy]

Let's consider this for celery v5

[yashvardhannanavati]

Can I please ask if there's a plan to implement this anytime soon? Would you consider if I make a PR for this?

[auvipy]

Can I please ask if there's a plan to implement this anytime soon? Would you consider if I make a PR for this?

of course we will consider your PR :D