[FilesystemBackend] Failure in unpickling a FilesystemBackend instance for a group

[singlaive]

Checklist

• [ x] I have verified that the issue exists against the master branch of Celery.
• This has already been asked to the discussion group first.
• [x ] I have read the relevant section in the
  contribution guide
  on reporting bugs.
• [x ] I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.4.2 (cliffs) kombu:4.6.8 py:3.7.5
            billiard:3.6.3.0 py-amqp:2.5.2
platform -> system:Darwin arch:64bit
            kernel version:18.7.0 imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Backend: file system
steriliser: pickle

Scenario: Define a group and retrieve the result, for instance, group([add.s(1,2), add.s(3,4)])().get()
This should give the correct result of two adding calculation

Actual Behavior

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3/lib/python3.7/site-packages/celery/result.py:229: in get
    on_message=on_message,
venv3/lib/python3.7/site-packages/celery/backends/base.py:579: in wait_for_pending
    no_ack=no_ack,
venv3/lib/python3.7/site-packages/celery/backends/base.py:606: in wait_for
    meta = self.get_task_meta(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:457: in get_task_meta
    meta = self._get_task_meta_for(task_id)
venv3/lib/python3.7/site-packages/celery/backends/base.py:791: in _get_task_meta_for
    return self.decode_result(meta)
venv3/lib/python3.7/site-packages/celery/backends/base.py:322: in decode_result
    return self.meta_from_decoded(self.decode(payload))
venv3/lib/python3.7/site-packages/celery/backends/base.py:333: in decode
    accept=self.accept)
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
    return decode(data)
/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py:130: in __exit__
    self.gen.throw(type, value, traceback)
venv3/lib/python3.7/site-packages/kombu/serialization.py:54: in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
venv3/lib/python3.7/site-packages/vine/five.py:194: in reraise
    raise value.with_traceback(tb)
venv3/lib/python3.7/site-packages/kombu/serialization.py:50: in _reraise_errors
    yield
venv3/lib/python3.7/site-packages/kombu/serialization.py:264: in loads
    return decode(data)
venv3/lib/python3.7/site-packages/kombu/serialization.py:338: in unpickle
    return pickle_loads(str_to_bytes(s))
venv3/lib/python3.7/site-packages/kombu/serialization.py:59: in pickle_loads
    return load(BytesIO(s))
venv3/lib/python3.7/site-packages/celery/backends/base.py:69: in unpickle_backend
    return cls(*args, app=current_app._get_current_object(), **kwargs)
venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:53: in __init__
    path = self._find_path(url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <celery.backends.filesystem.FilesystemBackend object at 0x10b2c7c90>, url = None

    def _find_path(self, url):
        if not url:
>           raise ImproperlyConfigured(E_NO_PATH_SET)
E           kombu.exceptions.DecodeError: You need to configure a path for the file-system backend

venv3/lib/python3.7/site-packages/celery/backends/filesystem.py:77: DecodeError

Solution

FilesystemBackend class defined in https://github.com/celery/celery/blob/master/celery/backends/filesystem.py#L34, misses __reduce__ method to pickle the instance properly. The following example would make it work.

def __reduce__(self, args=(), kwargs={}):
    kwargs.update(
        dict(url=self.url))
    return super(FilesystemBackend, self).__reduce__(args, kwargs)

[auvipy]

PR welcome with unit test