tickfails on deserialized Workflow

[abak]

If i start a workflow, serialize it to json, and then recreate my original Workflow, the tick function will raise an exception.

This is the stack trace :

 File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/workflows/workflow.py", line 411, in tick
    running = self._tick()
  File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/workflows/workflow.py", line 447, in _tick
    already_scheduled = self.are_scheduled(sigs_to_run)
  File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/workflows/workflow.py", line 385, in are_scheduled
    states = self.get_tasks_state([n.id for n in nodes_to_run])
  File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/workflows/workflow.py", line 271, in get_tasks_state
    for task_id in task_ids
  File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/workflows/workflow.py", line 271, in <dictcomp>
    for task_id in task_ids
  File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/celery/inspect.py", line 34, in get_task_state
    return ResultState(entities.AsyncResult(task_id))
  File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/celery/inspect.py", line 50, in __init__
    self.__inspect()
  File "/usr/local/lib/python3.7/site-packages/celery_dyrygent/celery/inspect.py", line 53, in __inspect
    self.success = self.result.successful()
  File "/usr/local/lib/python3.7/site-packages/celery/result.py", line 319, in successful
    return self.state == states.SUCCESS
  File "/usr/local/lib/python3.7/site-packages/celery/result.py", line 475, in state
    return self._get_task_meta()['status']
  File "/usr/local/lib/python3.7/site-packages/celery/result.py", line 414, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "/usr/local/lib/python3.7/site-packages/celery/backends/base.py", line 451, in get_task_meta
    meta = self._get_task_meta_for(task_id)
AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'

The root cause is that inspect.get_task_state will try to instanciate AsyncResult without specifying their result backend.

I do not know how that works normally. I will keep on looking in my application code, to make sure that the issue is not on my side.

[brabiega]

Celery result backend has to be enabled for the whole machinery to work.
It's configured in main celery configuration file.
https://docs.celeryproject.org/en/stable/userguide/configuration.html#task-result-backend-settings

In our prod environment we're using mysql.

At first glance it looks like you don't have result backend configured for your celery application.

[abak]

Oh no, i have a Redis backend configured and running.

The thing is, if i want to get a status from an id, i need to do something like:

celery.result.AsyncResult(uuid, /*my backend connection srting here*/)

As per the code i saw in inspect.py, there is no backend specified to the AsyncResult constructor, so i guess that can be the issue i'm running into.

[brabiega]

Hmm that's interesting.
As far as I understand when no backend is specified for AsyncResult, the default one is used (at least that's what seems to be happening for my environment).

Can I have a piece of code to reproduce this issue?

[brabiega]

Does your celery worker which handles workflow-processor have results backend enabled? You can see that when celery worker starts. E.g.

(venv3) root@dyrygent1:~/repos/celery-dyrygent/tests/integration# celery worker -l INFO
/root/repos/celery-dyrygent/venv3/lib/python3.6/site-packages/celery-4.4.2-py3.6.egg/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid,
 
 -------------- celery@dyrygent1 v4.4.2 (cliffs)
--- ***** ----- 
-- ******* ---- Linux-5.6.5-arch3-1-x86_64-with-Ubuntu-18.04-bionic 2020-04-25 10:00:51
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         default:0x7f166efa3828 (.default.Loader)
- ** ---------- .> transport:   amqp://celery:**@localhost:5672/celery
- ** ---------- .> results:     mysql+pymysql://celery:**@localhost/celery
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

[tasks]
  . testtasks.hello
  . workflow-processor

[2020-04-25 10:00:51,590: INFO/MainProcess] Connected to amqp://celery:**@127.0.0.1:5672/celery
[2020-04-25 10:00:51,597: INFO/MainProcess] mingle: searching for neighbors
[2020-04-25 10:00:52,639: INFO/MainProcess] mingle: all alone
[2020-04-25 10:00:52,648: INFO/MainProcess] celery@dyrygent1 ready.

[abak]

I'm trying to isolate a piece of code, and that is non trivial. My app that handle the workflow processor is a flask application, and it is not started as a celery worker.

When creating my flask app, i import a file that starts like that :

celery = Celery(
    "name"
)

celery.config_from_object('name.defaults.CeleryConfig')
register_workflow_processor(celery)

And the CeleryConfig object is as follows

class CeleryConfig:
    task_track_started = True
    broker_url = 'amqp://celery:password@rabbit:5672/celery'
    result_backend = 'redis://redis:6379/0'

BTW I managed to achieve what i was trying to do by using signals, instead of serialisation/deserialisation of the workflow object, so i don't have a probleme anymore.

[brabiega]

I'm closing this ticket as this is no longer an issue for the author and I'm not able to reproduce the problem.