Chain of groups causing duplicate tasks #8180

deklanw · 2023-04-04T14:52:04Z

Checklist

I have verified that the issue exists against the main branch of Celery.
This has already been asked to the discussions forum first.
I have read the relevant section in the
contribution guide
on reporting bugs.
I have checked the issues list
for similar or identical bug reports.
I have checked the pull requests list
for existing proposed fixes.
I have checked the commit log
to find out if the bug was already fixed in the main branch.
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
I have verified that the issue exists against the main branch of Celery.
I have included the contents of pip freeze in the issue.
I have included all the versions of all the external dependencies required
to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version
and/or implementation.
I have tried reproducing the issue on more than one message broker and/or
result backend.
I have tried reproducing the issue on more than one version of the message
broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#8062

Possible Duplicates

None

Environment & Settings

Celery version: 5.3.0b2

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown
Minimal Celery Version: N/A or Unknown
Minimal Kombu Version: N/A or Unknown
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

from celery.utils.log import get_task_logger
task_logger = get_task_logger(__name__)

import celery
assert celery.__version__ == '5.3.0b2'

@celery_app.task(bind=True)
def dummy_task(self, wn):
    task_logger.info(f'worker_num: {wn}')

@celery_app.task(bind=True)
def dummy_launch(self):
    # structure with apparently no issue
    # worker_structure = [
    #     [0, 1],
    #     [2, 3],
    #     [4],
    # ]

    # structure which causes issue. e.g., logs give 0,1,2,3,4,5,6,4,5,6
    worker_structure = [
        [0, 1],
        [2, 3],
        [4, 5],
        [6]
    ]

    # issue.
    worker_tasks = chain(group(dummy_task.si(wn) for wn in worker_nums)
        for worker_nums in worker_structure)
    
    start = [dummy_task.si('start')]
    end = [dummy_task.si('end')]

    # testing to see whether dummy tasks fix:

    # issue
    # worker_tasks = chain([group(dummy_task.si(wn) for wn in worker_nums)
    #     for worker_nums in worker_structure] + end)


    # no issue
    # worker_tasks = chain(start + [group(dummy_task.si(wn) for wn in worker_nums)
    #     for worker_nums in worker_structure])

    return self.replace(worker_tasks)

Expected Behavior

A chain of groups to only execute each group once.

Actual Behavior

A chain of groups executes some of the latter groups twice. Putting a dummy task at the front of the chain fixes (See minimal reproduction).

The text was updated successfully, but these errors were encountered:

open-collective-bot · 2023-04-04T14:52:08Z

Hey @deklanw 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

pencil · 2023-04-27T23:58:13Z

I just ran into this as well and found this bug report after hours of debugging and questioning my sanity.

@deklanw Thank you for sharing the workaround; putting an empty dummy task at the beginning really does seem to fix it!

Nusnus · 2024-03-19T13:27:06Z

Fixed in main: #7919, #8663

pencil · 2024-05-07T22:30:15Z

I am still experiencing this problem with Celery 5.4.0. I'll do more testing and report back later this week.

pencil · 2024-05-07T23:50:27Z

Here is an example that causes the duplicate execution behavior:

@app.task
def dummy_task(out: int | str) -> None:
    print(out)


def run_chained_groups() -> None:
    chain(
        chain(
            group(dummy_task.si(1), dummy_task.si(2), dummy_task.si(3)),
            group(dummy_task.si(4), dummy_task.si(5)),
        ),
        chain(group(dummy_task.si(6), dummy_task.si(7), dummy_task.si(8))),
    ).apply_async()

Output:

[2024-05-07 23:48:32,614: INFO/MainProcess] Task tasks.dummy_task[52d13df6-f22e-41b8-99ce-1f4e381e9084] received
[2024-05-07 23:48:32,615: WARNING/ForkPoolWorker-4] 1
[2024-05-07 23:48:32,615: INFO/MainProcess] Task tasks.dummy_task[25df42a8-c9b2-4d46-980c-ac224c62e4c9] received
[2024-05-07 23:48:32,616: INFO/ForkPoolWorker-4] Task tasks.dummy_task[52d13df6-f22e-41b8-99ce-1f4e381e9084] succeeded in 0.0009677530033513904s: None
[2024-05-07 23:48:32,616: WARNING/ForkPoolWorker-2] 2
[2024-05-07 23:48:32,617: INFO/MainProcess] Task tasks.dummy_task[0ea5ce27-79a1-4761-9eb2-62bd1411f026] received
[2024-05-07 23:48:32,617: INFO/ForkPoolWorker-2] Task tasks.dummy_task[25df42a8-c9b2-4d46-980c-ac224c62e4c9] succeeded in 0.001365919946692884s: None
[2024-05-07 23:48:32,618: WARNING/ForkPoolWorker-4] 3
[2024-05-07 23:48:32,621: INFO/MainProcess] Task tasks.dummy_task[c558d3f8-4aad-4f38-800e-d43dcacd41b3] received
[2024-05-07 23:48:32,621: INFO/ForkPoolWorker-4] Task tasks.dummy_task[0ea5ce27-79a1-4761-9eb2-62bd1411f026] succeeded in 0.003093007020652294s: None
[2024-05-07 23:48:32,622: WARNING/ForkPoolWorker-2] 4
[2024-05-07 23:48:32,622: INFO/MainProcess] Task tasks.dummy_task[47208488-9b2c-4689-86e7-a6f1c41a54aa] received
[2024-05-07 23:48:32,623: WARNING/ForkPoolWorker-4] 5
[2024-05-07 23:48:32,624: INFO/MainProcess] Task tasks.dummy_task[9ce584c6-9c7a-4b3c-aa7b-dc82a89e362b] received
[2024-05-07 23:48:32,625: INFO/ForkPoolWorker-2] Task tasks.dummy_task[c558d3f8-4aad-4f38-800e-d43dcacd41b3] succeeded in 0.0027152569964528084s: None
[2024-05-07 23:48:32,625: INFO/MainProcess] Task tasks.dummy_task[ef8ded62-5618-4ae7-87a8-9263e33ed6b3] received
[2024-05-07 23:48:32,626: WARNING/ForkPoolWorker-3] 6
[2024-05-07 23:48:32,626: INFO/MainProcess] Task tasks.dummy_task[702b2e23-f006-49da-80cd-e92ab340e6e4] received
[2024-05-07 23:48:32,626: WARNING/ForkPoolWorker-1] 7
[2024-05-07 23:48:32,627: INFO/ForkPoolWorker-3] Task tasks.dummy_task[9ce584c6-9c7a-4b3c-aa7b-dc82a89e362b] succeeded in 0.001047502039000392s: None
[2024-05-07 23:48:32,627: INFO/ForkPoolWorker-4] Task tasks.dummy_task[47208488-9b2c-4689-86e7-a6f1c41a54aa] succeeded in 0.00398255093023181s: None
[2024-05-07 23:48:32,628: WARNING/ForkPoolWorker-2] 8
[2024-05-07 23:48:32,628: INFO/MainProcess] Task tasks.dummy_task[9ce584c6-9c7a-4b3c-aa7b-dc82a89e362b] received
[2024-05-07 23:48:32,628: INFO/ForkPoolWorker-2] Task tasks.dummy_task[702b2e23-f006-49da-80cd-e92ab340e6e4] succeeded in 0.000443708966486156s: None
[2024-05-07 23:48:32,628: INFO/MainProcess] Task tasks.dummy_task[ef8ded62-5618-4ae7-87a8-9263e33ed6b3] received
[2024-05-07 23:48:32,629: WARNING/ForkPoolWorker-4] 6
[2024-05-07 23:48:32,629: WARNING/ForkPoolWorker-3] 7
[2024-05-07 23:48:32,629: INFO/ForkPoolWorker-1] Task tasks.dummy_task[ef8ded62-5618-4ae7-87a8-9263e33ed6b3] succeeded in 0.003555716946721077s: None
[2024-05-07 23:48:32,630: INFO/ForkPoolWorker-4] Task tasks.dummy_task[9ce584c6-9c7a-4b3c-aa7b-dc82a89e362b] succeeded in 0.0006144179496914148s: None
[2024-05-07 23:48:32,630: INFO/ForkPoolWorker-3] Task tasks.dummy_task[ef8ded62-5618-4ae7-87a8-9263e33ed6b3] succeeded in 0.0006913350662216544s: None
[2024-05-07 23:48:32,630: INFO/MainProcess] Task tasks.dummy_task[702b2e23-f006-49da-80cd-e92ab340e6e4] received
[2024-05-07 23:48:32,631: WARNING/ForkPoolWorker-4] 8
[2024-05-07 23:48:32,631: INFO/ForkPoolWorker-4] Task tasks.dummy_task[702b2e23-f006-49da-80cd-e92ab340e6e4] succeeded in 0.00029250001534819603s: None

As you can see, tasks 6, 7 and 8 run twice.

@Nusnus should I file a separate bug for this one?

Nusnus · 2024-05-09T10:36:17Z

@Nusnus should I file a separate bug for this one?

No, I’ll reopen this one.

Thank you for reporting this! I’ve been working hard on crushing canvas bugs in the last year so I’ll try to prioritize it high.

Nusnus · 2024-06-25T13:01:49Z

Fixed in #9021

deklanw added the Issue Type: Bug Report label Apr 4, 2023

Nusnus closed this as completed Mar 19, 2024

pencil mentioned this issue May 8, 2024

Canvas integration test for chained groups that currently fails #9013

Closed

Nusnus reopened this May 9, 2024

Nusnus closed this as completed Jun 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chain of groups causing duplicate tasks #8180

Chain of groups causing duplicate tasks #8180

deklanw commented Apr 4, 2023

open-collective-bot bot commented Apr 4, 2023

pencil commented Apr 27, 2023 •

edited

Loading

Nusnus commented Mar 19, 2024 •

edited

Loading

pencil commented May 7, 2024

pencil commented May 7, 2024 •

edited

Loading

Nusnus commented May 9, 2024

Nusnus commented Jun 25, 2024 •

edited

Loading

Chain of groups causing duplicate tasks #8180

Chain of groups causing duplicate tasks #8180

Comments

deklanw commented Apr 4, 2023

Checklist

Mandatory Debugging Information

Optional Debugging Information

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Steps to Reproduce

Required Dependencies

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

open-collective-bot bot commented Apr 4, 2023

pencil commented Apr 27, 2023 • edited Loading

Nusnus commented Mar 19, 2024 • edited Loading

pencil commented May 7, 2024

pencil commented May 7, 2024 • edited Loading

Nusnus commented May 9, 2024

Nusnus commented Jun 25, 2024 • edited Loading

pencil commented Apr 27, 2023 •

edited

Loading

Nusnus commented Mar 19, 2024 •

edited

Loading

pencil commented May 7, 2024 •

edited

Loading

Nusnus commented Jun 25, 2024 •

edited

Loading