Bugfix: Worker not consuming tasks after Redis broker restart #8796

Nusnus · 2024-01-16T22:06:05Z

TL;DR

Between Celery v5.3.0b1 -> v5.3.0b2, a commit was pushed that made a change that caused the infamous Redis bug. This PR reverts the change and adds automatic smoke tests that verify the bug is fixed. This test fails 100% (consistently) before reverting the change and passes 100% (consistently) after reverting the change.

Potentially fixes: #7276, #8091, #8030, #8384
Reverted change: #7925

Long Story

So, from the community perspective, “somewhere” after v5.2.x Redis reconnection caused the celery worker to stop consuming messages until the worker itself was restarted. This caused a huge headache, and as we lacked the required effort to fix the issue, it remained unhandled so far.
There were quite a few discussions about this bug, many offering attempts to diagnose the issue, others offering their setup where it happened, and possible workarounds. Unfortunately, I lacked the experience to fully understand all of the proposals, etc., but the information was still very useful, so I took a different approach to solving this problem.

As the main maintainer of the upcoming pytest-celery plugin release, I’ve taken “my baby” for a ride and started looking for the breaking commit, assuming somewhere along the git history, there should be a location where commit X has the bug and X-1 does not.

So I wrote a test that reproduces the bug for every celery version from v5.1.0-Latest

from __future__ import annotations

from time import sleep
from typing import Any

import pytest
import requests
from packaging.version import parse as parse_version
from pytest_celery import CeleryBrokerCluster, CeleryTestSetup, RedisTestBroker, default_worker_container  # noqa

from t.integration.tasks import identity
from t.smoke.workers.dev import SmokeWorkerContainer


class MyWorkerContainer(SmokeWorkerContainer):
    @classmethod
    def worker_queue(cls) -> str:
        return "celery"


@pytest.fixture(scope="session")
def default_worker_container_session_cls() -> type[SmokeWorkerContainer]:
    return MyWorkerContainer


def get_celery_versions(start_version, end_version):
    url = "https://pypi.org/pypi/celery/json"
    response = requests.get(url)
    data = response.json()
    all_versions = data["releases"].keys()

    filtered_versions = [
        v
        for v in all_versions
        if (
            parse_version(start_version)
            <= parse_version(v)
            <= parse_version(end_version)
            and not parse_version(v).is_prerelease
        )
    ]

    return sorted(filtered_versions, key=parse_version)


@pytest.fixture(scope="session", params=get_celery_versions("v5.1.0", "v5.3.6"))
def celery_version(request):
    return request.param


@pytest.fixture(scope="session")
def default_worker_celery_version(celery_version: str) -> str:
    return celery_version


@pytest.fixture
def celery_broker_cluster(celery_redis_broker: RedisTestBroker) -> CeleryBrokerCluster:
    cluster = CeleryBrokerCluster(celery_redis_broker)
    yield cluster
    cluster.teardown()


class test_which_celery_fails:
    def test_worker_consume_tasks_after_redis_broker_restart(
        self,
        celery_setup: CeleryTestSetup,
        celery_version: str,
        subtests: Any,
    ):
        with subtests.test(msg=f"Using Celery v{celery_version}"):
            msg = "Before broker restart"
            res = identity.s(msg).apply_async()
            assert res.get(timeout=10) == msg
            celery_setup.broker.kill()
            sleep(3)
            celery_setup.broker.restart()
            msg = "After broker restart"
            res = identity.s(msg).apply_async()
            assert res.get(timeout=10) == msg

Running it with pytest-xdist on a 10-CPU machine produced a very nice parallel run:

pytest -vvvv t/smoke -k test_which_celery_fails -n auto
======================================================================= test session starts =======================================================================
platform darwin -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0 -- /Users/nusnus/.pyenv/versions/3.12.1/envs/celery_py312/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/nusnus/dev/GitHub/celery
configfile: pyproject.toml
plugins: docker-tools-3.1.3, timeout-2.2.0, order-1.2.0, click-1.1.0, subtests-0.11.0, rerunfailures-13.0, celery-1.0.0b1, xdist-3.5.0
10 workers [16 items]
scheduling tests via LoadScheduling

t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.2.1-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.2.2-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.2.3-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.1.2-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.2.0-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.1.1-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.2.6-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.2.5-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.1.0-celery_setup_worker-celery_redis_backend]
t/smoke/tests/tomer/test_which_celery_fails.py::test_which_celery_fails::test_worker_consume_tasks_after_redis_broker_restart[5.2.4-celery_setup_worker-celery_redis_backend]
...

This allowed me to prove the bug exists and where it starts, which is correlated with the reports in the relevant issues.
So for the next step, I’ve modified the test to search on a deeper level, reproducing the bug on every commit between the passing version (v5.2.7) and the failing version (5.3.0).

FROM python:3.10-bookworm

# Create a user to run the worker
RUN adduser --disabled-password --gecos "" test_user

# Install system dependencies
RUN apt-get update && apt-get install -y build-essential git

# Set arguments
ARG CELERY_VERSION="HEAD"
ARG CELERY_LOG_LEVEL=INFO
ARG CELERY_WORKER_NAME=my_worker
ARG CELERY_WORKER_QUEUE=celery
ENV LOG_LEVEL=$CELERY_LOG_LEVEL
ENV WORKER_NAME=$CELERY_WORKER_NAME
ENV WORKER_QUEUE=$CELERY_WORKER_QUEUE

# Install packages
WORKDIR /src

RUN pip install --no-cache-dir --upgrade pip \
    && pip install pytest-celery==1.0.0b1
RUN git clone https://github.com/celery/celery.git

WORKDIR /src/celery

RUN git reset --hard $CELERY_VERSION
RUN pip install -e .

# The workdir must be /app
WORKDIR /app

# Switch to the test_user
USER test_user

# Start the celery worker
CMD celery -A app worker --loglevel=$LOG_LEVEL -n $WORKER_NAME@%h -Q $WORKER_QUEUE

from __future__ import annotations

import subprocess
from time import sleep
from typing import Any

import pytest
from pytest_celery import (CeleryBrokerCluster, CeleryTestSetup, CeleryTestWorker, CeleryWorkerCluster,
                           CeleryWorkerContainer, RedisTestBroker, defaults)
from pytest_docker_tools import build, container, fxtr

from celery import Celery
from t.integration.tasks import identity


def get_commit_hashes(start_tag: str, end_tag: str) -> list[str]:
    cmd = ["git", "rev-list", f"{start_tag}..{end_tag}"]
    result = subprocess.run(cmd, capture_output=True, text=True, check=True)
    return result.stdout.strip().split("\n")


# Requires running with pytest-xdist to trigger multiple docker build and avoid cached build for all params
@pytest.fixture(scope="session", params=get_commit_hashes("v5.2.7", "v5.3.0"))
# @pytest.fixture(
#     scope="session",
#     params=[
#         "029262fabe50dc941626bfa2a1727f6619e4c31f",
#         "24ac092e07403191002d700a8fad1bf5aa498138",
#         "0233c3b674dcfc6fff79f4161ca9a818dabf28e7"  # BUG!
#     ],
# )
def celery_commit(request):
    return request.param


class DetachedHeadWorkerContainer(CeleryWorkerContainer):
    @classmethod
    def version(cls) -> str:
        return fxtr("celery_commit")


celery_detached_head_worker_image = build(
    path=".",
    dockerfile="t/smoke/tests/tomer/Dockerfile",
    buildargs=DetachedHeadWorkerContainer.buildargs(),
)


celery_detached_head_worker_container = container(
    image="{celery_detached_head_worker_image.id}",
    environment=fxtr("default_worker_env"),
    network="{default_pytest_celery_network.name}",
    volumes={"{default_worker_volume.name}": defaults.DEFAULT_WORKER_VOLUME},
    wrapper_class=DetachedHeadWorkerContainer,
    timeout=defaults.DEFAULT_WORKER_CONTAINER_TIMEOUT,
)


@pytest.fixture
def celery_detached_head_worker(
    celery_detached_head_worker_container: DetachedHeadWorkerContainer,
    celery_setup_app: Celery,
) -> CeleryTestWorker:
    worker = CeleryTestWorker(
        celery_detached_head_worker_container, app=celery_setup_app
    )
    yield worker
    worker.teardown()


@pytest.fixture
def celery_worker_cluster(
    celery_detached_head_worker: CeleryTestWorker,
) -> CeleryWorkerCluster:
    cluster = CeleryWorkerCluster(celery_detached_head_worker)
    yield cluster
    cluster.teardown()


@pytest.fixture
def celery_broker_cluster(celery_redis_broker: RedisTestBroker) -> CeleryBrokerCluster:
    cluster = CeleryBrokerCluster(celery_redis_broker)
    yield cluster
    cluster.teardown()


class test_which_commit_fail:
    def test_worker_consume_tasks_after_redis_broker_restart(
        self,
        celery_setup: CeleryTestSetup,
        celery_commit: str,
        subtests: Any,
    ):
        with subtests.test(msg=f"Using Celery commit: {celery_commit}"):
            msg = "Before broker restart"
            res = identity.s(msg).apply_async()
            assert res.get(timeout=10) == msg
            celery_setup.broker.kill()
            sleep(3)
            celery_setup.broker.restart()
            msg = "After broker restart"
            res = identity.s(msg).apply_async()
            assert res.get(timeout=10) == msg

pytest -vvvv t/smoke -k test_which_commit_fail -n auto
======================================================================= test session starts =======================================================================
platform darwin -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0 -- /Users/nusnus/.pyenv/versions/3.12.1/envs/celery_py312/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/nusnus/dev/GitHub/celery
configfile: pyproject.toml
plugins: docker-tools-3.1.3, timeout-2.2.0, order-1.2.0, click-1.1.0, subtests-0.11.0, rerunfailures-13.0, celery-1.0.0b1, xdist-3.5.0
10 workers [410 items]
scheduling tests via LoadScheduling

t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[50d4c0b07a8aa5f079b7e3fdc5e765b77ea391fa-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[fe891a6c79f4eb476e6d588a39fb686f05f85ac5-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[631ad8e1358b79c88513c49229e757e5a624618c-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[1baca0ca90b5bd7f38e3d2ae2d513f24cc0613ea-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[0e92577eb1b5358c3bd4ebc7a5e880508481f981-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[525f90e4dafd153e7cd8cc0fb921c24a7f45eca0-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[c6b54074514b14b5b7b3d8e6a4885fc1699a3e39-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[ed71ebb2addd0579e2f64e15e9d44ec7a1e31434-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[5d3ec7c7c8420f6c07d7e280b486647a373ba208-celery_redis_backend]
t/smoke/tests/tomer/test_which_commit_fail.py::test_which_commit_fail::test_worker_consume_tasks_after_redis_broker_restart[a80da3965fefcf9c7638c0a264314cd194a71d1f-celery_redis_backend]
...

Of course, it took some playing around and debugging because pytest-xdist does not sort the runs according to the git history, but it was good enough to pinpoint the bugged commit.

I verified it again with a more specific range:

@pytest.fixture(
    scope="session",
    params=[
        "029262fabe50dc941626bfa2a1727f6619e4c31f",
        "24ac092e07403191002d700a8fad1bf5aa498138",
        "0233c3b674dcfc6fff79f4161ca9a818dabf28e7"  # BUG!
    ],
)

Git History

* 0233c3b67 - Add annotations to minimise differences with celery-aio-pool's tracer.py. (#7925) (1 year ago) <ShaheedHaque>
* 24ac092e0 - Scheduled weekly dependency update for week 01 (#7987) (1 year ago) <pyup.io bot>
* 029262fab - [pre-commit.ci] pre-commit autoupdate (1 year, 1 month ago) <pre-commit-ci[bot]>

Unfortunately, I still cannot explain 100% what happened, but due to the urgency of giving an answer to the community, and due to the lack of any other proper solution, and based on a functional smoke test, I am declaring this as the current fix until proven otherwise.

codecov · 2024-01-16T22:11:01Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f2407dc) 81.18% compared to head (3fd6c07) 81.17%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8796      +/-   ##
==========================================
- Coverage   81.18%   81.17%   -0.01%     
==========================================
  Files         148      148              
  Lines       18524    18521       -3     
  Branches     3165     3165              
==========================================
- Hits        15038    15035       -3     
  Misses       3197     3197              
  Partials      289      289

Flag	Coverage Δ
unittests	`81.15% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Nusnus · 2024-01-17T00:36:24Z

All smoke tests pass - they fail on GitHub due to technical limitations.
I’m working on a fix in #8797

tox -e 3.12-smoke -- -n auto
ROOT: will run in automatically provisioned tox, host /Users/nusnus/.pyenv/versions/3.12.1/envs/celery_py312/bin/python3.12 is missing [requires (has)]: tox-gh-actions
ROOT: provision> .tox/.tox/bin/python -m tox -e 3.12-smoke -- -n auto
ROOT: tox-gh-actions won't override envlist because tox is not running in GitHub Actions
.pkg: _optional_hooks> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_editable> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_editable> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
3.12-smoke: install_package> python -I -m pip install --force-reinstall --no-deps /Users/nusnus/dev/GitHub/celery/.tox/.tmp/package/104/celery-5.3.6-0.editable-py3-none-any.whl
3.12-smoke: commands[0]> pytest -xsv t/smoke --dist=loadscope --reruns 10 --reruns-delay 60 --rerun-except AssertionError -n auto
======================================================================= test session starts =======================================================================
platform darwin -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0 -- /Users/nusnus/dev/GitHub/celery/.tox/3.12-smoke/bin/python
cachedir: .tox/3.12-smoke/.pytest_cache
rootdir: /Users/nusnus/dev/GitHub/celery
configfile: pyproject.toml
plugins: docker-tools-3.1.3, timeout-2.2.0, github-actions-annotate-failures-0.2.0, order-1.2.0, click-1.1.0, subtests-0.11.0, cov-4.1.0, rerunfailures-13.0, celery-1.0.0b1, xdist-3.5.0, anyio-4.2.0
10 workers [137 items]
scheduling tests via LoadScopeScheduling

t/smoke/tests/test_control.py::test_control::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_init_handler-None]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SIGKILL-WorkerLostError]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.POOL_RESTART]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_redis_broker-celery_redis_backend-1]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-1]
t/smoke/tests/failover/test_broker_failover.py::test_broker_failover::test_killing_first_broker[celery_setup_worker-celery_redis_backend]
t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_killing_first_worker[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_KILL]
t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_after_publish[celery_setup_worker-celery_redis_broker]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_redis_broker-celery_redis_backend-1]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_redis_broker-celery_redis_backend-2]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
[gw2] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-1]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-2]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_init_handler-None]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_process_init_handler-None]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SIGKILL-WorkerLostError]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SYSTEM_EXIT-WorkerLostError]
[gw6] PASSED t/smoke/tests/test_control.py::test_control::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_control.py::test_control::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_redis_broker-celery_redis_backend-2]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_redis_broker-celery_redis_backend-42]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_process_init_handler-None]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_ready_handler-None]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.POOL_RESTART]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
[gw2] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-2]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-3]
[gw7] PASSED t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_after_publish[celery_setup_worker-celery_redis_broker]
t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_after_publish[celery_setup_worker-celery_rabbitmq_broker]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_ready_handler-None]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_process_shutdown_handler-shutdown]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SYSTEM_EXIT-WorkerLostError]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DELAY_TIMEOUT-TimeLimitExceeded]
[gw2] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-3]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-4]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_redis_broker-celery_redis_backend-42]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend-1]
[gw9] PASSED t/smoke/tests/failover/test_broker_failover.py::test_broker_failover::test_killing_first_broker[celery_setup_worker-celery_redis_backend]
t/smoke/tests/failover/test_broker_failover.py::test_broker_failover::test_reconnect_to_main[celery_setup_worker-celery_redis_backend]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_process_shutdown_handler-shutdown]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_shutdown_handler-shutdown]
[gw2] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-4]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-5]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DELAY_TIMEOUT-TimeLimitExceeded]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.EXHAUST_MEMORY-WorkerLostError]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_redis_broker-worker_shutdown_handler-shutdown]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_init_handler-None]
[gw2] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_redis_broker-celery_redis_backend-5]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-1]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.EXHAUST_MEMORY-WorkerLostError]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SIGKILL-WorkerLostError]
[gw8] PASSED t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_killing_first_worker[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_KILL]
t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_killing_first_worker[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_KILL]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
[gw6] PASSED t/smoke/tests/test_control.py::test_control::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/test_control.py::test_control::test_shutdown_exit_with_zero[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
[gw6] PASSED t/smoke/tests/test_control.py::test_control::test_shutdown_exit_with_zero[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_control.py::test_control::test_shutdown_exit_with_zero[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw7] PASSED t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_after_publish[celery_setup_worker-celery_rabbitmq_broker]
t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_before_publish[celery_setup_worker-celery_redis_broker]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend-1]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend-2]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]
[gw7] PASSED t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_before_publish[celery_setup_worker-celery_redis_broker]
t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_before_publish[celery_setup_worker-celery_rabbitmq_broker]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_init_handler-None]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_process_init_handler-None]
[gw8] PASSED t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_killing_first_worker[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_KILL]
t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_reconnect_to_restarted_worker[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_KILL]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SIGKILL-WorkerLostError]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SYSTEM_EXIT-WorkerLostError]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
[gw6] PASSED t/smoke/tests/test_control.py::test_control::test_shutdown_exit_with_zero[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/test_canvas.py::test_group::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend-2]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend-42]
[gw6] PASSED t/smoke/tests/test_canvas.py::test_group::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_canvas.py::test_group::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw8] PASSED t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_reconnect_to_restarted_worker[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_KILL]
t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_reconnect_to_restarted_worker[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_KILL]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_process_init_handler-None]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_ready_handler-None]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
[gw7] PASSED t/smoke/tests/stamping/test_revoke.py::test_revoke_by_stamped_headers::test_revoke_by_stamped_headers_before_publish[celery_setup_worker-celery_rabbitmq_broker]
t/smoke/tests/test_canvas.py::test_chain::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw7] PASSED t/smoke/tests/test_canvas.py::test_chain::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_canvas.py::test_chain::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend-42]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_redis_broker-celery_redis_backend-1]
[gw9] PASSED t/smoke/tests/failover/test_broker_failover.py::test_broker_failover::test_reconnect_to_main[celery_setup_worker-celery_redis_backend]
t/smoke/tests/failover/test_broker_failover.py::test_broker_failover::test_broker_failover_ui[celery_setup_worker-celery_redis_backend]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_ready_handler-None]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_process_shutdown_handler-shutdown]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SYSTEM_EXIT-WorkerLostError]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DELAY_TIMEOUT-TimeLimitExceeded]
[gw2] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-1]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-2]
[gw8] PASSED t/smoke/tests/failover/test_worker_failover.py::test_worker_failover::test_reconnect_to_restarted_worker[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_KILL]
t/smoke/tests/test_canvas.py::test_chord::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_redis_broker-celery_redis_backend-1]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_redis_broker-celery_redis_backend-2]
[gw6] PASSED t/smoke/tests/test_canvas.py::test_group::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_cancel_tasks_on_connection_loss::test_max_prefetch_passed_on_broker_restart[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.POOL_RESTART]
[gw8] PASSED t/smoke/tests/test_canvas.py::test_chord::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_canvas.py::test_chord::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
[gw6] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_cancel_tasks_on_connection_loss::test_max_prefetch_passed_on_broker_restart[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_cancel_tasks_on_connection_loss::test_max_prefetch_passed_on_broker_restart[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_redis_broker-celery_redis_backend-2]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_redis_broker-celery_redis_backend-42]
[gw7] PASSED t/smoke/tests/test_canvas.py::test_chain::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_false::test_max_prefetch_not_passed_on_broker_restart[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_redis_broker-celery_redis_backend-42]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend-1]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
[gw7] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_false::test_max_prefetch_not_passed_on_broker_restart[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_false::test_max_prefetch_not_passed_on_broker_restart[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DELAY_TIMEOUT-TimeLimitExceeded]
t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.EXHAUST_MEMORY-WorkerLostError]
[gw8] PASSED t/smoke/tests/test_canvas.py::test_chord::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/test_signals.py::test_after_task_publish::test_sanity[celery_setup_worker-celery_redis_broker]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_process_shutdown_handler-shutdown]
t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_shutdown_handler-shutdown]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.POOL_RESTART]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
[gw8] PASSED t/smoke/tests/test_signals.py::test_after_task_publish::test_sanity[celery_setup_worker-celery_redis_broker]
t/smoke/tests/test_signals.py::test_after_task_publish::test_sanity[celery_setup_worker-celery_rabbitmq_broker]
[gw9] PASSED t/smoke/tests/failover/test_broker_failover.py::test_broker_failover::test_broker_failover_ui[celery_setup_worker-celery_redis_backend]
t/smoke/tests/test_canvas.py::test_signature::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
[gw9] PASSED t/smoke/tests/test_canvas.py::test_signature::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_canvas.py::test_signature::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend-1]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend-2]
[gw2] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-2]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-3]
[gw5] PASSED t/smoke/tests/test_signals.py::test_signals::test_sanity[celery_setup_worker-celery_rabbitmq_broker-worker_shutdown_handler-shutdown]
t/smoke/tests/test_signals.py::test_before_task_publish::test_sanity[celery_setup_worker-celery_redis_broker]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_child_process_respawn[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.EXHAUST_MEMORY-WorkerLostError]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SIGKILL-Worker exited prematurely: signal 9 (SIGKILL)-None]
[gw8] PASSED t/smoke/tests/test_signals.py::test_after_task_publish::test_sanity[celery_setup_worker-celery_rabbitmq_broker]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
[gw5] PASSED t/smoke/tests/test_signals.py::test_before_task_publish::test_sanity[celery_setup_worker-celery_redis_broker]
t/smoke/tests/test_signals.py::test_before_task_publish::test_sanity[celery_setup_worker-celery_rabbitmq_broker]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SIGKILL-Worker exited prematurely: signal 9 (SIGKILL)-None]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SYSTEM_EXIT-Worker exited prematurely: exitcode 1-None]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.SYSTEM_EXIT-Worker exited prematurely: exitcode 1-None]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DELAY_TIMEOUT-Hard time limit (2s) exceeded for t.smoke.tasks.self_termination_delay_timeout-TimeLimitExceeded(2,)]
[gw6] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_cancel_tasks_on_connection_loss::test_max_prefetch_passed_on_broker_restart[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/test_tasks.py::test_replace::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw9] PASSED t/smoke/tests/test_canvas.py::test_signature::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DELAY_TIMEOUT-Hard time limit (2s) exceeded for t.smoke.tasks.self_termination_delay_timeout-TimeLimitExceeded(2,)]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.EXHAUST_MEMORY-Worker exited prematurely: signal 9 (SIGKILL)-None]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend-2]
t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend-42]
[gw7] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_false::test_max_prefetch_not_passed_on_broker_restart[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_visitor.py::test_stamping_visitor::test_callback[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.EXHAUST_MEMORY-Worker exited prematurely: signal 9 (SIGKILL)-None]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SIGKILL-Worker exited prematurely: signal 9 (SIGKILL)-None]
[gw5] PASSED t/smoke/tests/test_signals.py::test_before_task_publish::test_sanity[celery_setup_worker-celery_rabbitmq_broker]
[gw7] PASSED t/smoke/tests/stamping/test_visitor.py::test_stamping_visitor::test_callback[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_visitor.py::test_stamping_visitor::test_callback[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw2] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-3]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-4]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
[gw3] PASSED t/smoke/tests/test_thread_safe.py::test_thread_safety::test_multithread_task_publish[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend-42]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SIGKILL-Worker exited prematurely: signal 9 (SIGKILL)-None]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SYSTEM_EXIT-Worker exited prematurely: exitcode 1-None]
[gw6] PASSED t/smoke/tests/test_tasks.py::test_replace::test_sanity[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_tasks.py::test_replace::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]
[gw7] PASSED t/smoke/tests/stamping/test_visitor.py::test_stamping_visitor::test_callback[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.SYSTEM_EXIT-Worker exited prematurely: exitcode 1-None]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DELAY_TIMEOUT-Hard time limit (2s) exceeded for t.smoke.tasks.self_termination_delay_timeout-TimeLimitExceeded(2,)]
[gw6] PASSED t/smoke/tests/test_tasks.py::test_replace::test_sanity[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw2] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-4]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-5]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DELAY_TIMEOUT-Hard time limit (2s) exceeded for t.smoke.tasks.self_termination_delay_timeout-TimeLimitExceeded(2,)]
t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.EXHAUST_MEMORY-Worker exited prematurely: signal 9 (SIGKILL)-None]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
[gw1] PASSED t/smoke/tests/test_tasks.py::test_task_termination::test_terminated_task_logs_correct_error[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.EXHAUST_MEMORY-Worker exited prematurely: signal 9 (SIGKILL)-None]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_during_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.POOL_RESTART]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
[gw2] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_reducing_prefetch_count[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-5]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_prefetch_count_restored[celery_setup_worker-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
[gw2] XFAIL t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_prefetch_count_restored[celery_setup_worker-celery_redis_broker-celery_redis_backend]
t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_prefetch_count_restored[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.POOL_RESTART]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
[gw2] PASSED t/smoke/tests/test_consumer.py::test_worker_enable_prefetch_count_reduction_true::test_prefetch_count_restored[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_sanity_worker_hop[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_multiple_stamps_multiple_workers[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster0-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster0-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster1-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_redis_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.POOL_RESTART]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster1-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster2-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster2-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.POOL_RESTART]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster3-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
[gw0] SKIPPED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster3-celery_rabbitmq_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster4-celery_redis_broker-celery_redis_backend]
t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
[gw0] PASSED t/smoke/tests/stamping/test_hybrid_cluster.py::test_stamping_hybrid_worker_cluster::test_stamping_on_replace_with_legacy_worker_in_cluster[celery_worker_cluster4-celery_rabbitmq_broker-celery_redis_backend]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_GRACEFULLY]
t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]
[gw4] PASSED t/smoke/tests/test_worker.py::test_worker_restart::test_restart_between_task_execution[celery_setup_worker-celery_rabbitmq_broker-celery_redis_backend-Method.DOCKER_RESTART_FORCE]

===================================================== 115 passed, 14 skipped, 8 xfailed in 725.80s (0:12:05) ======================================================
.pkg: _exit> python /Users/nusnus/dev/GitHub/celery/.tox/.tox/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  3.12-smoke: OK (728.48=setup[1.44]+cmd[727.04] seconds)
  congratulations :) (728.80 seconds)

…s tracer.py. (celery#7925)" This reverts commit 0233c3b.

auvipy · 2024-01-17T18:09:19Z

Thats just wow

Nusnus · 2024-01-19T00:05:03Z

Thats just wow

Thank you 🙏@auvipy
This was a very long shot - I am glad it turned out useful after all 😊

hakanutku · 2024-01-23T10:11:48Z

Hey @Nusnus, thanks for all the work for the issue. But isn't it strange that type annotations caused this bug? Do you have any confirmation from those who experienced this bug? We upgraded to the newest version for now. I will post the updates to #7276

Nusnus · 2024-01-23T10:38:53Z

Hey @Nusnus, thanks for all the work for the issue. But isn't it strange that type annotations caused this bug? Do you have any confirmation from those who experienced this bug? We upgraded to the newest version for now. I will post the updates to #7276

Thank you 🙏

It appears the annotations PR changed more stuff (e.g imports) which could cause issues, so it probably wasn't the annotations themselves. trace.py unfortunately is a sensitive module.

Regarding updates, yes, it appears repeating restarts to Redis reproduce the issue, so this fixes for now just the first restart (without the suggested workarounds).

That being said, I need to confirm that myself as well and hopefully I'll be able to handle it when I do.

I should have updates by next week, I need to find some time to allocate for it.

Nusnus changed the title ~~Develop~~ Bugfix: Worker not consuming tasks after Redis broker restart Jan 16, 2024

Nusnus self-assigned this Jan 16, 2024

Nusnus added the PR Type: Bugfix label Jan 16, 2024

Nusnus added this to the 5.4 milestone Jan 16, 2024

Revert "Add annotations to minimise differences with celery-aio-pool'…

4565023

…s tracer.py. (celery#7925)" This reverts commit 0233c3b.

Nusnus force-pushed the develop branch from 0778c75 to d185327 Compare January 17, 2024 01:46

Added smoke test: test_worker_consume_tasks_after_redis_broker_restart

dbf419f

Nusnus force-pushed the develop branch from d185327 to dbf419f Compare January 17, 2024 08:59

Nusnus added 2 commits January 17, 2024 11:13

Removed Redis xfail from tests now that the bug is fixed

372cfe3

Renamed smoke tests CI jobs

3fd6c07

Nusnus marked this pull request as ready for review January 17, 2024 10:50

Nusnus requested a review from thedrow January 17, 2024 10:51

thedrow approved these changes Jan 17, 2024

View reviewed changes

Nusnus linked an issue Jan 17, 2024 that may be closed by this pull request

python 3.11 support #8227

Closed

4 tasks

Nusnus removed a link to an issue Jan 17, 2024

python 3.11 support #8227

Closed

4 tasks

Nusnus linked an issue Jan 17, 2024 that may be closed by this pull request

Don’t list redis as stable backend and/or link to issues/caveats #8568

Open

4 tasks

Nusnus removed a link to an issue Jan 17, 2024

Don’t list redis as stable backend and/or link to issues/caveats #8568

Open

4 tasks

Nusnus merged commit 78c06af into celery:main Jan 17, 2024
82 checks passed

Nusnus deleted the develop branch January 17, 2024 11:39

Nusnus mentioned this pull request Jan 21, 2024

Added examples/range celery/pytest-celery#169

Merged

PankajKhanwani mentioned this pull request Feb 13, 2024

Celery fails to handle scheduled tasks after connection recovery #8845

Open

18 tasks

Laityned mentioned this pull request Apr 20, 2024

Worker stops consuming tasks after redis reconnection on celery 5.2.3 #8030

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix: Worker not consuming tasks after Redis broker restart #8796

Bugfix: Worker not consuming tasks after Redis broker restart #8796

Nusnus commented Jan 16, 2024 •

edited

codecov bot commented Jan 16, 2024 •

edited

Nusnus commented Jan 17, 2024

auvipy commented Jan 17, 2024

Nusnus commented Jan 19, 2024

hakanutku commented Jan 23, 2024

Nusnus commented Jan 23, 2024

Bugfix: Worker not consuming tasks after Redis broker restart #8796

Bugfix: Worker not consuming tasks after Redis broker restart #8796

Conversation

Nusnus commented Jan 16, 2024 • edited

TL;DR

Long Story

codecov bot commented Jan 16, 2024 • edited

Codecov Report

Nusnus commented Jan 17, 2024

auvipy commented Jan 17, 2024

Nusnus commented Jan 19, 2024

hakanutku commented Jan 23, 2024

Nusnus commented Jan 23, 2024

Nusnus commented Jan 16, 2024 •

edited

codecov bot commented Jan 16, 2024 •

edited