Received error "[Errno 104] Connection reset by peer" after 180 seconds when waiting on task result #5358

chrahunt · 2019-02-23T18:00:53Z

celery issue

Checklist

I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
I have included all related issues and possible duplicate issues in this issue.
I have included the contents of pip freeze in the issue.
I have verified that the issue exists against the master branch of Celery.
I have tried reproducing the issue on more than one message broker and/or result backend.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with retries, ETA/Countdown & rate limits disabled.

Related Issues and Possible Duplicates

Related Issues

celery raise error: [Errno 104] Connection reset by peer after started #4867 - this was resolved using master, my issue was not
Connection reset by peer during on_error() callback calling a task #5247 - the scenario under which it is reproduced is different and they are using redis for transport and results

Possible Duplicates

None

Environment & Settings

Celery version: 4.3.0rc1 (rhubarb) (e257646)

celery report Output:

software -> celery:4.3.0rc1 (rhubarb) kombu:4.3.0 py:3.7.2
            billiard:3.6.0.0 py-amqp:2.4.1
platform -> system:Linux arch:64bit
            kernel version:4.19.11-041911-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:rpc://localhost/

broker_url: 'amqp://guest:********@localhost:5672//'
result_backend: 'rpc://localhost/'

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7.2
Minimal Broker Version: RabbitMQ 3.7.12
Minimal Result Backend Version: RabbitMQ 3.7.12
Minimal OS and/or Kernel Version: Ubuntu 18.04, 4.19.11
Minimal Broker Client Version: amqp 2.4.1
Minimal Result Backend Client Version: amqp 2.4.1

Python Packages

pip freeze Output:

amqp==2.4.1
billiard==3.6.0.0
-e git+git@github.com:celery/celery.git@e257646136e6fae73186d7385317f4e20cd36130#egg=celery
kombu==4.3.0
pytz==2018.9
redis==3.2.0
vine==1.2.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Execute

docker run -d --net=host redis:5.0.3
docker run -d --net=host rabbitmq:3.7.12

Save the following as repro.py.

import time

from datetime import datetime
from pathlib import Path

from celery import Celery


# not OK
app = Celery(Path(__file__).stem, backend='rpc://localhost', broker='pyamqp://')
# OK
#app = Celery(Path(__file__).stem, backend='rpc://localhost', broker='redis://')
# OK
#app = Celery(Path(__file__).stem, backend='redis://', broker='pyamqp://')


@app.task
def ping(v):
    time.sleep(10)
    return v


if __name__ == '__main__':
    start = time.time()
    tasks = [ping.delay(i) for i in range(100)]
    for task in tasks:
        result = task.get()
        now = time.time()
        timestamp = datetime.now().strftime('%Y-%m-%dT%H:%M:%S')
        print(f'{timestamp}: {result}: {now - start}')

Execute

celery -A repro worker --detach --hostname repro --logfile celery.log --loglevel DEBUG
python repro.py

Expected Behavior

The client should send all tasks to be executed, then wait on each task in turn. Each task should succeed and output should be printed in the console.

Actual Behavior

180 seconds (3 minutes) after the start of the client, a connection reset is received.

Traceback

...
2019-02-23T12:49:59: 66: 170.30147862434387
2019-02-23T12:49:59: 67: 170.30566883087158
Traceback (most recent call last):
  File "repro.py", line 27, in <module>
    result = task.get()
  File "/tmp/celery-repro/celery/celery/result.py", line 226, in get
    on_message=on_message,
  File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 188, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 255, in _wait_for_pending
    on_interval=on_interval):
  File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 56, in drain_events_until
    yield self.wait_for(p, wait, timeout=1)
  File "/tmp/celery-repro/celery/celery/backends/asynchronous.py", line 65, in wait_for
    wait(timeout=timeout)
  File "/tmp/celery-repro/celery/celery/backends/rpc.py", line 63, in drain_events
    return self._connection.drain_events(timeout=timeout)
  File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/kombu/connection.py", line 315, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/connection.py", line 500, in drain_events
    while not self.blocking_read(timeout):
  File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/connection.py", line 505, in blocking_read
    frame = self.transport.read_frame()
  File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/transport.py", line 256, in read_frame
    frame_header = read(7, True)
  File "/tmp/celery-repro/.venv/lib/python3.7/site-packages/amqp/transport.py", line 440, in _read
    s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer

Daemon logs

[2019-02-23 12:47:04,760: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-02-23 12:47:04,761: DEBUG/MainProcess] | Worker: Building graph...
[2019-02-23 12:47:04,762: DEBUG/MainProcess] | Worker: New boot order: {Beat, StateDB, Timer, Hub, Pool, Autoscaler, Consumer}
[2019-02-23 12:47:04,765: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-02-23 12:47:04,765: DEBUG/MainProcess] | Consumer: Building graph...
[2019-02-23 12:47:04,768: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Mingle, Gossip, Agent, Tasks, Control, event loop}
[2019-02-23 12:47:04,773: DEBUG/MainProcess] | Worker: Starting Hub
[2019-02-23 12:47:04,773: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,773: DEBUG/MainProcess] | Worker: Starting Pool
[2019-02-23 12:47:04,811: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,811: DEBUG/MainProcess] | Worker: Starting Consumer
[2019-02-23 12:47:04,811: DEBUG/MainProcess] | Consumer: Starting Connection
[2019-02-23 12:47:04,819: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:04,820: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-02-23 12:47:04,820: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,821: DEBUG/MainProcess] | Consumer: Starting Events
[2019-02-23 12:47:04,826: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:04,828: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,828: DEBUG/MainProcess] | Consumer: Starting Heart
[2019-02-23 12:47:04,828: DEBUG/MainProcess] using channel_id: 1
[2019-02-23 12:47:04,829: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:04,830: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:04,830: DEBUG/MainProcess] | Consumer: Starting Mingle
[2019-02-23 12:47:04,830: INFO/MainProcess] mingle: searching for neighbors
[2019-02-23 12:47:04,831: DEBUG/MainProcess] using channel_id: 1
[2019-02-23 12:47:04,831: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:04,839: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:04,840: DEBUG/MainProcess] using channel_id: 1
[2019-02-23 12:47:04,841: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:05,850: INFO/MainProcess] mingle: all alone
[2019-02-23 12:47:05,851: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,851: DEBUG/MainProcess] | Consumer: Starting Gossip
[2019-02-23 12:47:05,851: DEBUG/MainProcess] using channel_id: 2
[2019-02-23 12:47:05,854: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:05,857: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,857: DEBUG/MainProcess] | Consumer: Starting Tasks
[2019-02-23 12:47:05,860: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,860: DEBUG/MainProcess] | Consumer: Starting Control
[2019-02-23 12:47:05,860: DEBUG/MainProcess] using channel_id: 3
[2019-02-23 12:47:05,861: DEBUG/MainProcess] Channel open
[2019-02-23 12:47:05,863: DEBUG/MainProcess] ^-- substep ok
[2019-02-23 12:47:05,863: DEBUG/MainProcess] | Consumer: Starting event loop
[2019-02-23 12:47:05,863: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2019-02-23 12:47:05,864: INFO/MainProcess] celery@repro ready.
[2019-02-23 12:47:05,864: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2019-02-23 12:47:09,698: INFO/MainProcess] Received task: repro.ping[a233da5a-a69f-4d5e-b6e7-c7c29483aec4]  
[2019-02-23 12:47:09,702: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', {'lang': 'py', 'task': 'repro.ping', 'id': 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', 'parent_id': None, 'argsrepr': '(0,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a233da5a-a69f-4d5e-b6e7-c7c29483aec4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[0], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,703: INFO/MainProcess] Received task: repro.ping[a1569f36-1f95-4d2d-a8d0-94dfa9abe132]  
[2019-02-23 12:47:09,703: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', {'lang': 'py', 'task': 'repro.ping', 'id': 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', 'parent_id': None, 'argsrepr': '(1,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a1569f36-1f95-4d2d-a8d0-94dfa9abe132', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[1], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,703: INFO/MainProcess] Received task: repro.ping[58235f2a-1eb0-42e3-857f-3453aba57d32]  
[2019-02-23 12:47:09,703: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '58235f2a-1eb0-42e3-857f-3453aba57d32', {'lang': 'py', 'task': 'repro.ping', 'id': '58235f2a-1eb0-42e3-857f-3453aba57d32', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '58235f2a-1eb0-42e3-857f-3453aba57d32', 'parent_id': None, 'argsrepr': '(2,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '58235f2a-1eb0-42e3-857f-3453aba57d32', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[2], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,704: INFO/MainProcess] Received task: repro.ping[c4a63e95-ca68-4d22-9996-96490523b37d]  
[2019-02-23 12:47:09,705: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c4a63e95-ca68-4d22-9996-96490523b37d', {'lang': 'py', 'task': 'repro.ping', 'id': 'c4a63e95-ca68-4d22-9996-96490523b37d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c4a63e95-ca68-4d22-9996-96490523b37d', 'parent_id': None, 'argsrepr': '(3,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c4a63e95-ca68-4d22-9996-96490523b37d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[3], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:09,705: INFO/MainProcess] Received task: repro.ping[eab3cfe4-2b5e-4de7-bb5f-3d569871c85c]  
[2019-02-23 12:47:09,706: INFO/MainProcess] Received task: repro.ping[97a705fc-2208-4551-90b5-3fc1d54d79a0]  
[2019-02-23 12:47:09,706: DEBUG/MainProcess] Task accepted: repro.ping[58235f2a-1eb0-42e3-857f-3453aba57d32] pid:24588
[2019-02-23 12:47:09,706: DEBUG/MainProcess] Task accepted: repro.ping[a233da5a-a69f-4d5e-b6e7-c7c29483aec4] pid:24587
[2019-02-23 12:47:09,706: DEBUG/MainProcess] Task accepted: repro.ping[a1569f36-1f95-4d2d-a8d0-94dfa9abe132] pid:24589
[2019-02-23 12:47:09,707: INFO/MainProcess] Received task: repro.ping[a1eb440e-708c-4df2-a843-d678fca5876b]  
[2019-02-23 12:47:09,707: INFO/MainProcess] Received task: repro.ping[275b1a5a-6472-45f0-8710-4593e5f80b5a]  
[2019-02-23 12:47:09,707: INFO/MainProcess] Received task: repro.ping[6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc]  
[2019-02-23 12:47:09,708: INFO/MainProcess] Received task: repro.ping[31c2c135-76a5-4682-b400-3508ab9746ee]  
[2019-02-23 12:47:09,708: INFO/MainProcess] Received task: repro.ping[c7a5aba5-29f8-4046-9662-271035823431]  
[2019-02-23 12:47:09,708: INFO/MainProcess] Received task: repro.ping[e063884c-b7ba-48f6-8623-38a6d2ab174c]  
[2019-02-23 12:47:09,709: DEBUG/MainProcess] Task accepted: repro.ping[c4a63e95-ca68-4d22-9996-96490523b37d] pid:24590
[2019-02-23 12:47:09,709: INFO/MainProcess] Received task: repro.ping[f85a56c1-d26d-4fa2-8f81-1cfe60fff298]  
[2019-02-23 12:47:09,710: INFO/MainProcess] Received task: repro.ping[efa85c32-ef70-453b-97e8-57497f1245c2]  
[2019-02-23 12:47:09,711: INFO/MainProcess] Received task: repro.ping[b0957ce3-4915-4f86-a728-4d47759bee51]  
[2019-02-23 12:47:09,712: INFO/MainProcess] Received task: repro.ping[910ff039-f563-4f00-a631-fe131c9cedb4]  
[2019-02-23 12:47:09,714: INFO/MainProcess] Received task: repro.ping[3c059acc-519b-49b2-ba3d-e5d93bbcc2da]  
[2019-02-23 12:47:09,715: INFO/MainProcess] Received task: repro.ping[d75703fc-5e87-4152-97c7-72607249ecd8]  
[2019-02-23 12:47:09,715: INFO/MainProcess] Received task: repro.ping[b78ee077-4e13-4f38-b7f4-5baef4e88a1b]  
[2019-02-23 12:47:09,716: INFO/MainProcess] Received task: repro.ping[82ba77ec-6874-4414-a814-fe432e871005]  
[2019-02-23 12:47:19,758: DEBUG/ForkPoolWorker-4] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,760: DEBUG/ForkPoolWorker-3] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,762: DEBUG/ForkPoolWorker-4] using channel_id: 1
[2019-02-23 12:47:19,763: DEBUG/ForkPoolWorker-4] Channel open
[2019-02-23 12:47:19,762: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,764: DEBUG/ForkPoolWorker-3] using channel_id: 1
[2019-02-23 12:47:19,765: INFO/ForkPoolWorker-4] Task repro.ping[c4a63e95-ca68-4d22-9996-96490523b37d] succeeded in 10.05700475804042s: 3
[2019-02-23 12:47:19,765: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', {'lang': 'py', 'task': 'repro.ping', 'id': 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', 'parent_id': None, 'argsrepr': '(4,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'eab3cfe4-2b5e-4de7-bb5f-3d569871c85c', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[4], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,766: DEBUG/ForkPoolWorker-3] Channel open
[2019-02-23 12:47:19,767: INFO/ForkPoolWorker-3] Task repro.ping[a1569f36-1f95-4d2d-a8d0-94dfa9abe132] succeeded in 10.061639431049116s: 1
[2019-02-23 12:47:19,767: DEBUG/MainProcess] Task accepted: repro.ping[eab3cfe4-2b5e-4de7-bb5f-3d569871c85c] pid:24590
[2019-02-23 12:47:19,768: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '97a705fc-2208-4551-90b5-3fc1d54d79a0', {'lang': 'py', 'task': 'repro.ping', 'id': '97a705fc-2208-4551-90b5-3fc1d54d79a0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '97a705fc-2208-4551-90b5-3fc1d54d79a0', 'parent_id': None, 'argsrepr': '(5,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '97a705fc-2208-4551-90b5-3fc1d54d79a0', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[5], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,769: DEBUG/ForkPoolWorker-2] using channel_id: 1
[2019-02-23 12:47:19,769: INFO/MainProcess] Received task: repro.ping[91dbdb86-cbe6-4a57-b669-400211b3d477]  
[2019-02-23 12:47:19,770: DEBUG/MainProcess] Task accepted: repro.ping[97a705fc-2208-4551-90b5-3fc1d54d79a0] pid:24589
[2019-02-23 12:47:19,771: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-23 12:47:19,771: DEBUG/ForkPoolWorker-2] Channel open
[2019-02-23 12:47:19,771: INFO/MainProcess] Received task: repro.ping[d3767e82-7208-4b02-a8df-467e260f5afd]  
[2019-02-23 12:47:19,772: INFO/ForkPoolWorker-2] Task repro.ping[58235f2a-1eb0-42e3-857f-3453aba57d32] succeeded in 10.067062080022879s: 2
[2019-02-23 12:47:19,772: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a1eb440e-708c-4df2-a843-d678fca5876b', {'lang': 'py', 'task': 'repro.ping', 'id': 'a1eb440e-708c-4df2-a843-d678fca5876b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a1eb440e-708c-4df2-a843-d678fca5876b', 'parent_id': None, 'argsrepr': '(6,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a1eb440e-708c-4df2-a843-d678fca5876b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[6], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,772: DEBUG/ForkPoolWorker-1] using channel_id: 1
[2019-02-23 12:47:19,773: DEBUG/MainProcess] Task accepted: repro.ping[a1eb440e-708c-4df2-a843-d678fca5876b] pid:24588
[2019-02-23 12:47:19,775: DEBUG/ForkPoolWorker-1] Channel open
[2019-02-23 12:47:19,775: INFO/MainProcess] Received task: repro.ping[0208e029-4283-4a1d-b978-a2e12efc892a]  
[2019-02-23 12:47:19,775: INFO/ForkPoolWorker-1] Task repro.ping[a233da5a-a69f-4d5e-b6e7-c7c29483aec4] succeeded in 10.070296013960615s: 0
[2019-02-23 12:47:19,776: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '275b1a5a-6472-45f0-8710-4593e5f80b5a', {'lang': 'py', 'task': 'repro.ping', 'id': '275b1a5a-6472-45f0-8710-4593e5f80b5a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '275b1a5a-6472-45f0-8710-4593e5f80b5a', 'parent_id': None, 'argsrepr': '(7,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '275b1a5a-6472-45f0-8710-4593e5f80b5a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[7], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:19,776: DEBUG/MainProcess] Task accepted: repro.ping[275b1a5a-6472-45f0-8710-4593e5f80b5a] pid:24587
[2019-02-23 12:47:19,777: INFO/MainProcess] Received task: repro.ping[dd272fa0-226f-4b76-97a7-6b66d2a6f8ec]  
[2019-02-23 12:47:25,864: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:25,864: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: None/None, now - 36/130, monotonic - 827970.404138137, last_heartbeat_sent - 827970.404134595, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:29,771: INFO/ForkPoolWorker-4] Task repro.ping[eab3cfe4-2b5e-4de7-bb5f-3d569871c85c] succeeded in 10.002901564934291s: 4
[2019-02-23 12:47:29,771: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', {'lang': 'py', 'task': 'repro.ping', 'id': '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', 'parent_id': None, 'argsrepr': '(8,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[8], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,773: DEBUG/MainProcess] Task accepted: repro.ping[6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc] pid:24590
[2019-02-23 12:47:29,776: INFO/MainProcess] Received task: repro.ping[55146a44-57b3-4738-b0d8-af09d99db266]  
[2019-02-23 12:47:29,780: INFO/ForkPoolWorker-3] Task repro.ping[97a705fc-2208-4551-90b5-3fc1d54d79a0] succeeded in 10.00881665002089s: 5
[2019-02-23 12:47:29,781: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '31c2c135-76a5-4682-b400-3508ab9746ee', {'lang': 'py', 'task': 'repro.ping', 'id': '31c2c135-76a5-4682-b400-3508ab9746ee', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '31c2c135-76a5-4682-b400-3508ab9746ee', 'parent_id': None, 'argsrepr': '(9,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '31c2c135-76a5-4682-b400-3508ab9746ee', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[9], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,783: DEBUG/MainProcess] Task accepted: repro.ping[31c2c135-76a5-4682-b400-3508ab9746ee] pid:24589
[2019-02-23 12:47:29,784: INFO/ForkPoolWorker-2] Task repro.ping[a1eb440e-708c-4df2-a843-d678fca5876b] succeeded in 10.010429346002638s: 6
[2019-02-23 12:47:29,784: INFO/ForkPoolWorker-1] Task repro.ping[275b1a5a-6472-45f0-8710-4593e5f80b5a] succeeded in 10.008141046972014s: 7
[2019-02-23 12:47:29,786: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c7a5aba5-29f8-4046-9662-271035823431', {'lang': 'py', 'task': 'repro.ping', 'id': 'c7a5aba5-29f8-4046-9662-271035823431', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c7a5aba5-29f8-4046-9662-271035823431', 'parent_id': None, 'argsrepr': '(10,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c7a5aba5-29f8-4046-9662-271035823431', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,787: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'e063884c-b7ba-48f6-8623-38a6d2ab174c', {'lang': 'py', 'task': 'repro.ping', 'id': 'e063884c-b7ba-48f6-8623-38a6d2ab174c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e063884c-b7ba-48f6-8623-38a6d2ab174c', 'parent_id': None, 'argsrepr': '(11,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'e063884c-b7ba-48f6-8623-38a6d2ab174c', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[11], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:29,790: INFO/MainProcess] Received task: repro.ping[4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f]  
[2019-02-23 12:47:29,792: DEBUG/MainProcess] Task accepted: repro.ping[c7a5aba5-29f8-4046-9662-271035823431] pid:24587
[2019-02-23 12:47:29,792: DEBUG/MainProcess] Task accepted: repro.ping[e063884c-b7ba-48f6-8623-38a6d2ab174c] pid:24588
[2019-02-23 12:47:29,795: INFO/MainProcess] Received task: repro.ping[07fcf53c-b692-46ff-bd1a-58f337694f72]  
[2019-02-23 12:47:29,797: INFO/MainProcess] Received task: repro.ping[3ffbf0d3-3624-407f-a3fb-ad3dbbab6855]  
[2019-02-23 12:47:39,784: INFO/ForkPoolWorker-4] Task repro.ping[6ee7bc0b-1c84-4973-8954-c6b2f17bb9cc] succeeded in 10.011397822992876s: 8
[2019-02-23 12:47:39,785: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', {'lang': 'py', 'task': 'repro.ping', 'id': 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', 'parent_id': None, 'argsrepr': '(12,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f85a56c1-d26d-4fa2-8f81-1cfe60fff298', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[12], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,787: DEBUG/MainProcess] Task accepted: repro.ping[f85a56c1-d26d-4fa2-8f81-1cfe60fff298] pid:24590
[2019-02-23 12:47:39,790: INFO/MainProcess] Received task: repro.ping[eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a]  
[2019-02-23 12:47:39,792: INFO/ForkPoolWorker-3] Task repro.ping[31c2c135-76a5-4682-b400-3508ab9746ee] succeeded in 10.009430586942472s: 9
[2019-02-23 12:47:39,793: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'efa85c32-ef70-453b-97e8-57497f1245c2', {'lang': 'py', 'task': 'repro.ping', 'id': 'efa85c32-ef70-453b-97e8-57497f1245c2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'efa85c32-ef70-453b-97e8-57497f1245c2', 'parent_id': None, 'argsrepr': '(13,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'efa85c32-ef70-453b-97e8-57497f1245c2', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[13], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,794: DEBUG/MainProcess] Task accepted: repro.ping[efa85c32-ef70-453b-97e8-57497f1245c2] pid:24589
[2019-02-23 12:47:39,797: INFO/MainProcess] Received task: repro.ping[040e0b9e-7bd7-4b25-aa59-660e40531563]  
[2019-02-23 12:47:39,800: INFO/ForkPoolWorker-1] Task repro.ping[c7a5aba5-29f8-4046-9662-271035823431] succeeded in 10.009039321914315s: 10
[2019-02-23 12:47:39,800: INFO/ForkPoolWorker-2] Task repro.ping[e063884c-b7ba-48f6-8623-38a6d2ab174c] succeeded in 10.008984215906821s: 11
[2019-02-23 12:47:39,801: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'b0957ce3-4915-4f86-a728-4d47759bee51', {'lang': 'py', 'task': 'repro.ping', 'id': 'b0957ce3-4915-4f86-a728-4d47759bee51', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b0957ce3-4915-4f86-a728-4d47759bee51', 'parent_id': None, 'argsrepr': '(14,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'b0957ce3-4915-4f86-a728-4d47759bee51', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[14], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,802: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '910ff039-f563-4f00-a631-fe131c9cedb4', {'lang': 'py', 'task': 'repro.ping', 'id': '910ff039-f563-4f00-a631-fe131c9cedb4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '910ff039-f563-4f00-a631-fe131c9cedb4', 'parent_id': None, 'argsrepr': '(15,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '910ff039-f563-4f00-a631-fe131c9cedb4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[15], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:39,804: DEBUG/MainProcess] Task accepted: repro.ping[910ff039-f563-4f00-a631-fe131c9cedb4] pid:24588
[2019-02-23 12:47:39,804: DEBUG/MainProcess] Task accepted: repro.ping[b0957ce3-4915-4f86-a728-4d47759bee51] pid:24587
[2019-02-23 12:47:39,807: INFO/MainProcess] Received task: repro.ping[c9365f4b-74aa-4ad7-a78b-8527306b5c9b]  
[2019-02-23 12:47:39,809: INFO/MainProcess] Received task: repro.ping[f91595cc-22f6-402d-8e5f-0ca907a72b97]  
[2019-02-23 12:47:45,866: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:45,866: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 36/130, now - 44/185, monotonic - 827990.406034226, last_heartbeat_sent - 827990.406033, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:47:49,795: INFO/ForkPoolWorker-4] Task repro.ping[f85a56c1-d26d-4fa2-8f81-1cfe60fff298] succeeded in 10.008562478935346s: 12
[2019-02-23 12:47:49,796: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', {'lang': 'py', 'task': 'repro.ping', 'id': '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', 'parent_id': None, 'argsrepr': '(16,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '3c059acc-519b-49b2-ba3d-e5d93bbcc2da', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[16], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,798: DEBUG/MainProcess] Task accepted: repro.ping[3c059acc-519b-49b2-ba3d-e5d93bbcc2da] pid:24590
[2019-02-23 12:47:49,799: INFO/MainProcess] Received task: repro.ping[af2377f8-3ef9-4750-afee-f70bc9e4a6c8]  
[2019-02-23 12:47:49,803: INFO/ForkPoolWorker-3] Task repro.ping[efa85c32-ef70-453b-97e8-57497f1245c2] succeeded in 10.009383243974298s: 13
[2019-02-23 12:47:49,804: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd75703fc-5e87-4152-97c7-72607249ecd8', {'lang': 'py', 'task': 'repro.ping', 'id': 'd75703fc-5e87-4152-97c7-72607249ecd8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd75703fc-5e87-4152-97c7-72607249ecd8', 'parent_id': None, 'argsrepr': '(17,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd75703fc-5e87-4152-97c7-72607249ecd8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[17], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,805: DEBUG/MainProcess] Task accepted: repro.ping[d75703fc-5e87-4152-97c7-72607249ecd8] pid:24589
[2019-02-23 12:47:49,808: INFO/MainProcess] Received task: repro.ping[c674700d-be4d-4bfa-a890-e26fd937b30a]  
[2019-02-23 12:47:49,812: INFO/ForkPoolWorker-2] Task repro.ping[910ff039-f563-4f00-a631-fe131c9cedb4] succeeded in 10.008233243017457s: 15
[2019-02-23 12:47:49,812: INFO/ForkPoolWorker-1] Task repro.ping[b0957ce3-4915-4f86-a728-4d47759bee51] succeeded in 10.008232522988692s: 14
[2019-02-23 12:47:49,812: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', {'lang': 'py', 'task': 'repro.ping', 'id': 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', 'parent_id': None, 'argsrepr': '(18,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'b78ee077-4e13-4f38-b7f4-5baef4e88a1b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[18], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,813: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '82ba77ec-6874-4414-a814-fe432e871005', {'lang': 'py', 'task': 'repro.ping', 'id': '82ba77ec-6874-4414-a814-fe432e871005', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '82ba77ec-6874-4414-a814-fe432e871005', 'parent_id': None, 'argsrepr': '(19,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '82ba77ec-6874-4414-a814-fe432e871005', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[19], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:49,815: DEBUG/MainProcess] Task accepted: repro.ping[b78ee077-4e13-4f38-b7f4-5baef4e88a1b] pid:24587
[2019-02-23 12:47:49,815: DEBUG/MainProcess] Task accepted: repro.ping[82ba77ec-6874-4414-a814-fe432e871005] pid:24588
[2019-02-23 12:47:49,817: INFO/MainProcess] Received task: repro.ping[5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f]  
[2019-02-23 12:47:49,818: INFO/MainProcess] Received task: repro.ping[01dff44f-d3eb-4538-8f07-281e9733531b]  
[2019-02-23 12:47:59,809: INFO/ForkPoolWorker-4] Task repro.ping[3c059acc-519b-49b2-ba3d-e5d93bbcc2da] succeeded in 10.011367695988156s: 16
[2019-02-23 12:47:59,810: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '91dbdb86-cbe6-4a57-b669-400211b3d477', {'lang': 'py', 'task': 'repro.ping', 'id': '91dbdb86-cbe6-4a57-b669-400211b3d477', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '91dbdb86-cbe6-4a57-b669-400211b3d477', 'parent_id': None, 'argsrepr': '(20,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '91dbdb86-cbe6-4a57-b669-400211b3d477', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[20], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,812: DEBUG/MainProcess] Task accepted: repro.ping[91dbdb86-cbe6-4a57-b669-400211b3d477] pid:24590
[2019-02-23 12:47:59,815: INFO/MainProcess] Received task: repro.ping[9ac70541-1428-4877-a1a7-ca4d9196ea76]  
[2019-02-23 12:47:59,816: INFO/ForkPoolWorker-3] Task repro.ping[d75703fc-5e87-4152-97c7-72607249ecd8] succeeded in 10.010803959914483s: 17
[2019-02-23 12:47:59,817: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd3767e82-7208-4b02-a8df-467e260f5afd', {'lang': 'py', 'task': 'repro.ping', 'id': 'd3767e82-7208-4b02-a8df-467e260f5afd', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd3767e82-7208-4b02-a8df-467e260f5afd', 'parent_id': None, 'argsrepr': '(21,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd3767e82-7208-4b02-a8df-467e260f5afd', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[21], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,819: DEBUG/MainProcess] Task accepted: repro.ping[d3767e82-7208-4b02-a8df-467e260f5afd] pid:24589
[2019-02-23 12:47:59,823: INFO/MainProcess] Received task: repro.ping[372e0fca-0934-4260-9ac8-5cb5f3c8dfef]  
[2019-02-23 12:47:59,824: INFO/ForkPoolWorker-1] Task repro.ping[b78ee077-4e13-4f38-b7f4-5baef4e88a1b] succeeded in 10.009969289996661s: 18
[2019-02-23 12:47:59,827: INFO/ForkPoolWorker-2] Task repro.ping[82ba77ec-6874-4414-a814-fe432e871005] succeeded in 10.012633722042665s: 19
[2019-02-23 12:47:59,827: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '0208e029-4283-4a1d-b978-a2e12efc892a', {'lang': 'py', 'task': 'repro.ping', 'id': '0208e029-4283-4a1d-b978-a2e12efc892a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0208e029-4283-4a1d-b978-a2e12efc892a', 'parent_id': None, 'argsrepr': '(22,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '0208e029-4283-4a1d-b978-a2e12efc892a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[22], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,829: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', {'lang': 'py', 'task': 'repro.ping', 'id': 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', 'parent_id': None, 'argsrepr': '(23,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'dd272fa0-226f-4b76-97a7-6b66d2a6f8ec', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[23], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:47:59,831: DEBUG/MainProcess] Task accepted: repro.ping[dd272fa0-226f-4b76-97a7-6b66d2a6f8ec] pid:24588
[2019-02-23 12:47:59,832: DEBUG/MainProcess] Task accepted: repro.ping[0208e029-4283-4a1d-b978-a2e12efc892a] pid:24587
[2019-02-23 12:47:59,836: INFO/MainProcess] Received task: repro.ping[23920044-7453-426c-a804-8016859410c4]  
[2019-02-23 12:47:59,837: INFO/MainProcess] Received task: repro.ping[0b8c91fe-c9a8-49bc-b857-f04f12692fec]  
[2019-02-23 12:48:05,868: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:05,868: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 44/185, now - 52/239, monotonic - 828010.408377561, last_heartbeat_sent - 828010.408374154, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:09,816: INFO/ForkPoolWorker-4] Task repro.ping[91dbdb86-cbe6-4a57-b669-400211b3d477] succeeded in 10.004684619954787s: 20
[2019-02-23 12:48:09,817: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '55146a44-57b3-4738-b0d8-af09d99db266', {'lang': 'py', 'task': 'repro.ping', 'id': '55146a44-57b3-4738-b0d8-af09d99db266', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '55146a44-57b3-4738-b0d8-af09d99db266', 'parent_id': None, 'argsrepr': '(24,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '55146a44-57b3-4738-b0d8-af09d99db266', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[24], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,819: DEBUG/MainProcess] Task accepted: repro.ping[55146a44-57b3-4738-b0d8-af09d99db266] pid:24590
[2019-02-23 12:48:09,822: INFO/MainProcess] Received task: repro.ping[8b919229-36ab-4387-b8ff-446e461e462e]  
[2019-02-23 12:48:09,828: INFO/ForkPoolWorker-3] Task repro.ping[d3767e82-7208-4b02-a8df-467e260f5afd] succeeded in 10.0091544049792s: 21
[2019-02-23 12:48:09,829: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', {'lang': 'py', 'task': 'repro.ping', 'id': '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', 'parent_id': None, 'argsrepr': '(25,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[25], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,831: DEBUG/MainProcess] Task accepted: repro.ping[4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f] pid:24589
[2019-02-23 12:48:09,834: INFO/MainProcess] Received task: repro.ping[66105ea1-c574-4838-b297-cd1f861ac2fd]  
[2019-02-23 12:48:09,840: INFO/ForkPoolWorker-1] Task repro.ping[0208e029-4283-4a1d-b978-a2e12efc892a] succeeded in 10.009812505100854s: 22
[2019-02-23 12:48:09,841: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '07fcf53c-b692-46ff-bd1a-58f337694f72', {'lang': 'py', 'task': 'repro.ping', 'id': '07fcf53c-b692-46ff-bd1a-58f337694f72', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '07fcf53c-b692-46ff-bd1a-58f337694f72', 'parent_id': None, 'argsrepr': '(26,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '07fcf53c-b692-46ff-bd1a-58f337694f72', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[26], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,843: INFO/ForkPoolWorker-2] Task repro.ping[dd272fa0-226f-4b76-97a7-6b66d2a6f8ec] succeeded in 10.012293447973207s: 23
[2019-02-23 12:48:09,845: DEBUG/MainProcess] Task accepted: repro.ping[07fcf53c-b692-46ff-bd1a-58f337694f72] pid:24587
[2019-02-23 12:48:09,845: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', {'lang': 'py', 'task': 'repro.ping', 'id': '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', 'parent_id': None, 'argsrepr': '(27,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '3ffbf0d3-3624-407f-a3fb-ad3dbbab6855', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[27], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:09,848: INFO/MainProcess] Received task: repro.ping[322e398a-d9a2-486a-90d4-0012f0f1d804]  
[2019-02-23 12:48:09,850: DEBUG/MainProcess] Task accepted: repro.ping[3ffbf0d3-3624-407f-a3fb-ad3dbbab6855] pid:24588
[2019-02-23 12:48:09,852: INFO/MainProcess] Received task: repro.ping[d37541c2-81b1-429c-87b3-842d5bf76242]  
[2019-02-23 12:48:19,827: INFO/ForkPoolWorker-4] Task repro.ping[55146a44-57b3-4738-b0d8-af09d99db266] succeeded in 10.008311487035826s: 24
[2019-02-23 12:48:19,828: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', {'lang': 'py', 'task': 'repro.ping', 'id': 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', 'parent_id': None, 'argsrepr': '(28,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[28], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,828: DEBUG/MainProcess] Task accepted: repro.ping[eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a] pid:24590
[2019-02-23 12:48:19,830: INFO/MainProcess] Received task: repro.ping[ba330fe7-2223-4581-b803-6ba957c30f56]  
[2019-02-23 12:48:19,839: INFO/ForkPoolWorker-3] Task repro.ping[4a9e5f83-6fa2-4c09-a925-e51d3ad9f48f] succeeded in 10.00862043199595s: 25
[2019-02-23 12:48:19,840: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '040e0b9e-7bd7-4b25-aa59-660e40531563', {'lang': 'py', 'task': 'repro.ping', 'id': '040e0b9e-7bd7-4b25-aa59-660e40531563', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '040e0b9e-7bd7-4b25-aa59-660e40531563', 'parent_id': None, 'argsrepr': '(29,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '040e0b9e-7bd7-4b25-aa59-660e40531563', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[29], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,841: DEBUG/MainProcess] Task accepted: repro.ping[040e0b9e-7bd7-4b25-aa59-660e40531563] pid:24589
[2019-02-23 12:48:19,843: INFO/MainProcess] Received task: repro.ping[73383b31-c16d-442a-b35e-51dcbd248c64]  
[2019-02-23 12:48:19,851: INFO/ForkPoolWorker-1] Task repro.ping[07fcf53c-b692-46ff-bd1a-58f337694f72] succeeded in 10.008613758021966s: 26
[2019-02-23 12:48:19,851: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', {'lang': 'py', 'task': 'repro.ping', 'id': 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', 'parent_id': None, 'argsrepr': '(30,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c9365f4b-74aa-4ad7-a78b-8527306b5c9b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[30], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,852: DEBUG/MainProcess] Task accepted: repro.ping[c9365f4b-74aa-4ad7-a78b-8527306b5c9b] pid:24587
[2019-02-23 12:48:19,853: INFO/MainProcess] Received task: repro.ping[f71cd3c6-eeb3-424f-8a8a-a9830fed5a49]  
[2019-02-23 12:48:19,856: INFO/ForkPoolWorker-2] Task repro.ping[3ffbf0d3-3624-407f-a3fb-ad3dbbab6855] succeeded in 10.008551157079637s: 27
[2019-02-23 12:48:19,856: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f91595cc-22f6-402d-8e5f-0ca907a72b97', {'lang': 'py', 'task': 'repro.ping', 'id': 'f91595cc-22f6-402d-8e5f-0ca907a72b97', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f91595cc-22f6-402d-8e5f-0ca907a72b97', 'parent_id': None, 'argsrepr': '(31,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f91595cc-22f6-402d-8e5f-0ca907a72b97', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[31], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:19,857: DEBUG/MainProcess] Task accepted: repro.ping[f91595cc-22f6-402d-8e5f-0ca907a72b97] pid:24588
[2019-02-23 12:48:19,857: INFO/MainProcess] Received task: repro.ping[c5e658bb-43d9-4caa-99a6-63c9b5616b06]  
[2019-02-23 12:48:25,871: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:25,871: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 52/239, now - 60/293, monotonic - 828030.411141046, last_heartbeat_sent - 828030.411137678, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:29,840: INFO/ForkPoolWorker-4] Task repro.ping[eb59f7a0-16f4-4cd4-9d49-c6f8a08cd46a] succeeded in 10.011380798998289s: 28
[2019-02-23 12:48:29,841: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', {'lang': 'py', 'task': 'repro.ping', 'id': 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', 'parent_id': None, 'argsrepr': '(32,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'af2377f8-3ef9-4750-afee-f70bc9e4a6c8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[32], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,842: DEBUG/MainProcess] Task accepted: repro.ping[af2377f8-3ef9-4750-afee-f70bc9e4a6c8] pid:24590
[2019-02-23 12:48:29,845: INFO/MainProcess] Received task: repro.ping[bb2b74b4-1385-4321-8f8a-685dd1198873]  
[2019-02-23 12:48:29,852: INFO/ForkPoolWorker-3] Task repro.ping[040e0b9e-7bd7-4b25-aa59-660e40531563] succeeded in 10.01107842195779s: 29
[2019-02-23 12:48:29,853: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c674700d-be4d-4bfa-a890-e26fd937b30a', {'lang': 'py', 'task': 'repro.ping', 'id': 'c674700d-be4d-4bfa-a890-e26fd937b30a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c674700d-be4d-4bfa-a890-e26fd937b30a', 'parent_id': None, 'argsrepr': '(33,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c674700d-be4d-4bfa-a890-e26fd937b30a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[33], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,855: DEBUG/MainProcess] Task accepted: repro.ping[c674700d-be4d-4bfa-a890-e26fd937b30a] pid:24589
[2019-02-23 12:48:29,857: INFO/MainProcess] Received task: repro.ping[e8d7d0fe-700d-4b55-b3ef-422265f142a1]  
[2019-02-23 12:48:29,862: INFO/ForkPoolWorker-1] Task repro.ping[c9365f4b-74aa-4ad7-a78b-8527306b5c9b] succeeded in 10.009829671005718s: 30
[2019-02-23 12:48:29,863: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', {'lang': 'py', 'task': 'repro.ping', 'id': '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', 'parent_id': None, 'argsrepr': '(34,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[34], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,864: DEBUG/MainProcess] Task accepted: repro.ping[5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f] pid:24587
[2019-02-23 12:48:29,866: INFO/MainProcess] Received task: repro.ping[1a663699-ca2a-4ae1-a9ab-01094ad98e0d]  
[2019-02-23 12:48:29,868: INFO/ForkPoolWorker-2] Task repro.ping[f91595cc-22f6-402d-8e5f-0ca907a72b97] succeeded in 10.01112971897237s: 31
[2019-02-23 12:48:29,869: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '01dff44f-d3eb-4538-8f07-281e9733531b', {'lang': 'py', 'task': 'repro.ping', 'id': '01dff44f-d3eb-4538-8f07-281e9733531b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '01dff44f-d3eb-4538-8f07-281e9733531b', 'parent_id': None, 'argsrepr': '(35,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '01dff44f-d3eb-4538-8f07-281e9733531b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[35], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:29,870: DEBUG/MainProcess] Task accepted: repro.ping[01dff44f-d3eb-4538-8f07-281e9733531b] pid:24588
[2019-02-23 12:48:29,873: INFO/MainProcess] Received task: repro.ping[2c529c3a-2a82-45ce-bbf9-6b305a425efa]  
[2019-02-23 12:48:39,852: INFO/ForkPoolWorker-4] Task repro.ping[af2377f8-3ef9-4750-afee-f70bc9e4a6c8] succeeded in 10.009791500982828s: 32
[2019-02-23 12:48:39,853: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '9ac70541-1428-4877-a1a7-ca4d9196ea76', {'lang': 'py', 'task': 'repro.ping', 'id': '9ac70541-1428-4877-a1a7-ca4d9196ea76', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9ac70541-1428-4877-a1a7-ca4d9196ea76', 'parent_id': None, 'argsrepr': '(36,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '9ac70541-1428-4877-a1a7-ca4d9196ea76', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[36], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,856: DEBUG/MainProcess] Task accepted: repro.ping[9ac70541-1428-4877-a1a7-ca4d9196ea76] pid:24590
[2019-02-23 12:48:39,859: INFO/MainProcess] Received task: repro.ping[f3835161-e5d0-478a-87fc-b48f4f8ca51a]  
[2019-02-23 12:48:39,866: INFO/ForkPoolWorker-3] Task repro.ping[c674700d-be4d-4bfa-a890-e26fd937b30a] succeeded in 10.01175982109271s: 33
[2019-02-23 12:48:39,867: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', {'lang': 'py', 'task': 'repro.ping', 'id': '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', 'parent_id': None, 'argsrepr': '(37,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '372e0fca-0934-4260-9ac8-5cb5f3c8dfef', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[37], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,868: DEBUG/MainProcess] Task accepted: repro.ping[372e0fca-0934-4260-9ac8-5cb5f3c8dfef] pid:24589
[2019-02-23 12:48:39,870: INFO/MainProcess] Received task: repro.ping[d3ab0805-a4e6-4e57-a2e3-94e1fa957fa9]  
[2019-02-23 12:48:39,871: INFO/ForkPoolWorker-1] Task repro.ping[5c5fb57e-3408-409c-8c8e-4af1eb8d6e0f] succeeded in 10.007380731985904s: 34
[2019-02-23 12:48:39,872: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '23920044-7453-426c-a804-8016859410c4', {'lang': 'py', 'task': 'repro.ping', 'id': '23920044-7453-426c-a804-8016859410c4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '23920044-7453-426c-a804-8016859410c4', 'parent_id': None, 'argsrepr': '(38,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '23920044-7453-426c-a804-8016859410c4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[38], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,873: DEBUG/MainProcess] Task accepted: repro.ping[23920044-7453-426c-a804-8016859410c4] pid:24587
[2019-02-23 12:48:39,874: INFO/MainProcess] Received task: repro.ping[97d308ed-27d9-4755-88fb-dee6c6ee13f0]  
[2019-02-23 12:48:39,879: INFO/ForkPoolWorker-2] Task repro.ping[01dff44f-d3eb-4538-8f07-281e9733531b] succeeded in 10.009314763941802s: 35
[2019-02-23 12:48:39,880: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '0b8c91fe-c9a8-49bc-b857-f04f12692fec', {'lang': 'py', 'task': 'repro.ping', 'id': '0b8c91fe-c9a8-49bc-b857-f04f12692fec', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0b8c91fe-c9a8-49bc-b857-f04f12692fec', 'parent_id': None, 'argsrepr': '(39,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '0b8c91fe-c9a8-49bc-b857-f04f12692fec', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[39], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:39,881: DEBUG/MainProcess] Task accepted: repro.ping[0b8c91fe-c9a8-49bc-b857-f04f12692fec] pid:24588
[2019-02-23 12:48:39,883: INFO/MainProcess] Received task: repro.ping[7df6a52e-4f4d-4610-94eb-464ef0b4c0e6]  
[2019-02-23 12:48:45,873: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:45,873: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 60/293, now - 68/347, monotonic - 828050.413444373, last_heartbeat_sent - 828050.413440785, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:48:49,867: INFO/ForkPoolWorker-4] Task repro.ping[9ac70541-1428-4877-a1a7-ca4d9196ea76] succeeded in 10.011515118996613s: 36
[2019-02-23 12:48:49,868: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '8b919229-36ab-4387-b8ff-446e461e462e', {'lang': 'py', 'task': 'repro.ping', 'id': '8b919229-36ab-4387-b8ff-446e461e462e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8b919229-36ab-4387-b8ff-446e461e462e', 'parent_id': None, 'argsrepr': '(40,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '8b919229-36ab-4387-b8ff-446e461e462e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[40], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,870: DEBUG/MainProcess] Task accepted: repro.ping[8b919229-36ab-4387-b8ff-446e461e462e] pid:24590
[2019-02-23 12:48:49,872: INFO/MainProcess] Received task: repro.ping[6f6f6073-a6de-4a99-be5d-625ac6c31bd8]  
[2019-02-23 12:48:49,878: INFO/ForkPoolWorker-3] Task repro.ping[372e0fca-0934-4260-9ac8-5cb5f3c8dfef] succeeded in 10.009640446980484s: 37
[2019-02-23 12:48:49,879: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '66105ea1-c574-4838-b297-cd1f861ac2fd', {'lang': 'py', 'task': 'repro.ping', 'id': '66105ea1-c574-4838-b297-cd1f861ac2fd', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '66105ea1-c574-4838-b297-cd1f861ac2fd', 'parent_id': None, 'argsrepr': '(41,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '66105ea1-c574-4838-b297-cd1f861ac2fd', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[41], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,880: DEBUG/MainProcess] Task accepted: repro.ping[66105ea1-c574-4838-b297-cd1f861ac2fd] pid:24589
[2019-02-23 12:48:49,883: INFO/MainProcess] Received task: repro.ping[19060c06-4928-4f63-88af-f7bdfacdb948]  
[2019-02-23 12:48:49,884: INFO/ForkPoolWorker-1] Task repro.ping[23920044-7453-426c-a804-8016859410c4] succeeded in 10.01139860006515s: 38
[2019-02-23 12:48:49,885: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '322e398a-d9a2-486a-90d4-0012f0f1d804', {'lang': 'py', 'task': 'repro.ping', 'id': '322e398a-d9a2-486a-90d4-0012f0f1d804', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '322e398a-d9a2-486a-90d4-0012f0f1d804', 'parent_id': None, 'argsrepr': '(42,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '322e398a-d9a2-486a-90d4-0012f0f1d804', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[42], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,886: DEBUG/MainProcess] Task accepted: repro.ping[322e398a-d9a2-486a-90d4-0012f0f1d804] pid:24587
[2019-02-23 12:48:49,888: INFO/ForkPoolWorker-2] Task repro.ping[0b8c91fe-c9a8-49bc-b857-f04f12692fec] succeeded in 10.007518506026827s: 39
[2019-02-23 12:48:49,889: INFO/MainProcess] Received task: repro.ping[6fe4fe91-b320-40f0-ba71-45b013e71837]  
[2019-02-23 12:48:49,890: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd37541c2-81b1-429c-87b3-842d5bf76242', {'lang': 'py', 'task': 'repro.ping', 'id': 'd37541c2-81b1-429c-87b3-842d5bf76242', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd37541c2-81b1-429c-87b3-842d5bf76242', 'parent_id': None, 'argsrepr': '(43,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd37541c2-81b1-429c-87b3-842d5bf76242', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[43], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:49,891: DEBUG/MainProcess] Task accepted: repro.ping[d37541c2-81b1-429c-87b3-842d5bf76242] pid:24588
[2019-02-23 12:48:49,894: INFO/MainProcess] Received task: repro.ping[18be5fb9-0d29-405d-a0db-2d38028d1a2a]  
[2019-02-23 12:48:59,881: INFO/ForkPoolWorker-4] Task repro.ping[8b919229-36ab-4387-b8ff-446e461e462e] succeeded in 10.011438557994552s: 40
[2019-02-23 12:48:59,882: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'ba330fe7-2223-4581-b803-6ba957c30f56', {'lang': 'py', 'task': 'repro.ping', 'id': 'ba330fe7-2223-4581-b803-6ba957c30f56', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ba330fe7-2223-4581-b803-6ba957c30f56', 'parent_id': None, 'argsrepr': '(44,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'ba330fe7-2223-4581-b803-6ba957c30f56', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[44], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,884: DEBUG/MainProcess] Task accepted: repro.ping[ba330fe7-2223-4581-b803-6ba957c30f56] pid:24590
[2019-02-23 12:48:59,887: INFO/MainProcess] Received task: repro.ping[f793c1c3-f8eb-411b-bf20-50844468c3a4]  
[2019-02-23 12:48:59,890: INFO/ForkPoolWorker-3] Task repro.ping[66105ea1-c574-4838-b297-cd1f861ac2fd] succeeded in 10.009387177997269s: 41
[2019-02-23 12:48:59,892: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '73383b31-c16d-442a-b35e-51dcbd248c64', {'lang': 'py', 'task': 'repro.ping', 'id': '73383b31-c16d-442a-b35e-51dcbd248c64', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '73383b31-c16d-442a-b35e-51dcbd248c64', 'parent_id': None, 'argsrepr': '(45,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '73383b31-c16d-442a-b35e-51dcbd248c64', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[45], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,894: DEBUG/MainProcess] Task accepted: repro.ping[73383b31-c16d-442a-b35e-51dcbd248c64] pid:24589
[2019-02-23 12:48:59,897: INFO/MainProcess] Received task: repro.ping[ea1ae0a8-800e-4da4-9042-bb32994a75a9]  
[2019-02-23 12:48:59,900: INFO/ForkPoolWorker-2] Task repro.ping[d37541c2-81b1-429c-87b3-842d5bf76242] succeeded in 10.009441644069739s: 43
[2019-02-23 12:48:59,901: INFO/ForkPoolWorker-1] Task repro.ping[322e398a-d9a2-486a-90d4-0012f0f1d804] succeeded in 10.015280571999028s: 42
[2019-02-23 12:48:59,904: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', {'lang': 'py', 'task': 'repro.ping', 'id': 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', 'parent_id': None, 'argsrepr': '(46,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f71cd3c6-eeb3-424f-8a8a-a9830fed5a49', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[46], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,905: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', {'lang': 'py', 'task': 'repro.ping', 'id': 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', 'parent_id': None, 'argsrepr': '(47,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c5e658bb-43d9-4caa-99a6-63c9b5616b06', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[47], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:48:59,907: DEBUG/MainProcess] Task accepted: repro.ping[f71cd3c6-eeb3-424f-8a8a-a9830fed5a49] pid:24587
[2019-02-23 12:48:59,908: DEBUG/MainProcess] Task accepted: repro.ping[c5e658bb-43d9-4caa-99a6-63c9b5616b06] pid:24588
[2019-02-23 12:48:59,911: INFO/MainProcess] Received task: repro.ping[5e3de42f-a8b8-4674-98ee-c3c1242102b5]  
[2019-02-23 12:48:59,913: INFO/MainProcess] Received task: repro.ping[57aa3cec-d384-4d90-a96c-a8b02afb0054]  
[2019-02-23 12:49:05,876: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:05,876: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 68/347, now - 76/401, monotonic - 828070.416255205, last_heartbeat_sent - 828070.416251962, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:09,892: INFO/ForkPoolWorker-4] Task repro.ping[ba330fe7-2223-4581-b803-6ba957c30f56] succeeded in 10.008885035058483s: 44
[2019-02-23 12:49:09,894: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'bb2b74b4-1385-4321-8f8a-685dd1198873', {'lang': 'py', 'task': 'repro.ping', 'id': 'bb2b74b4-1385-4321-8f8a-685dd1198873', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'bb2b74b4-1385-4321-8f8a-685dd1198873', 'parent_id': None, 'argsrepr': '(48,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'bb2b74b4-1385-4321-8f8a-685dd1198873', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[48], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,895: DEBUG/MainProcess] Task accepted: repro.ping[bb2b74b4-1385-4321-8f8a-685dd1198873] pid:24590
[2019-02-23 12:49:09,899: INFO/MainProcess] Received task: repro.ping[3ddc421f-a045-4c9e-bbcd-9181ee22103f]  
[2019-02-23 12:49:09,904: INFO/ForkPoolWorker-3] Task repro.ping[73383b31-c16d-442a-b35e-51dcbd248c64] succeeded in 10.0099140930688s: 45
[2019-02-23 12:49:09,905: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', {'lang': 'py', 'task': 'repro.ping', 'id': 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', 'parent_id': None, 'argsrepr': '(49,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'e8d7d0fe-700d-4b55-b3ef-422265f142a1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[49], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,907: DEBUG/MainProcess] Task accepted: repro.ping[e8d7d0fe-700d-4b55-b3ef-422265f142a1] pid:24589
[2019-02-23 12:49:09,910: INFO/MainProcess] Received task: repro.ping[b15e95fe-17a1-4f86-a88e-686ec768e4fa]  
[2019-02-23 12:49:09,916: INFO/ForkPoolWorker-1] Task repro.ping[f71cd3c6-eeb3-424f-8a8a-a9830fed5a49] succeeded in 10.009642486926168s: 46
[2019-02-23 12:49:09,917: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', {'lang': 'py', 'task': 'repro.ping', 'id': '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', 'parent_id': None, 'argsrepr': '(50,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '1a663699-ca2a-4ae1-a9ab-01094ad98e0d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[50], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,918: INFO/ForkPoolWorker-2] Task repro.ping[c5e658bb-43d9-4caa-99a6-63c9b5616b06] succeeded in 10.011810526018962s: 47
[2019-02-23 12:49:09,919: DEBUG/MainProcess] Task accepted: repro.ping[1a663699-ca2a-4ae1-a9ab-01094ad98e0d] pid:24587
[2019-02-23 12:49:09,920: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '2c529c3a-2a82-45ce-bbf9-6b305a425efa', {'lang': 'py', 'task': 'repro.ping', 'id': '2c529c3a-2a82-45ce-bbf9-6b305a425efa', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '2c529c3a-2a82-45ce-bbf9-6b305a425efa', 'parent_id': None, 'argsrepr': '(51,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '2c529c3a-2a82-45ce-bbf9-6b305a425efa', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[51], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:09,922: DEBUG/MainProcess] Task accepted: repro.ping[2c529c3a-2a82-45ce-bbf9-6b305a425efa] pid:24588
[2019-02-23 12:49:09,924: INFO/MainProcess] Received task: repro.ping[a7795001-72f4-459a-aa67-0f8ff84b8c98]  
[2019-02-23 12:49:09,926: INFO/MainProcess] Received task: repro.ping[19b79375-25c4-44f9-a1b8-8eeed0ca2718]  
[2019-02-23 12:49:19,903: INFO/ForkPoolWorker-4] Task repro.ping[bb2b74b4-1385-4321-8f8a-685dd1198873] succeeded in 10.007830242044292s: 48
[2019-02-23 12:49:19,904: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', {'lang': 'py', 'task': 'repro.ping', 'id': 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', 'parent_id': None, 'argsrepr': '(52,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f3835161-e5d0-478a-87fc-b48f4f8ca51a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[52], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,904: DEBUG/MainProcess] Task accepted: repro.ping[f3835161-e5d0-478a-87fc-b48f4f8ca51a] pid:24590
[2019-02-23 12:49:19,906: INFO/MainProcess] Received task: repro.ping[99f3cc1d-580f-426e-b95a-a36a136fddd1]  
[2019-02-23 12:49:19,915: INFO/ForkPoolWorker-3] Task repro.ping[e8d7d0fe-700d-4b55-b3ef-422265f142a1] succeeded in 10.008701610029675s: 49
[2019-02-23 12:49:19,915: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', {'lang': 'py', 'task': 'repro.ping', 'id': 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', 'parent_id': None, 'argsrepr': '(53,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd3ab0805-a4e6-4e57-a2e3-94e1fa957fa9', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[53], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,916: DEBUG/MainProcess] Task accepted: repro.ping[d3ab0805-a4e6-4e57-a2e3-94e1fa957fa9] pid:24589
[2019-02-23 12:49:19,918: INFO/MainProcess] Received task: repro.ping[fc54e4cd-65c6-4a00-a580-4c09feb1186e]  
[2019-02-23 12:49:19,927: INFO/ForkPoolWorker-1] Task repro.ping[1a663699-ca2a-4ae1-a9ab-01094ad98e0d] succeeded in 10.008626613067463s: 50
[2019-02-23 12:49:19,928: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '97d308ed-27d9-4755-88fb-dee6c6ee13f0', {'lang': 'py', 'task': 'repro.ping', 'id': '97d308ed-27d9-4755-88fb-dee6c6ee13f0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '97d308ed-27d9-4755-88fb-dee6c6ee13f0', 'parent_id': None, 'argsrepr': '(54,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '97d308ed-27d9-4755-88fb-dee6c6ee13f0', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[54], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,928: DEBUG/MainProcess] Task accepted: repro.ping[97d308ed-27d9-4755-88fb-dee6c6ee13f0] pid:24587
[2019-02-23 12:49:19,929: INFO/MainProcess] Received task: repro.ping[18d236e6-81b9-4031-b5b5-0166822e03c5]  
[2019-02-23 12:49:19,931: INFO/ForkPoolWorker-2] Task repro.ping[2c529c3a-2a82-45ce-bbf9-6b305a425efa] succeeded in 10.009170565987006s: 51
[2019-02-23 12:49:19,931: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', {'lang': 'py', 'task': 'repro.ping', 'id': '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', 'parent_id': None, 'argsrepr': '(55,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '7df6a52e-4f4d-4610-94eb-464ef0b4c0e6', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[55], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:19,932: DEBUG/MainProcess] Task accepted: repro.ping[7df6a52e-4f4d-4610-94eb-464ef0b4c0e6] pid:24588
[2019-02-23 12:49:19,933: INFO/MainProcess] Received task: repro.ping[f7b10df0-5050-4cda-b2a1-6752c6d33002]  
[2019-02-23 12:49:25,878: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:25,878: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 76/401, now - 84/455, monotonic - 828090.418472588, last_heartbeat_sent - 828090.418469348, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:29,915: INFO/ForkPoolWorker-4] Task repro.ping[f3835161-e5d0-478a-87fc-b48f4f8ca51a] succeeded in 10.011374757043086s: 52
[2019-02-23 12:49:29,916: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', {'lang': 'py', 'task': 'repro.ping', 'id': '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', 'parent_id': None, 'argsrepr': '(56,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6f6f6073-a6de-4a99-be5d-625ac6c31bd8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[56], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,918: DEBUG/MainProcess] Task accepted: repro.ping[6f6f6073-a6de-4a99-be5d-625ac6c31bd8] pid:24590
[2019-02-23 12:49:29,920: INFO/MainProcess] Received task: repro.ping[2b58a237-5dc0-44ad-99a1-70aa12361d02]  
[2019-02-23 12:49:29,928: INFO/ForkPoolWorker-3] Task repro.ping[d3ab0805-a4e6-4e57-a2e3-94e1fa957fa9] succeeded in 10.012014006963s: 53
[2019-02-23 12:49:29,929: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '19060c06-4928-4f63-88af-f7bdfacdb948', {'lang': 'py', 'task': 'repro.ping', 'id': '19060c06-4928-4f63-88af-f7bdfacdb948', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '19060c06-4928-4f63-88af-f7bdfacdb948', 'parent_id': None, 'argsrepr': '(57,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '19060c06-4928-4f63-88af-f7bdfacdb948', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[57], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,930: DEBUG/MainProcess] Task accepted: repro.ping[19060c06-4928-4f63-88af-f7bdfacdb948] pid:24589
[2019-02-23 12:49:29,933: INFO/MainProcess] Received task: repro.ping[a9e5d083-2314-4d61-b9b3-040231a0f7aa]  
[2019-02-23 12:49:29,939: INFO/ForkPoolWorker-1] Task repro.ping[97d308ed-27d9-4755-88fb-dee6c6ee13f0] succeeded in 10.011442068964243s: 54
[2019-02-23 12:49:29,940: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6fe4fe91-b320-40f0-ba71-45b013e71837', {'lang': 'py', 'task': 'repro.ping', 'id': '6fe4fe91-b320-40f0-ba71-45b013e71837', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6fe4fe91-b320-40f0-ba71-45b013e71837', 'parent_id': None, 'argsrepr': '(58,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6fe4fe91-b320-40f0-ba71-45b013e71837', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[58], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,942: INFO/ForkPoolWorker-2] Task repro.ping[7df6a52e-4f4d-4610-94eb-464ef0b4c0e6] succeeded in 10.009457940002903s: 55
[2019-02-23 12:49:29,943: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '18be5fb9-0d29-405d-a0db-2d38028d1a2a', {'lang': 'py', 'task': 'repro.ping', 'id': '18be5fb9-0d29-405d-a0db-2d38028d1a2a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '18be5fb9-0d29-405d-a0db-2d38028d1a2a', 'parent_id': None, 'argsrepr': '(59,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '18be5fb9-0d29-405d-a0db-2d38028d1a2a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[59], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:29,943: DEBUG/MainProcess] Task accepted: repro.ping[6fe4fe91-b320-40f0-ba71-45b013e71837] pid:24587
[2019-02-23 12:49:29,946: INFO/MainProcess] Received task: repro.ping[55610225-f895-4d65-8c86-30115fe3b289]  
[2019-02-23 12:49:29,947: DEBUG/MainProcess] Task accepted: repro.ping[18be5fb9-0d29-405d-a0db-2d38028d1a2a] pid:24588
[2019-02-23 12:49:29,949: INFO/MainProcess] Received task: repro.ping[41f062cc-8d89-4b89-882b-5a16df33c4cf]  
[2019-02-23 12:49:39,928: INFO/ForkPoolWorker-4] Task repro.ping[6f6f6073-a6de-4a99-be5d-625ac6c31bd8] succeeded in 10.01031665992923s: 56
[2019-02-23 12:49:39,929: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f793c1c3-f8eb-411b-bf20-50844468c3a4', {'lang': 'py', 'task': 'repro.ping', 'id': 'f793c1c3-f8eb-411b-bf20-50844468c3a4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f793c1c3-f8eb-411b-bf20-50844468c3a4', 'parent_id': None, 'argsrepr': '(60,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f793c1c3-f8eb-411b-bf20-50844468c3a4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[60], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,930: DEBUG/MainProcess] Task accepted: repro.ping[f793c1c3-f8eb-411b-bf20-50844468c3a4] pid:24590
[2019-02-23 12:49:39,933: INFO/MainProcess] Received task: repro.ping[03218e58-eb5f-4c86-b465-9399bdc3bb9d]  
[2019-02-23 12:49:39,941: INFO/ForkPoolWorker-3] Task repro.ping[19060c06-4928-4f63-88af-f7bdfacdb948] succeeded in 10.010959548992105s: 57
[2019-02-23 12:49:39,942: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', {'lang': 'py', 'task': 'repro.ping', 'id': 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', 'parent_id': None, 'argsrepr': '(61,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'ea1ae0a8-800e-4da4-9042-bb32994a75a9', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[61], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,943: DEBUG/MainProcess] Task accepted: repro.ping[ea1ae0a8-800e-4da4-9042-bb32994a75a9] pid:24589
[2019-02-23 12:49:39,946: INFO/MainProcess] Received task: repro.ping[76dee37c-4dd5-4d2a-810d-2f93726706d2]  
[2019-02-23 12:49:39,952: INFO/ForkPoolWorker-1] Task repro.ping[6fe4fe91-b320-40f0-ba71-45b013e71837] succeeded in 10.009412415907718s: 58
[2019-02-23 12:49:39,953: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '5e3de42f-a8b8-4674-98ee-c3c1242102b5', {'lang': 'py', 'task': 'repro.ping', 'id': '5e3de42f-a8b8-4674-98ee-c3c1242102b5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '5e3de42f-a8b8-4674-98ee-c3c1242102b5', 'parent_id': None, 'argsrepr': '(62,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '5e3de42f-a8b8-4674-98ee-c3c1242102b5', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[62], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,955: DEBUG/MainProcess] Task accepted: repro.ping[5e3de42f-a8b8-4674-98ee-c3c1242102b5] pid:24587
[2019-02-23 12:49:39,956: INFO/ForkPoolWorker-2] Task repro.ping[18be5fb9-0d29-405d-a0db-2d38028d1a2a] succeeded in 10.010617804946378s: 59
[2019-02-23 12:49:39,957: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '57aa3cec-d384-4d90-a96c-a8b02afb0054', {'lang': 'py', 'task': 'repro.ping', 'id': '57aa3cec-d384-4d90-a96c-a8b02afb0054', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '57aa3cec-d384-4d90-a96c-a8b02afb0054', 'parent_id': None, 'argsrepr': '(63,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '57aa3cec-d384-4d90-a96c-a8b02afb0054', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[63], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:39,959: INFO/MainProcess] Received task: repro.ping[19e75040-a721-44df-9c0f-32a79e51fb98]  
[2019-02-23 12:49:39,960: DEBUG/MainProcess] Task accepted: repro.ping[57aa3cec-d384-4d90-a96c-a8b02afb0054] pid:24588
[2019-02-23 12:49:39,963: INFO/MainProcess] Received task: repro.ping[08edbae5-80e1-4f37-bb61-f55c6c6ef618]  
[2019-02-23 12:49:45,881: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:45,881: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 84/455, now - 92/509, monotonic - 828110.421275947, last_heartbeat_sent - 828110.421272869, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:49:49,942: INFO/ForkPoolWorker-4] Task repro.ping[f793c1c3-f8eb-411b-bf20-50844468c3a4] succeeded in 10.01146399998106s: 60
[2019-02-23 12:49:49,943: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '3ddc421f-a045-4c9e-bbcd-9181ee22103f', {'lang': 'py', 'task': 'repro.ping', 'id': '3ddc421f-a045-4c9e-bbcd-9181ee22103f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '3ddc421f-a045-4c9e-bbcd-9181ee22103f', 'parent_id': None, 'argsrepr': '(64,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '3ddc421f-a045-4c9e-bbcd-9181ee22103f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[64], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,944: DEBUG/MainProcess] Task accepted: repro.ping[3ddc421f-a045-4c9e-bbcd-9181ee22103f] pid:24590
[2019-02-23 12:49:49,947: INFO/MainProcess] Received task: repro.ping[2328251e-1e29-49d9-8fdc-936a577b64ba]  
[2019-02-23 12:49:49,952: INFO/ForkPoolWorker-3] Task repro.ping[ea1ae0a8-800e-4da4-9042-bb32994a75a9] succeeded in 10.008566718897782s: 61
[2019-02-23 12:49:49,953: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', {'lang': 'py', 'task': 'repro.ping', 'id': 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', 'parent_id': None, 'argsrepr': '(65,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'b15e95fe-17a1-4f86-a88e-686ec768e4fa', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[65], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,954: DEBUG/MainProcess] Task accepted: repro.ping[b15e95fe-17a1-4f86-a88e-686ec768e4fa] pid:24589
[2019-02-23 12:49:49,957: INFO/MainProcess] Received task: repro.ping[9d5dc543-cbe5-471f-9cdb-983e6a47ef0b]  
[2019-02-23 12:49:49,964: INFO/ForkPoolWorker-1] Task repro.ping[5e3de42f-a8b8-4674-98ee-c3c1242102b5] succeeded in 10.00951830809936s: 62
[2019-02-23 12:49:49,965: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a7795001-72f4-459a-aa67-0f8ff84b8c98', {'lang': 'py', 'task': 'repro.ping', 'id': 'a7795001-72f4-459a-aa67-0f8ff84b8c98', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a7795001-72f4-459a-aa67-0f8ff84b8c98', 'parent_id': None, 'argsrepr': '(66,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a7795001-72f4-459a-aa67-0f8ff84b8c98', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[66], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,967: DEBUG/MainProcess] Task accepted: repro.ping[a7795001-72f4-459a-aa67-0f8ff84b8c98] pid:24587
[2019-02-23 12:49:49,968: INFO/ForkPoolWorker-2] Task repro.ping[57aa3cec-d384-4d90-a96c-a8b02afb0054] succeeded in 10.00838961894624s: 63
[2019-02-23 12:49:49,969: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '19b79375-25c4-44f9-a1b8-8eeed0ca2718', {'lang': 'py', 'task': 'repro.ping', 'id': '19b79375-25c4-44f9-a1b8-8eeed0ca2718', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '19b79375-25c4-44f9-a1b8-8eeed0ca2718', 'parent_id': None, 'argsrepr': '(67,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '19b79375-25c4-44f9-a1b8-8eeed0ca2718', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[67], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:49,971: INFO/MainProcess] Received task: repro.ping[8a8bec13-1de9-428a-aaa6-c292b8e9e97c]  
[2019-02-23 12:49:49,972: DEBUG/MainProcess] Task accepted: repro.ping[19b79375-25c4-44f9-a1b8-8eeed0ca2718] pid:24588
[2019-02-23 12:49:49,974: INFO/MainProcess] Received task: repro.ping[6ddfa78d-b7a4-4197-a490-fb52e87be667]  
[2019-02-23 12:49:59,952: INFO/ForkPoolWorker-4] Task repro.ping[3ddc421f-a045-4c9e-bbcd-9181ee22103f] succeeded in 10.00812103797216s: 64
[2019-02-23 12:49:59,953: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '99f3cc1d-580f-426e-b95a-a36a136fddd1', {'lang': 'py', 'task': 'repro.ping', 'id': '99f3cc1d-580f-426e-b95a-a36a136fddd1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '99f3cc1d-580f-426e-b95a-a36a136fddd1', 'parent_id': None, 'argsrepr': '(68,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '99f3cc1d-580f-426e-b95a-a36a136fddd1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[68], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,955: DEBUG/MainProcess] Task accepted: repro.ping[99f3cc1d-580f-426e-b95a-a36a136fddd1] pid:24590
[2019-02-23 12:49:59,957: INFO/MainProcess] Received task: repro.ping[20f83297-3bfe-4eb2-8484-2b2f0443dd46]  
[2019-02-23 12:49:59,964: INFO/ForkPoolWorker-3] Task repro.ping[b15e95fe-17a1-4f86-a88e-686ec768e4fa] succeeded in 10.009902358986437s: 65
[2019-02-23 12:49:59,965: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', {'lang': 'py', 'task': 'repro.ping', 'id': 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', 'parent_id': None, 'argsrepr': '(69,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'fc54e4cd-65c6-4a00-a580-4c09feb1186e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[69], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,966: DEBUG/MainProcess] Task accepted: repro.ping[fc54e4cd-65c6-4a00-a580-4c09feb1186e] pid:24589
[2019-02-23 12:49:59,969: INFO/MainProcess] Received task: repro.ping[f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e]  
[2019-02-23 12:49:59,976: INFO/ForkPoolWorker-1] Task repro.ping[a7795001-72f4-459a-aa67-0f8ff84b8c98] succeeded in 10.00928835105151s: 66
[2019-02-23 12:49:59,977: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '18d236e6-81b9-4031-b5b5-0166822e03c5', {'lang': 'py', 'task': 'repro.ping', 'id': '18d236e6-81b9-4031-b5b5-0166822e03c5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '18d236e6-81b9-4031-b5b5-0166822e03c5', 'parent_id': None, 'argsrepr': '(70,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '18d236e6-81b9-4031-b5b5-0166822e03c5', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[70], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,978: DEBUG/MainProcess] Task accepted: repro.ping[18d236e6-81b9-4031-b5b5-0166822e03c5] pid:24587
[2019-02-23 12:49:59,980: INFO/ForkPoolWorker-2] Task repro.ping[19b79375-25c4-44f9-a1b8-8eeed0ca2718] succeeded in 10.008277541957796s: 67
[2019-02-23 12:49:59,981: INFO/MainProcess] Received task: repro.ping[cc639408-eda2-4743-ae2d-83a76d66691a]  
[2019-02-23 12:49:59,982: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f7b10df0-5050-4cda-b2a1-6752c6d33002', {'lang': 'py', 'task': 'repro.ping', 'id': 'f7b10df0-5050-4cda-b2a1-6752c6d33002', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f7b10df0-5050-4cda-b2a1-6752c6d33002', 'parent_id': None, 'argsrepr': '(71,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f7b10df0-5050-4cda-b2a1-6752c6d33002', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[71], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:49:59,984: DEBUG/MainProcess] Task accepted: repro.ping[f7b10df0-5050-4cda-b2a1-6752c6d33002] pid:24588
[2019-02-23 12:49:59,986: INFO/MainProcess] Received task: repro.ping[82e68b10-48c8-4343-b1e1-83d65712681a]  
[2019-02-23 12:50:05,883: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:05,883: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 92/509, now - 100/563, monotonic - 828130.423378809, last_heartbeat_sent - 828130.423375836, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:09,966: INFO/ForkPoolWorker-4] Task repro.ping[99f3cc1d-580f-426e-b95a-a36a136fddd1] succeeded in 10.01196253998205s: 68
[2019-02-23 12:50:09,967: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '2b58a237-5dc0-44ad-99a1-70aa12361d02', {'lang': 'py', 'task': 'repro.ping', 'id': '2b58a237-5dc0-44ad-99a1-70aa12361d02', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '2b58a237-5dc0-44ad-99a1-70aa12361d02', 'parent_id': None, 'argsrepr': '(72,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '2b58a237-5dc0-44ad-99a1-70aa12361d02', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[72], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,969: DEBUG/MainProcess] Task accepted: repro.ping[2b58a237-5dc0-44ad-99a1-70aa12361d02] pid:24590
[2019-02-23 12:50:09,971: INFO/MainProcess] Received task: repro.ping[0e75e7fd-0925-4a7a-932d-aff67100a893]  
[2019-02-23 12:50:09,976: INFO/ForkPoolWorker-3] Task repro.ping[fc54e4cd-65c6-4a00-a580-4c09feb1186e] succeeded in 10.009725094074383s: 69
[2019-02-23 12:50:09,976: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', {'lang': 'py', 'task': 'repro.ping', 'id': 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', 'parent_id': None, 'argsrepr': '(73,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a9e5d083-2314-4d61-b9b3-040231a0f7aa', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[73], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,978: DEBUG/MainProcess] Task accepted: repro.ping[a9e5d083-2314-4d61-b9b3-040231a0f7aa] pid:24589
[2019-02-23 12:50:09,980: INFO/MainProcess] Received task: repro.ping[6f64bb8f-849d-43cd-8361-64946efd769b]  
[2019-02-23 12:50:09,988: INFO/ForkPoolWorker-1] Task repro.ping[18d236e6-81b9-4031-b5b5-0166822e03c5] succeeded in 10.009856261895038s: 70
[2019-02-23 12:50:09,989: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '55610225-f895-4d65-8c86-30115fe3b289', {'lang': 'py', 'task': 'repro.ping', 'id': '55610225-f895-4d65-8c86-30115fe3b289', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '55610225-f895-4d65-8c86-30115fe3b289', 'parent_id': None, 'argsrepr': '(74,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '55610225-f895-4d65-8c86-30115fe3b289', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[74], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,992: INFO/ForkPoolWorker-2] Task repro.ping[f7b10df0-5050-4cda-b2a1-6752c6d33002] succeeded in 10.00843263999559s: 71
[2019-02-23 12:50:09,992: DEBUG/MainProcess] Task accepted: repro.ping[55610225-f895-4d65-8c86-30115fe3b289] pid:24587
[2019-02-23 12:50:09,993: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '41f062cc-8d89-4b89-882b-5a16df33c4cf', {'lang': 'py', 'task': 'repro.ping', 'id': '41f062cc-8d89-4b89-882b-5a16df33c4cf', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '41f062cc-8d89-4b89-882b-5a16df33c4cf', 'parent_id': None, 'argsrepr': '(75,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '41f062cc-8d89-4b89-882b-5a16df33c4cf', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[75], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:09,995: INFO/MainProcess] Received task: repro.ping[fe5f5b82-a723-4625-bc47-50a2ce124d55]  
[2019-02-23 12:50:09,997: DEBUG/MainProcess] Task accepted: repro.ping[41f062cc-8d89-4b89-882b-5a16df33c4cf] pid:24588
[2019-02-23 12:50:09,999: INFO/MainProcess] Received task: repro.ping[a0c88e06-0bd6-4f20-bfbf-c07848901c1b]  
[2019-02-23 12:50:19,972: INFO/ForkPoolWorker-4] Task repro.ping[2b58a237-5dc0-44ad-99a1-70aa12361d02] succeeded in 10.003870679996908s: 72
[2019-02-23 12:50:19,973: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '03218e58-eb5f-4c86-b465-9399bdc3bb9d', {'lang': 'py', 'task': 'repro.ping', 'id': '03218e58-eb5f-4c86-b465-9399bdc3bb9d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '03218e58-eb5f-4c86-b465-9399bdc3bb9d', 'parent_id': None, 'argsrepr': '(76,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '03218e58-eb5f-4c86-b465-9399bdc3bb9d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[76], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:19,974: DEBUG/MainProcess] Task accepted: repro.ping[03218e58-eb5f-4c86-b465-9399bdc3bb9d] pid:24590
[2019-02-23 12:50:19,977: INFO/MainProcess] Received task: repro.ping[d895b100-d028-4277-9e24-4c6a03bb03a7]  
[2019-02-23 12:50:19,988: INFO/ForkPoolWorker-3] Task repro.ping[a9e5d083-2314-4d61-b9b3-040231a0f7aa] succeeded in 10.010169839020818s: 73
[2019-02-23 12:50:19,989: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '76dee37c-4dd5-4d2a-810d-2f93726706d2', {'lang': 'py', 'task': 'repro.ping', 'id': '76dee37c-4dd5-4d2a-810d-2f93726706d2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '76dee37c-4dd5-4d2a-810d-2f93726706d2', 'parent_id': None, 'argsrepr': '(77,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '76dee37c-4dd5-4d2a-810d-2f93726706d2', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[77], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:19,990: DEBUG/MainProcess] Task accepted: repro.ping[76dee37c-4dd5-4d2a-810d-2f93726706d2] pid:24589
[2019-02-23 12:50:19,993: INFO/MainProcess] Received task: repro.ping[73f89536-2609-460a-8a40-d5beaf0b1be7]  
[2019-02-23 12:50:20,002: INFO/ForkPoolWorker-1] Task repro.ping[55610225-f895-4d65-8c86-30115fe3b289] succeeded in 10.011466949945316s: 74
[2019-02-23 12:50:20,003: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '19e75040-a721-44df-9c0f-32a79e51fb98', {'lang': 'py', 'task': 'repro.ping', 'id': '19e75040-a721-44df-9c0f-32a79e51fb98', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '19e75040-a721-44df-9c0f-32a79e51fb98', 'parent_id': None, 'argsrepr': '(78,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '19e75040-a721-44df-9c0f-32a79e51fb98', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[78], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:20,005: INFO/ForkPoolWorker-2] Task repro.ping[41f062cc-8d89-4b89-882b-5a16df33c4cf] succeeded in 10.008303896989673s: 75
[2019-02-23 12:50:20,005: DEBUG/MainProcess] Task accepted: repro.ping[19e75040-a721-44df-9c0f-32a79e51fb98] pid:24587
[2019-02-23 12:50:20,008: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '08edbae5-80e1-4f37-bb61-f55c6c6ef618', {'lang': 'py', 'task': 'repro.ping', 'id': '08edbae5-80e1-4f37-bb61-f55c6c6ef618', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '08edbae5-80e1-4f37-bb61-f55c6c6ef618', 'parent_id': None, 'argsrepr': '(79,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '08edbae5-80e1-4f37-bb61-f55c6c6ef618', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[79], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:20,010: INFO/MainProcess] Received task: repro.ping[a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b]  
[2019-02-23 12:50:20,011: DEBUG/MainProcess] Task accepted: repro.ping[08edbae5-80e1-4f37-bb61-f55c6c6ef618] pid:24588
[2019-02-23 12:50:20,014: INFO/MainProcess] Received task: repro.ping[ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7]  
[2019-02-23 12:50:25,885: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:25,885: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 100/563, now - 108/617, monotonic - 828150.425361468, last_heartbeat_sent - 828150.425358227, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:29,987: INFO/ForkPoolWorker-4] Task repro.ping[03218e58-eb5f-4c86-b465-9399bdc3bb9d] succeeded in 10.012512292014435s: 76
[2019-02-23 12:50:29,988: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '2328251e-1e29-49d9-8fdc-936a577b64ba', {'lang': 'py', 'task': 'repro.ping', 'id': '2328251e-1e29-49d9-8fdc-936a577b64ba', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '2328251e-1e29-49d9-8fdc-936a577b64ba', 'parent_id': None, 'argsrepr': '(80,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '2328251e-1e29-49d9-8fdc-936a577b64ba', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[80], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:29,989: DEBUG/MainProcess] Task accepted: repro.ping[2328251e-1e29-49d9-8fdc-936a577b64ba] pid:24590
[2019-02-23 12:50:29,991: INFO/MainProcess] Received task: repro.ping[c388da65-384b-4fd6-b773-fafb843dca0d]  
[2019-02-23 12:50:30,000: INFO/ForkPoolWorker-3] Task repro.ping[76dee37c-4dd5-4d2a-810d-2f93726706d2] succeeded in 10.009812535019591s: 77
[2019-02-23 12:50:30,001: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', {'lang': 'py', 'task': 'repro.ping', 'id': '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', 'parent_id': None, 'argsrepr': '(81,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '9d5dc543-cbe5-471f-9cdb-983e6a47ef0b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[81], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:30,002: DEBUG/MainProcess] Task accepted: repro.ping[9d5dc543-cbe5-471f-9cdb-983e6a47ef0b] pid:24589
[2019-02-23 12:50:30,004: INFO/MainProcess] Received task: repro.ping[04ce4455-f390-4b6c-adff-2a7771ae9ee5]  
[2019-02-23 12:50:30,016: INFO/ForkPoolWorker-1] Task repro.ping[19e75040-a721-44df-9c0f-32a79e51fb98] succeeded in 10.011256138095632s: 78
[2019-02-23 12:50:30,017: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', {'lang': 'py', 'task': 'repro.ping', 'id': '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', 'parent_id': None, 'argsrepr': '(82,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '8a8bec13-1de9-428a-aaa6-c292b8e9e97c', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[82], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:30,018: DEBUG/MainProcess] Task accepted: repro.ping[8a8bec13-1de9-428a-aaa6-c292b8e9e97c] pid:24587
[2019-02-23 12:50:30,021: INFO/MainProcess] Received task: repro.ping[9c9605c3-ab18-45b7-b565-55c5bdb856b7]  
[2019-02-23 12:50:30,022: INFO/ForkPoolWorker-2] Task repro.ping[08edbae5-80e1-4f37-bb61-f55c6c6ef618] succeeded in 10.011361771030352s: 79
[2019-02-23 12:50:30,023: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6ddfa78d-b7a4-4197-a490-fb52e87be667', {'lang': 'py', 'task': 'repro.ping', 'id': '6ddfa78d-b7a4-4197-a490-fb52e87be667', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6ddfa78d-b7a4-4197-a490-fb52e87be667', 'parent_id': None, 'argsrepr': '(83,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6ddfa78d-b7a4-4197-a490-fb52e87be667', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[83], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:30,025: DEBUG/MainProcess] Task accepted: repro.ping[6ddfa78d-b7a4-4197-a490-fb52e87be667] pid:24588
[2019-02-23 12:50:30,027: INFO/MainProcess] Received task: repro.ping[441dd75a-5691-4002-a9b6-765ad4485ed1]  
[2019-02-23 12:50:39,999: INFO/ForkPoolWorker-4] Task repro.ping[2328251e-1e29-49d9-8fdc-936a577b64ba] succeeded in 10.009627755964175s: 80
[2019-02-23 12:50:40,000: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '20f83297-3bfe-4eb2-8484-2b2f0443dd46', {'lang': 'py', 'task': 'repro.ping', 'id': '20f83297-3bfe-4eb2-8484-2b2f0443dd46', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '20f83297-3bfe-4eb2-8484-2b2f0443dd46', 'parent_id': None, 'argsrepr': '(84,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '20f83297-3bfe-4eb2-8484-2b2f0443dd46', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[84], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,001: DEBUG/MainProcess] Task accepted: repro.ping[20f83297-3bfe-4eb2-8484-2b2f0443dd46] pid:24590
[2019-02-23 12:50:40,012: INFO/ForkPoolWorker-3] Task repro.ping[9d5dc543-cbe5-471f-9cdb-983e6a47ef0b] succeeded in 10.010064441012219s: 81
[2019-02-23 12:50:40,013: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', {'lang': 'py', 'task': 'repro.ping', 'id': 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', 'parent_id': None, 'argsrepr': '(85,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[85], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,014: DEBUG/MainProcess] Task accepted: repro.ping[f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e] pid:24589
[2019-02-23 12:50:40,029: INFO/ForkPoolWorker-1] Task repro.ping[8a8bec13-1de9-428a-aaa6-c292b8e9e97c] succeeded in 10.011171153048053s: 82
[2019-02-23 12:50:40,030: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'cc639408-eda2-4743-ae2d-83a76d66691a', {'lang': 'py', 'task': 'repro.ping', 'id': 'cc639408-eda2-4743-ae2d-83a76d66691a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'cc639408-eda2-4743-ae2d-83a76d66691a', 'parent_id': None, 'argsrepr': '(86,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'cc639408-eda2-4743-ae2d-83a76d66691a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[86], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,032: DEBUG/MainProcess] Task accepted: repro.ping[cc639408-eda2-4743-ae2d-83a76d66691a] pid:24587
[2019-02-23 12:50:40,036: INFO/ForkPoolWorker-2] Task repro.ping[6ddfa78d-b7a4-4197-a490-fb52e87be667] succeeded in 10.011511688004248s: 83
[2019-02-23 12:50:40,037: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '82e68b10-48c8-4343-b1e1-83d65712681a', {'lang': 'py', 'task': 'repro.ping', 'id': '82e68b10-48c8-4343-b1e1-83d65712681a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '82e68b10-48c8-4343-b1e1-83d65712681a', 'parent_id': None, 'argsrepr': '(87,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '82e68b10-48c8-4343-b1e1-83d65712681a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[87], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:40,039: DEBUG/MainProcess] Task accepted: repro.ping[82e68b10-48c8-4343-b1e1-83d65712681a] pid:24588
[2019-02-23 12:50:45,887: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:45,887: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 108/617, now - 116/659, monotonic - 828170.427266152, last_heartbeat_sent - 828170.427263, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:50:50,012: INFO/ForkPoolWorker-4] Task repro.ping[20f83297-3bfe-4eb2-8484-2b2f0443dd46] succeeded in 10.011206874041818s: 84
[2019-02-23 12:50:50,013: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '0e75e7fd-0925-4a7a-932d-aff67100a893', {'lang': 'py', 'task': 'repro.ping', 'id': '0e75e7fd-0925-4a7a-932d-aff67100a893', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0e75e7fd-0925-4a7a-932d-aff67100a893', 'parent_id': None, 'argsrepr': '(88,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '0e75e7fd-0925-4a7a-932d-aff67100a893', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[88], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,015: DEBUG/MainProcess] Task accepted: repro.ping[0e75e7fd-0925-4a7a-932d-aff67100a893] pid:24590
[2019-02-23 12:50:50,024: INFO/ForkPoolWorker-3] Task repro.ping[f73b1e5b-9cec-4fdf-9e1d-ee02dc13716e] succeeded in 10.010063005029224s: 85
[2019-02-23 12:50:50,025: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '6f64bb8f-849d-43cd-8361-64946efd769b', {'lang': 'py', 'task': 'repro.ping', 'id': '6f64bb8f-849d-43cd-8361-64946efd769b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6f64bb8f-849d-43cd-8361-64946efd769b', 'parent_id': None, 'argsrepr': '(89,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '6f64bb8f-849d-43cd-8361-64946efd769b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[89], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,026: DEBUG/MainProcess] Task accepted: repro.ping[6f64bb8f-849d-43cd-8361-64946efd769b] pid:24589
[2019-02-23 12:50:50,040: INFO/ForkPoolWorker-1] Task repro.ping[cc639408-eda2-4743-ae2d-83a76d66691a] succeeded in 10.00802940595895s: 86
[2019-02-23 12:50:50,040: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'fe5f5b82-a723-4625-bc47-50a2ce124d55', {'lang': 'py', 'task': 'repro.ping', 'id': 'fe5f5b82-a723-4625-bc47-50a2ce124d55', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'fe5f5b82-a723-4625-bc47-50a2ce124d55', 'parent_id': None, 'argsrepr': '(90,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'fe5f5b82-a723-4625-bc47-50a2ce124d55', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[90], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,041: DEBUG/MainProcess] Task accepted: repro.ping[fe5f5b82-a723-4625-bc47-50a2ce124d55] pid:24587
[2019-02-23 12:50:50,047: INFO/ForkPoolWorker-2] Task repro.ping[82e68b10-48c8-4343-b1e1-83d65712681a] succeeded in 10.008901569060981s: 87
[2019-02-23 12:50:50,048: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', {'lang': 'py', 'task': 'repro.ping', 'id': 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', 'parent_id': None, 'argsrepr': '(91,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a0c88e06-0bd6-4f20-bfbf-c07848901c1b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[91], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:50:50,049: DEBUG/MainProcess] Task accepted: repro.ping[a0c88e06-0bd6-4f20-bfbf-c07848901c1b] pid:24588
[2019-02-23 12:51:00,025: INFO/ForkPoolWorker-4] Task repro.ping[0e75e7fd-0925-4a7a-932d-aff67100a893] succeeded in 10.010544363060035s: 88
[2019-02-23 12:51:00,025: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'd895b100-d028-4277-9e24-4c6a03bb03a7', {'lang': 'py', 'task': 'repro.ping', 'id': 'd895b100-d028-4277-9e24-4c6a03bb03a7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd895b100-d028-4277-9e24-4c6a03bb03a7', 'parent_id': None, 'argsrepr': '(92,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'd895b100-d028-4277-9e24-4c6a03bb03a7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[92], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,026: DEBUG/MainProcess] Task accepted: repro.ping[d895b100-d028-4277-9e24-4c6a03bb03a7] pid:24590
[2019-02-23 12:51:00,035: INFO/ForkPoolWorker-3] Task repro.ping[6f64bb8f-849d-43cd-8361-64946efd769b] succeeded in 10.009111239924096s: 89
[2019-02-23 12:51:00,035: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '73f89536-2609-460a-8a40-d5beaf0b1be7', {'lang': 'py', 'task': 'repro.ping', 'id': '73f89536-2609-460a-8a40-d5beaf0b1be7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '73f89536-2609-460a-8a40-d5beaf0b1be7', 'parent_id': None, 'argsrepr': '(93,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '73f89536-2609-460a-8a40-d5beaf0b1be7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[93], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,036: DEBUG/MainProcess] Task accepted: repro.ping[73f89536-2609-460a-8a40-d5beaf0b1be7] pid:24589
[2019-02-23 12:51:00,052: INFO/ForkPoolWorker-1] Task repro.ping[fe5f5b82-a723-4625-bc47-50a2ce124d55] succeeded in 10.010521587100811s: 90
[2019-02-23 12:51:00,052: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', {'lang': 'py', 'task': 'repro.ping', 'id': 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', 'parent_id': None, 'argsrepr': '(94,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[94], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,052: DEBUG/MainProcess] Task accepted: repro.ping[a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b] pid:24587
[2019-02-23 12:51:00,059: INFO/ForkPoolWorker-2] Task repro.ping[a0c88e06-0bd6-4f20-bfbf-c07848901c1b] succeeded in 10.009964159922674s: 91
[2019-02-23 12:51:00,059: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', {'lang': 'py', 'task': 'repro.ping', 'id': 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', 'parent_id': None, 'argsrepr': '(95,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[95], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:00,060: DEBUG/MainProcess] Task accepted: repro.ping[ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7] pid:24588
[2019-02-23 12:51:05,889: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:05,889: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 116/659, now - 124/689, monotonic - 828190.429048136, last_heartbeat_sent - 828190.429045052, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:10,037: INFO/ForkPoolWorker-4] Task repro.ping[d895b100-d028-4277-9e24-4c6a03bb03a7] succeeded in 10.011421873001382s: 92
[2019-02-23 12:51:10,038: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', 'c388da65-384b-4fd6-b773-fafb843dca0d', {'lang': 'py', 'task': 'repro.ping', 'id': 'c388da65-384b-4fd6-b773-fafb843dca0d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c388da65-384b-4fd6-b773-fafb843dca0d', 'parent_id': None, 'argsrepr': '(96,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': 'c388da65-384b-4fd6-b773-fafb843dca0d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[96], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,039: DEBUG/MainProcess] Task accepted: repro.ping[c388da65-384b-4fd6-b773-fafb843dca0d] pid:24590
[2019-02-23 12:51:10,044: INFO/ForkPoolWorker-3] Task repro.ping[73f89536-2609-460a-8a40-d5beaf0b1be7] succeeded in 10.008303378010169s: 93
[2019-02-23 12:51:10,045: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '04ce4455-f390-4b6c-adff-2a7771ae9ee5', {'lang': 'py', 'task': 'repro.ping', 'id': '04ce4455-f390-4b6c-adff-2a7771ae9ee5', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '04ce4455-f390-4b6c-adff-2a7771ae9ee5', 'parent_id': None, 'argsrepr': '(97,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '04ce4455-f390-4b6c-adff-2a7771ae9ee5', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[97], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,046: DEBUG/MainProcess] Task accepted: repro.ping[04ce4455-f390-4b6c-adff-2a7771ae9ee5] pid:24589
[2019-02-23 12:51:10,064: INFO/ForkPoolWorker-1] Task repro.ping[a6927c58-b7c1-4c2e-b7c7-cdfbe7c9ea0b] succeeded in 10.011372304987162s: 94
[2019-02-23 12:51:10,065: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '9c9605c3-ab18-45b7-b565-55c5bdb856b7', {'lang': 'py', 'task': 'repro.ping', 'id': '9c9605c3-ab18-45b7-b565-55c5bdb856b7', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9c9605c3-ab18-45b7-b565-55c5bdb856b7', 'parent_id': None, 'argsrepr': '(98,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '9c9605c3-ab18-45b7-b565-55c5bdb856b7', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[98], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,066: DEBUG/MainProcess] Task accepted: repro.ping[9c9605c3-ab18-45b7-b565-55c5bdb856b7] pid:24587
[2019-02-23 12:51:10,068: INFO/ForkPoolWorker-2] Task repro.ping[ea0e4c62-73d1-4d34-b98a-ad7851c1cbc7] succeeded in 10.008219892974012s: 95
[2019-02-23 12:51:10,069: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f92462897b8> (args:('repro.ping', '441dd75a-5691-4002-a9b6-765ad4485ed1', {'lang': 'py', 'task': 'repro.ping', 'id': '441dd75a-5691-4002-a9b6-765ad4485ed1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '441dd75a-5691-4002-a9b6-765ad4485ed1', 'parent_id': None, 'argsrepr': '(99,)', 'kwargsrepr': '{}', 'origin': 'gen24599@repro', 'reply_to': '13b4de4e-3451-330c-837d-91fcc7545757', 'correlation_id': '441dd75a-5691-4002-a9b6-765ad4485ed1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[99], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-23 12:51:10,070: DEBUG/MainProcess] Task accepted: repro.ping[441dd75a-5691-4002-a9b6-765ad4485ed1] pid:24588
[2019-02-23 12:51:20,050: INFO/ForkPoolWorker-4] Task repro.ping[c388da65-384b-4fd6-b773-fafb843dca0d] succeeded in 10.011102308984846s: 96
[2019-02-23 12:51:20,056: INFO/ForkPoolWorker-3] Task repro.ping[04ce4455-f390-4b6c-adff-2a7771ae9ee5] succeeded in 10.009683967917226s: 97
[2019-02-23 12:51:20,072: INFO/ForkPoolWorker-1] Task repro.ping[9c9605c3-ab18-45b7-b565-55c5bdb856b7] succeeded in 10.00583470694255s: 98
[2019-02-23 12:51:20,072: INFO/ForkPoolWorker-2] Task repro.ping[441dd75a-5691-4002-a9b6-765ad4485ed1] succeeded in 10.002141434000805s: 99
[2019-02-23 12:51:25,891: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:25,891: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 124/689, now - 128/719, monotonic - 828210.431072338, last_heartbeat_sent - 828210.431069012, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:45,893: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:51:45,893: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 128/719, now - 128/749, monotonic - 828230.433275143, last_heartbeat_sent - 828210.431069012, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:52:05,894: DEBUG/MainProcess] heartbeat_tick : for connection c96dac8c7b174cb3af302fd13ce1db55
[2019-02-23 12:52:05,895: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 128/749, now - 128/779, monotonic - 828250.43490132, last_heartbeat_sent - 828210.431069012, heartbeat int. - 60 for connection c96dac8c7b174cb3af302fd13ce1db55

Combinations tested:

broker	backend	issue seen
`pyampq://`	`rpc://localhost`	YES
`redis://`	`rpc://localhost`	NO
`pyamqp://`	`redis://`	NO

The text was updated successfully, but these errors were encountered:

marcuslind90 · 2019-02-25T18:12:36Z

I'm also experiencing this on 4.3.0rc1 even though it has been said in issues that this should be resolved in 4.3.0.

Using broker amqp://rabbitmq and result backend rpc

thedrow · 2019-02-26T09:23:24Z

Could you please try to reproduce this on the solo pool to ensure this is not a problem with prefork or gevent?

chrahunt · 2019-02-28T01:42:30Z

The issue still occurs when using the solo pool.

I used the same instructions as above, except the following command-line to start the worker:

celery -A repro worker --detach --hostname repro --logfile celery.log --loglevel DEBUG --pool solo

Script output/traceback

2019-02-27T20:25:14: 0: 10.03208875656128
2019-02-27T20:25:24: 1: 20.04633927345276
2019-02-27T20:25:34: 2: 30.0596764087677
2019-02-27T20:25:44: 3: 40.07441329956055
2019-02-27T20:25:54: 4: 50.089730739593506
2019-02-27T20:26:04: 5: 60.10718393325806
2019-02-27T20:26:14: 6: 70.12166166305542
2019-02-27T20:26:24: 7: 80.13692331314087
2019-02-27T20:26:34: 8: 90.14560532569885
2019-02-27T20:26:44: 9: 100.15549874305725
2019-02-27T20:26:54: 10: 110.1715259552002
2019-02-27T20:27:04: 11: 120.18532824516296
2019-02-27T20:27:14: 12: 130.1992917060852
2019-02-27T20:27:24: 13: 140.20705103874207
2019-02-27T20:27:34: 14: 150.21889424324036
2019-02-27T20:27:44: 15: 160.23034191131592
2019-02-27T20:27:54: 16: 170.23874020576477
Traceback (most recent call last):
  File "repro.py", line 27, in <module>
    result = task.get()
  File "/home/chris/PersonalDocuments/projects/celery/celery/celery/result.py", line 226, in get
    on_message=on_message,
  File "/home/chris/PersonalDocuments/projects/celery/celery/celery/backends/asynchronous.py", line 188, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/home/chris/PersonalDocuments/projects/celery/celery/celery/backends/asynchronous.py", line 255, in _wait_for_pending
    on_interval=on_interval):
  File "/home/chris/PersonalDocuments/projects/celery/celery/celery/backends/asynchronous.py", line 56, in drain_events_until
    yield self.wait_for(p, wait, timeout=1)
  File "/home/chris/PersonalDocuments/projects/celery/celery/celery/backends/asynchronous.py", line 65, in wait_for
    wait(timeout=timeout)
  File "/home/chris/PersonalDocuments/projects/celery/celery/celery/backends/rpc.py", line 63, in drain_events
    return self._connection.drain_events(timeout=timeout)
  File "/home/chris/.virtualenvs/celery-venv/lib/python3.7/site-packages/kombu/connection.py", line 315, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/home/chris/.virtualenvs/celery-venv/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/home/chris/.virtualenvs/celery-venv/lib/python3.7/site-packages/amqp/connection.py", line 500, in drain_events
    while not self.blocking_read(timeout):
  File "/home/chris/.virtualenvs/celery-venv/lib/python3.7/site-packages/amqp/connection.py", line 505, in blocking_read
    frame = self.transport.read_frame()
  File "/home/chris/.virtualenvs/celery-venv/lib/python3.7/site-packages/amqp/transport.py", line 256, in read_frame
    frame_header = read(7, True)
  File "/home/chris/.virtualenvs/celery-venv/lib/python3.7/site-packages/amqp/transport.py", line 440, in _read
    s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer

Log output

[2019-02-27 20:24:56,077: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-02-27 20:24:56,077: DEBUG/MainProcess] | Worker: Building graph...
[2019-02-27 20:24:56,078: DEBUG/MainProcess] | Worker: New boot order: {Beat, Timer, Hub, Pool, Autoscaler, StateDB, Consumer}
[2019-02-27 20:24:56,083: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-02-27 20:24:56,083: DEBUG/MainProcess] | Consumer: Building graph...
[2019-02-27 20:24:56,086: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Gossip, Heart, Agent, event loop}
[2019-02-27 20:24:56,091: DEBUG/MainProcess] | Worker: Starting Hub
[2019-02-27 20:24:56,091: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:56,091: DEBUG/MainProcess] | Worker: Starting Pool
[2019-02-27 20:24:56,091: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:56,091: DEBUG/MainProcess] | Worker: Starting Consumer
[2019-02-27 20:24:56,091: DEBUG/MainProcess] | Consumer: Starting Connection
[2019-02-27 20:24:56,098: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-27 20:24:56,099: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2019-02-27 20:24:56,100: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:56,100: DEBUG/MainProcess] | Consumer: Starting Events
[2019-02-27 20:24:56,105: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-27 20:24:56,106: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:56,106: DEBUG/MainProcess] | Consumer: Starting Mingle
[2019-02-27 20:24:56,106: INFO/MainProcess] mingle: searching for neighbors
[2019-02-27 20:24:56,106: DEBUG/MainProcess] using channel_id: 1
[2019-02-27 20:24:56,107: DEBUG/MainProcess] Channel open
[2019-02-27 20:24:56,114: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@repro', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 21.2.6', 'product': 'RabbitMQ', 'version': '3.7.12'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
[2019-02-27 20:24:56,115: DEBUG/MainProcess] using channel_id: 1
[2019-02-27 20:24:56,116: DEBUG/MainProcess] Channel open
[2019-02-27 20:24:57,125: INFO/MainProcess] mingle: all alone
[2019-02-27 20:24:57,125: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:57,125: DEBUG/MainProcess] | Consumer: Starting Tasks
[2019-02-27 20:24:57,132: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:57,132: DEBUG/MainProcess] | Consumer: Starting Control
[2019-02-27 20:24:57,132: DEBUG/MainProcess] using channel_id: 2
[2019-02-27 20:24:57,133: DEBUG/MainProcess] Channel open
[2019-02-27 20:24:57,135: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:57,135: DEBUG/MainProcess] | Consumer: Starting Gossip
[2019-02-27 20:24:57,135: DEBUG/MainProcess] using channel_id: 3
[2019-02-27 20:24:57,136: DEBUG/MainProcess] Channel open
[2019-02-27 20:24:57,140: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:57,140: DEBUG/MainProcess] | Consumer: Starting Heart
[2019-02-27 20:24:57,141: DEBUG/MainProcess] using channel_id: 1
[2019-02-27 20:24:57,141: DEBUG/MainProcess] Channel open
[2019-02-27 20:24:57,142: DEBUG/MainProcess] ^-- substep ok
[2019-02-27 20:24:57,142: DEBUG/MainProcess] | Consumer: Starting event loop
[2019-02-27 20:24:57,142: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2019-02-27 20:24:57,143: INFO/MainProcess] celery@repro ready.
[2019-02-27 20:24:57,143: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2019-02-27 20:25:04,771: INFO/MainProcess] Received task: repro.ping[d092d3dd-627b-4ded-b069-7606ffc865d0]  
[2019-02-27 20:25:04,771: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'd092d3dd-627b-4ded-b069-7606ffc865d0', {'lang': 'py', 'task': 'repro.ping', 'id': 'd092d3dd-627b-4ded-b069-7606ffc865d0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd092d3dd-627b-4ded-b069-7606ffc865d0', 'parent_id': None, 'argsrepr': '(0,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'd092d3dd-627b-4ded-b069-7606ffc865d0', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[0], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:25:04,771: DEBUG/MainProcess] Task accepted: repro.ping[d092d3dd-627b-4ded-b069-7606ffc865d0] pid:24165
[2019-02-27 20:25:14,780: INFO/MainProcess] Task repro.ping[d092d3dd-627b-4ded-b069-7606ffc865d0] succeeded in 10.008613193873316s: 0
[2019-02-27 20:25:14,784: INFO/MainProcess] Received task: repro.ping[a0f553f9-b53f-43e3-82de-f1d1da7ccdf4]  
[2019-02-27 20:25:14,785: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'a0f553f9-b53f-43e3-82de-f1d1da7ccdf4', {'lang': 'py', 'task': 'repro.ping', 'id': 'a0f553f9-b53f-43e3-82de-f1d1da7ccdf4', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'a0f553f9-b53f-43e3-82de-f1d1da7ccdf4', 'parent_id': None, 'argsrepr': '(1,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'a0f553f9-b53f-43e3-82de-f1d1da7ccdf4', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[1], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:25:14,785: DEBUG/MainProcess] Task accepted: repro.ping[a0f553f9-b53f-43e3-82de-f1d1da7ccdf4] pid:24165
[2019-02-27 20:25:24,796: INFO/MainProcess] Task repro.ping[a0f553f9-b53f-43e3-82de-f1d1da7ccdf4] succeeded in 10.010682038962841s: 1
[2019-02-27 20:25:24,799: DEBUG/MainProcess] heartbeat_tick : for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:25:24,799: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: None/None, now - 29/46, monotonic - 1201049.339046822, last_heartbeat_sent - 1201049.339043466, heartbeat int. - 60 for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:25:24,800: INFO/MainProcess] Received task: repro.ping[8cb75d69-c191-4bb2-b51e-f4e00c669262]  
[2019-02-27 20:25:24,800: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '8cb75d69-c191-4bb2-b51e-f4e00c669262', {'lang': 'py', 'task': 'repro.ping', 'id': '8cb75d69-c191-4bb2-b51e-f4e00c669262', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8cb75d69-c191-4bb2-b51e-f4e00c669262', 'parent_id': None, 'argsrepr': '(2,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '8cb75d69-c191-4bb2-b51e-f4e00c669262', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[2], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:25:24,801: DEBUG/MainProcess] Task accepted: repro.ping[8cb75d69-c191-4bb2-b51e-f4e00c669262] pid:24165
[2019-02-27 20:25:34,808: INFO/MainProcess] Task repro.ping[8cb75d69-c191-4bb2-b51e-f4e00c669262] succeeded in 10.006755573907867s: 2
[2019-02-27 20:25:34,809: INFO/MainProcess] Received task: repro.ping[6087eedb-cdd8-4a7c-a90c-f51bdb6ff0e6]  
[2019-02-27 20:25:34,810: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '6087eedb-cdd8-4a7c-a90c-f51bdb6ff0e6', {'lang': 'py', 'task': 'repro.ping', 'id': '6087eedb-cdd8-4a7c-a90c-f51bdb6ff0e6', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6087eedb-cdd8-4a7c-a90c-f51bdb6ff0e6', 'parent_id': None, 'argsrepr': '(3,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '6087eedb-cdd8-4a7c-a90c-f51bdb6ff0e6', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[3], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:25:34,810: DEBUG/MainProcess] Task accepted: repro.ping[6087eedb-cdd8-4a7c-a90c-f51bdb6ff0e6] pid:24165
[2019-02-27 20:25:44,822: INFO/MainProcess] Task repro.ping[6087eedb-cdd8-4a7c-a90c-f51bdb6ff0e6] succeeded in 10.011461528018117s: 3
[2019-02-27 20:25:44,825: INFO/MainProcess] Received task: repro.ping[fe919351-b405-4169-a425-5bc89a6b5a45]  
[2019-02-27 20:25:44,825: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'fe919351-b405-4169-a425-5bc89a6b5a45', {'lang': 'py', 'task': 'repro.ping', 'id': 'fe919351-b405-4169-a425-5bc89a6b5a45', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'fe919351-b405-4169-a425-5bc89a6b5a45', 'parent_id': None, 'argsrepr': '(4,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'fe919351-b405-4169-a425-5bc89a6b5a45', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[4], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:25:44,826: DEBUG/MainProcess] Task accepted: repro.ping[fe919351-b405-4169-a425-5bc89a6b5a45] pid:24165
[2019-02-27 20:25:54,838: INFO/MainProcess] Task repro.ping[fe919351-b405-4169-a425-5bc89a6b5a45] succeeded in 10.011443878058344s: 4
[2019-02-27 20:25:54,840: DEBUG/MainProcess] heartbeat_tick : for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:25:54,841: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 29/46, now - 31/55, monotonic - 1201079.381102741, last_heartbeat_sent - 1201079.381097917, heartbeat int. - 60 for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:25:54,843: INFO/MainProcess] Received task: repro.ping[d2280c7a-66dd-4003-a86a-ce1311e85405]  
[2019-02-27 20:25:54,843: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'd2280c7a-66dd-4003-a86a-ce1311e85405', {'lang': 'py', 'task': 'repro.ping', 'id': 'd2280c7a-66dd-4003-a86a-ce1311e85405', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd2280c7a-66dd-4003-a86a-ce1311e85405', 'parent_id': None, 'argsrepr': '(5,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'd2280c7a-66dd-4003-a86a-ce1311e85405', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[5], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:25:54,844: DEBUG/MainProcess] Task accepted: repro.ping[d2280c7a-66dd-4003-a86a-ce1311e85405] pid:24165
[2019-02-27 20:26:04,855: INFO/MainProcess] Task repro.ping[d2280c7a-66dd-4003-a86a-ce1311e85405] succeeded in 10.010887932032347s: 5
[2019-02-27 20:26:04,857: INFO/MainProcess] Received task: repro.ping[c39bc287-14b1-43ac-ba44-5a081f27de93]  
[2019-02-27 20:26:04,857: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'c39bc287-14b1-43ac-ba44-5a081f27de93', {'lang': 'py', 'task': 'repro.ping', 'id': 'c39bc287-14b1-43ac-ba44-5a081f27de93', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'c39bc287-14b1-43ac-ba44-5a081f27de93', 'parent_id': None, 'argsrepr': '(6,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'c39bc287-14b1-43ac-ba44-5a081f27de93', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[6], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:26:04,858: DEBUG/MainProcess] Task accepted: repro.ping[c39bc287-14b1-43ac-ba44-5a081f27de93] pid:24165
[2019-02-27 20:26:14,870: INFO/MainProcess] Task repro.ping[c39bc287-14b1-43ac-ba44-5a081f27de93] succeeded in 10.011449411977082s: 6
[2019-02-27 20:26:14,872: INFO/MainProcess] Received task: repro.ping[1a3d0643-176b-4bdb-be63-3a1f41aa101e]  
[2019-02-27 20:26:14,873: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '1a3d0643-176b-4bdb-be63-3a1f41aa101e', {'lang': 'py', 'task': 'repro.ping', 'id': '1a3d0643-176b-4bdb-be63-3a1f41aa101e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '1a3d0643-176b-4bdb-be63-3a1f41aa101e', 'parent_id': None, 'argsrepr': '(7,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '1a3d0643-176b-4bdb-be63-3a1f41aa101e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[7], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:26:14,873: DEBUG/MainProcess] Task accepted: repro.ping[1a3d0643-176b-4bdb-be63-3a1f41aa101e] pid:24165
[2019-02-27 20:26:24,885: INFO/MainProcess] Task repro.ping[1a3d0643-176b-4bdb-be63-3a1f41aa101e] succeeded in 10.011497425148264s: 7
[2019-02-27 20:26:24,887: INFO/MainProcess] Received task: repro.ping[4f958ce5-2cd3-4b9e-8d1a-9be25f170a7e]  
[2019-02-27 20:26:24,887: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '4f958ce5-2cd3-4b9e-8d1a-9be25f170a7e', {'lang': 'py', 'task': 'repro.ping', 'id': '4f958ce5-2cd3-4b9e-8d1a-9be25f170a7e', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '4f958ce5-2cd3-4b9e-8d1a-9be25f170a7e', 'parent_id': None, 'argsrepr': '(8,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '4f958ce5-2cd3-4b9e-8d1a-9be25f170a7e', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[8], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:26:24,888: DEBUG/MainProcess] Task accepted: repro.ping[4f958ce5-2cd3-4b9e-8d1a-9be25f170a7e] pid:24165
[2019-02-27 20:26:34,891: INFO/MainProcess] Task repro.ping[4f958ce5-2cd3-4b9e-8d1a-9be25f170a7e] succeeded in 10.002451698994264s: 8
[2019-02-27 20:26:34,892: DEBUG/MainProcess] heartbeat_tick : for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:26:34,893: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 31/55, now - 34/67, monotonic - 1201119.432809567, last_heartbeat_sent - 1201119.432806966, heartbeat int. - 60 for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:26:34,894: INFO/MainProcess] Received task: repro.ping[e4ddbc88-6da6-411e-9eeb-10114313b90d]  
[2019-02-27 20:26:34,894: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'e4ddbc88-6da6-411e-9eeb-10114313b90d', {'lang': 'py', 'task': 'repro.ping', 'id': 'e4ddbc88-6da6-411e-9eeb-10114313b90d', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e4ddbc88-6da6-411e-9eeb-10114313b90d', 'parent_id': None, 'argsrepr': '(9,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'e4ddbc88-6da6-411e-9eeb-10114313b90d', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[9], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:26:34,894: DEBUG/MainProcess] Task accepted: repro.ping[e4ddbc88-6da6-411e-9eeb-10114313b90d] pid:24165
[2019-02-27 20:26:44,905: INFO/MainProcess] Task repro.ping[e4ddbc88-6da6-411e-9eeb-10114313b90d] succeeded in 10.01076826499775s: 9
[2019-02-27 20:26:44,908: INFO/MainProcess] Received task: repro.ping[64385ba3-4e63-4a3b-bae4-b65c3658e8e8]  
[2019-02-27 20:26:44,909: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '64385ba3-4e63-4a3b-bae4-b65c3658e8e8', {'lang': 'py', 'task': 'repro.ping', 'id': '64385ba3-4e63-4a3b-bae4-b65c3658e8e8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '64385ba3-4e63-4a3b-bae4-b65c3658e8e8', 'parent_id': None, 'argsrepr': '(10,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '64385ba3-4e63-4a3b-bae4-b65c3658e8e8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:26:44,910: DEBUG/MainProcess] Task accepted: repro.ping[64385ba3-4e63-4a3b-bae4-b65c3658e8e8] pid:24165
[2019-02-27 20:26:54,920: INFO/MainProcess] Task repro.ping[64385ba3-4e63-4a3b-bae4-b65c3658e8e8] succeeded in 10.009829680202529s: 10
[2019-02-27 20:26:54,922: INFO/MainProcess] Received task: repro.ping[09a2ea1f-6418-44b5-9214-c510c40bdda8]  
[2019-02-27 20:26:54,922: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '09a2ea1f-6418-44b5-9214-c510c40bdda8', {'lang': 'py', 'task': 'repro.ping', 'id': '09a2ea1f-6418-44b5-9214-c510c40bdda8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '09a2ea1f-6418-44b5-9214-c510c40bdda8', 'parent_id': None, 'argsrepr': '(11,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '09a2ea1f-6418-44b5-9214-c510c40bdda8', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[11], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:26:54,923: DEBUG/MainProcess] Task accepted: repro.ping[09a2ea1f-6418-44b5-9214-c510c40bdda8] pid:24165
[2019-02-27 20:27:04,934: INFO/MainProcess] Task repro.ping[09a2ea1f-6418-44b5-9214-c510c40bdda8] succeeded in 10.01108838710934s: 11
[2019-02-27 20:27:04,935: INFO/MainProcess] Received task: repro.ping[b233320e-c8bf-4f14-b48a-baf60234da88]  
[2019-02-27 20:27:04,935: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'b233320e-c8bf-4f14-b48a-baf60234da88', {'lang': 'py', 'task': 'repro.ping', 'id': 'b233320e-c8bf-4f14-b48a-baf60234da88', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'b233320e-c8bf-4f14-b48a-baf60234da88', 'parent_id': None, 'argsrepr': '(12,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'b233320e-c8bf-4f14-b48a-baf60234da88', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[12], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:27:04,936: DEBUG/MainProcess] Task accepted: repro.ping[b233320e-c8bf-4f14-b48a-baf60234da88] pid:24165
[2019-02-27 20:27:14,947: INFO/MainProcess] Task repro.ping[b233320e-c8bf-4f14-b48a-baf60234da88] succeeded in 10.011413905071095s: 12
[2019-02-27 20:27:14,949: INFO/MainProcess] Received task: repro.ping[ebb99e3b-6499-4dac-8cda-a50468f7f6a6]  
[2019-02-27 20:27:14,950: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'ebb99e3b-6499-4dac-8cda-a50468f7f6a6', {'lang': 'py', 'task': 'repro.ping', 'id': 'ebb99e3b-6499-4dac-8cda-a50468f7f6a6', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ebb99e3b-6499-4dac-8cda-a50468f7f6a6', 'parent_id': None, 'argsrepr': '(13,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'ebb99e3b-6499-4dac-8cda-a50468f7f6a6', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[13], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:27:14,950: DEBUG/MainProcess] Task accepted: repro.ping[ebb99e3b-6499-4dac-8cda-a50468f7f6a6] pid:24165
[2019-02-27 20:27:24,955: INFO/MainProcess] Task repro.ping[ebb99e3b-6499-4dac-8cda-a50468f7f6a6] succeeded in 10.004966460168362s: 13
[2019-02-27 20:27:24,957: DEBUG/MainProcess] heartbeat_tick : for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:27:24,958: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 34/67, now - 38/82, monotonic - 1201169.497844129, last_heartbeat_sent - 1201169.49784148, heartbeat int. - 60 for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:27:24,959: INFO/MainProcess] Received task: repro.ping[9ab9b0bb-7274-4139-8d6c-bdef5ef6414f]  
[2019-02-27 20:27:24,959: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '9ab9b0bb-7274-4139-8d6c-bdef5ef6414f', {'lang': 'py', 'task': 'repro.ping', 'id': '9ab9b0bb-7274-4139-8d6c-bdef5ef6414f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9ab9b0bb-7274-4139-8d6c-bdef5ef6414f', 'parent_id': None, 'argsrepr': '(14,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '9ab9b0bb-7274-4139-8d6c-bdef5ef6414f', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[14], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:27:24,959: DEBUG/MainProcess] Task accepted: repro.ping[9ab9b0bb-7274-4139-8d6c-bdef5ef6414f] pid:24165
[2019-02-27 20:27:34,967: INFO/MainProcess] Task repro.ping[9ab9b0bb-7274-4139-8d6c-bdef5ef6414f] succeeded in 10.007927135797217s: 14
[2019-02-27 20:27:34,968: INFO/MainProcess] Received task: repro.ping[adda9b29-3c6e-4004-9a83-ef506ba754af]  
[2019-02-27 20:27:34,969: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', 'adda9b29-3c6e-4004-9a83-ef506ba754af', {'lang': 'py', 'task': 'repro.ping', 'id': 'adda9b29-3c6e-4004-9a83-ef506ba754af', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'adda9b29-3c6e-4004-9a83-ef506ba754af', 'parent_id': None, 'argsrepr': '(15,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': 'adda9b29-3c6e-4004-9a83-ef506ba754af', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[15], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:27:34,969: DEBUG/MainProcess] Task accepted: repro.ping[adda9b29-3c6e-4004-9a83-ef506ba754af] pid:24165
[2019-02-27 20:27:44,979: INFO/MainProcess] Task repro.ping[adda9b29-3c6e-4004-9a83-ef506ba754af] succeeded in 10.009986410848796s: 15
[2019-02-27 20:27:44,980: INFO/MainProcess] Received task: repro.ping[0fa1265d-14d2-476b-a3b7-77a2fc1284d2]  
[2019-02-27 20:27:44,980: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '0fa1265d-14d2-476b-a3b7-77a2fc1284d2', {'lang': 'py', 'task': 'repro.ping', 'id': '0fa1265d-14d2-476b-a3b7-77a2fc1284d2', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '0fa1265d-14d2-476b-a3b7-77a2fc1284d2', 'parent_id': None, 'argsrepr': '(16,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '0fa1265d-14d2-476b-a3b7-77a2fc1284d2', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[16], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:27:44,980: DEBUG/MainProcess] Task accepted: repro.ping[0fa1265d-14d2-476b-a3b7-77a2fc1284d2] pid:24165
[2019-02-27 20:27:54,987: INFO/MainProcess] Task repro.ping[0fa1265d-14d2-476b-a3b7-77a2fc1284d2] succeeded in 10.007031311979517s: 16
[2019-02-27 20:27:54,988: INFO/MainProcess] Received task: repro.ping[59d8ea5e-8189-4b9f-a0ae-ac9d76102c68]  
[2019-02-27 20:27:54,989: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '59d8ea5e-8189-4b9f-a0ae-ac9d76102c68', {'lang': 'py', 'task': 'repro.ping', 'id': '59d8ea5e-8189-4b9f-a0ae-ac9d76102c68', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '59d8ea5e-8189-4b9f-a0ae-ac9d76102c68', 'parent_id': None, 'argsrepr': '(17,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '59d8ea5e-8189-4b9f-a0ae-ac9d76102c68', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[17], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:27:54,989: DEBUG/MainProcess] Task accepted: repro.ping[59d8ea5e-8189-4b9f-a0ae-ac9d76102c68] pid:24165
[2019-02-27 20:28:05,000: INFO/MainProcess] Task repro.ping[59d8ea5e-8189-4b9f-a0ae-ac9d76102c68] succeeded in 10.010713741881773s: 17
[2019-02-27 20:28:05,001: INFO/MainProcess] Received task: repro.ping[063e7f67-f64e-4385-973e-2e2bba1dba9a]  
[2019-02-27 20:28:05,001: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '063e7f67-f64e-4385-973e-2e2bba1dba9a', {'lang': 'py', 'task': 'repro.ping', 'id': '063e7f67-f64e-4385-973e-2e2bba1dba9a', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '063e7f67-f64e-4385-973e-2e2bba1dba9a', 'parent_id': None, 'argsrepr': '(18,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '063e7f67-f64e-4385-973e-2e2bba1dba9a', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[18], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:28:05,002: DEBUG/MainProcess] Task accepted: repro.ping[063e7f67-f64e-4385-973e-2e2bba1dba9a] pid:24165
[2019-02-27 20:28:15,012: INFO/MainProcess] Task repro.ping[063e7f67-f64e-4385-973e-2e2bba1dba9a] succeeded in 10.010194696951658s: 18
[2019-02-27 20:28:15,015: DEBUG/MainProcess] heartbeat_tick : for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:28:15,015: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 38/82, now - 43/101, monotonic - 1201219.555538933, last_heartbeat_sent - 1201219.555537767, heartbeat int. - 60 for connection e583634090164c0f848ad742e5ac2bc0
[2019-02-27 20:28:15,016: INFO/MainProcess] Received task: repro.ping[7eb4f51c-daf0-4254-8065-b3bfd5183609]  
[2019-02-27 20:28:15,016: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '7eb4f51c-daf0-4254-8065-b3bfd5183609', {'lang': 'py', 'task': 'repro.ping', 'id': '7eb4f51c-daf0-4254-8065-b3bfd5183609', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '7eb4f51c-daf0-4254-8065-b3bfd5183609', 'parent_id': None, 'argsrepr': '(19,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '7eb4f51c-daf0-4254-8065-b3bfd5183609', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[19], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:28:15,016: DEBUG/MainProcess] Task accepted: repro.ping[7eb4f51c-daf0-4254-8065-b3bfd5183609] pid:24165
[2019-02-27 20:28:25,026: INFO/MainProcess] Task repro.ping[7eb4f51c-daf0-4254-8065-b3bfd5183609] succeeded in 10.009277933975682s: 19
[2019-02-27 20:28:25,026: INFO/MainProcess] Received task: repro.ping[62c45b3a-4e4e-4595-945c-8dac5184d8cf]  
[2019-02-27 20:28:25,027: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fadadebaa60> (args:('repro.ping', '62c45b3a-4e4e-4595-945c-8dac5184d8cf', {'lang': 'py', 'task': 'repro.ping', 'id': '62c45b3a-4e4e-4595-945c-8dac5184d8cf', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '62c45b3a-4e4e-4595-945c-8dac5184d8cf', 'parent_id': None, 'argsrepr': '(20,)', 'kwargsrepr': '{}', 'origin': 'gen24187@repro', 'reply_to': '0096f45a-611f-3b7e-b600-0bf1658468c3', 'correlation_id': '62c45b3a-4e4e-4595-945c-8dac5184d8cf', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': False}}, '[[20], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-02-27 20:28:25,027: DEBUG/MainProcess] Task accepted: repro.ping[62c45b3a-4e4e-4595-945c-8dac5184d8cf] pid:24165
[2019-02-27 20:28:35,035: INFO/MainProcess] Task repro.ping[62c45b3a-4e4e-4595-945c-8dac5184d8cf] succeeded in 10.008485928876325s: 20

thedrow · 2019-02-28T14:54:39Z

Do you have anything useful in RabbitMQ's logs?
Is the result consumed or does it remain in the queue?

last-partizan · 2019-03-15T11:21:46Z

@thedrow i am trying to reproduce this issue, it throws Errno 104 after 360seconds in my case.

And result is consumed (I'm checking Queue celery in rabbitmq management web-interface, and there is no messages).

Rabbitmq logs does not have anything at all.

Traceback from repro.py


2019-03-15T13:10:38: 33: 344.73499369621277
2019-03-15T13:10:48: 34: 354.86570978164673
Traceback (most recent call last):
  File "repro.py", line 31, in 
    result = ping.delay(i).get()
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/celery/result.py", line 226, in get                   
    on_message=on_message,
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
    on_interval=on_interval):
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 56, in drain_events_until
    yield self.wait_for(p, wait, timeout=1)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/celery/backends/asynchronous.py", line 65, in wait_for
    wait(timeout=timeout)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/celery/backends/rpc.py", line 63, in drain_events     
    return self._connection.drain_events(timeout=timeout)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/kombu/connection.py", line 315, in drain_events       
    return self.transport.drain_events(self.connection, **kwargs)                                                               
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events 
    return connection.drain_events(**kwargs)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/amqp/connection.py", line 500, in drain_events        
    while not self.blocking_read(timeout):
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/amqp/connection.py", line 505, in blocking_read       
    frame = self.transport.read_frame()
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/amqp/transport.py", line 256, in read_frame           
    frame_header = read(7, True)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.7/site-packages/amqp/transport.py", line 440, in _read                
    s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer

Same traceback shared via sentry: https://sentry.io/share/issue/393ef556ff514bf9a019d858283bed49/

thedrow · 2019-03-24T06:51:34Z

Could this be related to client heartbeat?
Can you decrease the value of the heartbeat to make it more frequent?

last-partizan · 2019-03-24T08:41:03Z

If i set broker_heartbeat to 20 seconds, it will still fail after 355 seconds.

chrisportela · 2019-04-03T16:04:10Z

I'm am seeing the same problem with RabbitMQ 3.2.4, Python 2.7.15rc1, Ubuntu 18.04, and latest Kombu/Celery via pyampq.

Shookit · 2019-04-04T13:57:48Z

Just chiming in, but I see the same thing happening on 4.3.0, but on Python 3.6.

My worker is running under the following command:

celery worker --loglevel=INFO --without-mingle --without-gossip -P solo -Ofair --queues celery --maxtasksperchild 10 -A celery_main.celery -- worker_prefetch_multiplier=1

last-partizan · 2019-04-04T14:31:48Z

@thedrow It is related to heartbeat after all.

But i was setting heartbeat this way:

app = Celery(
    "repro",
    backend='rpc://',
    broker='amqp://',
    broker_heartbeat=10,
)

And this way heartbeat still uses default value 120.

So i changed code to

app = Celery(
    "repro",
    backend='rpc://',
    broker='amqp://',
)
app.config_from_object({
    "broker_heartbeat": 10,
})

And now it fails very fast:

> pipenv run python repro.py
2019-04-04T17:29:07: 0: 10.707868337631226
2019-04-04T17:29:17: 1: 21.063886642456055
Traceback (most recent call last):
  File "repro.py", line 31, in <module>
    result = ping.delay(i).get()
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/celery/result.py", line 226, in get               
    on_message=on_message,
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 188, in wait_for_pending
    for _ in self._wait_for_pending(result, **kwargs):
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 255, in _wait_for_pending
    on_interval=on_interval):
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 56, in drain_events_until
    yield self.wait_for(p, wait, timeout=1)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 65, in wait_for
    wait(timeout=timeout)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/celery/backends/rpc.py", line 63, in drain_events 
    return self._connection.drain_events(timeout=timeout)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/kombu/connection.py", line 315, in drain_events   
    return self.transport.drain_events(self.connection, **kwargs)                                                           
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/amqp/connection.py", line 500, in drain_events    
    while not self.blocking_read(timeout):
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/amqp/connection.py", line 505, in blocking_read   
    frame = self.transport.read_frame()
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/amqp/transport.py", line 256, in read_frame       
    frame_header = read(7, True)
  File "/home/serg/work/git/celery-5358/.venv/lib/python3.6/site-packages/amqp/transport.py", line 440, in _read            
    s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer

last-partizan · 2019-04-04T14:53:25Z

Looks like it is same bug as #4876

And can be fixed by reverting #4148

thedrow · 2019-04-04T15:56:37Z

Yes. I'm going to close this as a duplicate of #4876.
We can't revert #4148 because it exposes another bug.
A solution to both would be much appreciated.

chrisportela · 2019-04-04T16:29:49Z

Looking at #4876 I wanted to also add that I'm only seeing this issue in the client (Django server) so it does seem to be a duplicate.

venu13 · 2020-09-11T20:32:07Z

What is the solution to this bug?

thedrow · 2020-09-13T10:46:43Z

We don't have one.
We ended up reverting #4148 because this bug affects more people than the other one.

Ixiodor · 2020-09-28T16:26:43Z

Well...reverted when? This BUG still persist so the revert didn't worked seems. (4.4.7)
My worker is restarting forever without progression.
Edit:
Launched manually a worker as solo pool and unlocked the situation...but this is a unacceptable fix.

thedrow · 2020-10-01T13:25:42Z

I agree it isn't an acceptable fix but we don't have a solution currently.
If you can, please investigate.

Ixiodor · 2020-10-01T13:32:08Z

I'll try DEBUG on free time.
However, seems something related to reciving a too large message. I never get this error on ~200 queue messages but if my queue reaches something like 2000, i get a connection reset by peer. (Seems like Celery or Broker get flooded)

preeth1 · 2021-03-05T19:44:12Z

It is confusing to close issues that haven't been solved yet

thedrow added the Status: Feedback Needed ✘ label Feb 26, 2019

thedrow added Issue Type: Bug Report Component: RPC Results Backend and removed Status: Feedback Needed ✘ labels Feb 28, 2019

auvipy added this to the 4.3.x Maintenance milestone Mar 3, 2019

thedrow mentioned this issue Mar 12, 2019

Release version 4.3 of Celery #5180

Closed

thedrow closed this as completed Apr 4, 2019

thedrow added the Status: Duplicate ✘ label Apr 4, 2019

palvarezcordoba mentioned this issue Apr 16, 2019

Connection reset and timeout getting results because hearbeats are missed #5464

Closed

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Received error "[Errno 104] Connection reset by peer" after 180 seconds when waiting on task result #5358

Received error "[Errno 104] Connection reset by peer" after 180 seconds when waiting on task result #5358

chrahunt commented Feb 23, 2019 •

edited by thedrow

Loading

marcuslind90 commented Feb 25, 2019 •

edited

Loading

thedrow commented Feb 26, 2019

chrahunt commented Feb 28, 2019

thedrow commented Feb 28, 2019

last-partizan commented Mar 15, 2019

thedrow commented Mar 24, 2019

last-partizan commented Mar 24, 2019

chrisportela commented Apr 3, 2019

Shookit commented Apr 4, 2019

last-partizan commented Apr 4, 2019

last-partizan commented Apr 4, 2019

thedrow commented Apr 4, 2019

chrisportela commented Apr 4, 2019

venu13 commented Sep 11, 2020

thedrow commented Sep 13, 2020

Ixiodor commented Sep 28, 2020 •

edited

Loading

thedrow commented Oct 1, 2020

Ixiodor commented Oct 1, 2020

preeth1 commented Mar 5, 2021 •

edited

Loading

Received error "[Errno 104] Connection reset by peer" after 180 seconds when waiting on task result #5358

Received error "[Errno 104] Connection reset by peer" after 180 seconds when waiting on task result #5358

Comments

chrahunt commented Feb 23, 2019 • edited by thedrow Loading

celery issue

Checklist

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Steps to Reproduce

Required Dependencies

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

marcuslind90 commented Feb 25, 2019 • edited Loading

thedrow commented Feb 26, 2019

chrahunt commented Feb 28, 2019

thedrow commented Feb 28, 2019

last-partizan commented Mar 15, 2019

thedrow commented Mar 24, 2019

last-partizan commented Mar 24, 2019

chrisportela commented Apr 3, 2019

Shookit commented Apr 4, 2019

last-partizan commented Apr 4, 2019

last-partizan commented Apr 4, 2019

thedrow commented Apr 4, 2019

chrisportela commented Apr 4, 2019

venu13 commented Sep 11, 2020

thedrow commented Sep 13, 2020

Ixiodor commented Sep 28, 2020 • edited Loading

thedrow commented Oct 1, 2020

Ixiodor commented Oct 1, 2020

preeth1 commented Mar 5, 2021 • edited Loading

chrahunt commented Feb 23, 2019 •

edited by thedrow

Loading

marcuslind90 commented Feb 25, 2019 •

edited

Loading

Ixiodor commented Sep 28, 2020 •

edited

Loading

preeth1 commented Mar 5, 2021 •

edited

Loading