Using -c 2 to start celery then revoke task twice, waiting for new worker create then start the same new task will happen "ValueError: invalid file descriptor 12" #5782

zhenjiazheng · 2019-10-21T07:08:07Z

Checklist

I have read the relevant section in the
contribution guide
on reporting bugs.
I have checked the issues list
for similar or identical bug reports.
I have checked the pull requests list
for existing proposed fixes.
I have checked the commit log
to find out if the bug was already fixed in the master branch.
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
I have verified that the issue exists against the master branch of Celery.
I have included the contents of pip freeze in the issue.
I have included all the versions of all the external dependencies required
to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version
and/or implementation.
I have tried reproducing the issue on more than one message broker and/or
result backend.
I have tried reproducing the issue on more than one version of the message
broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:
software -> celery:4.3.0 (rhubarb) kombu:4.6.3 py:3.7.4
billiard:3.6.0.0 py-amqp:2.5.0
platform -> system:Darwin arch:64bit
kernel version:19.0.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:amqp://guest:**@172.20.25.197:5672/vhost

BROKER_URL: 'amqp://guest:@ip:5672/vhost'
CELERY_RESULT_BACKEND: 'amqp://guest:@ip:5672/vhost'

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown
Minimal Celery Version: N/A or Unknown
Minimal Kombu Version: N/A or Unknown
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aiofiles==0.4.0
aiohttp==3.5.4
aiokafka==0.5.2
alembic==1.0.11
amqp==2.5.0
aniso8601==8.0.0
appnope==0.1.0
APScheduler==3.6.1
asn1crypto==0.24.0
async-timeout==3.0.1
atomicwrites==1.3.0
attrs==19.1.0
autopep8==1.3.4
Babel==2.7.0
backcall==0.1.0
bandit==1.6.2
bcrypt==3.1.7
beautifulsoup4==4.8.0
billiard==3.6.0.0
blinker==1.4
cassandra-driver==3.18.0
celery==4.3.0
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
Click==7.0
ConcurrentLogHandler==0.9.1
crypto==1.4.1
cryptography==2.7
cycler==0.10.0
Cython==0.29.13
decorator==4.4.0
defusedxml==0.6.0
Django==2.2.4
docopt==0.6.2
elasticsearch==7.0.2
enum34==1.1.6
et-xmlfile==1.0.1
filetype==1.0.5
fire==0.2.1
Flask==1.1.1
flask-apidoc==1.2.0
Flask-APScheduler==1.11.0
Flask-BabelEx==0.9.3
Flask-Cors==3.0.8
Flask-Elasticsearch==0.2.5
Flask-Login==0.4.1
Flask-Mail==0.9.1
Flask-Migrate==2.5.2
Flask-Principal==0.4.0
Flask-PyMongo==2.3.0
flask-redis==0.4.0
Flask-RESTful==0.3.7
flask-restplus==0.13.0
Flask-Script==2.0.6
Flask-Security==3.0.0
Flask-SocketIO==4.2.0
Flask-SQLAlchemy==2.4.0
Flask-Uploads==0.2.1
Flask-WTF==0.14.2
flower==0.9.3
future==0.17.1
gevent==1.4.0
gitdb2==2.0.5
GitPython==2.1.13
google==2.0.2
greenlet==0.4.15
gunicorn==19.9.0
hdfs==2.5.8
hdfs3==0.3.1
idna==2.8
importlib-metadata==0.18
ipdb==0.12.2
ipython==7.7.0
ipython-genutils==0.2.0
itsdangerous==1.1.0
jdcal==1.4.1
jedi==0.14.1
jieba==0.39
Jinja2==2.10.1
jira==2.0.0
JPype1==0.6.3
jsonpath-rw==1.4.0
jsonpickle==1.2
jsonschema==3.0.2
kafka-python==1.4.6
kazoo==2.5.0
kiwisolver==1.1.0
kombu==4.6.3
macaca==0.1.0
Mako==1.1.0
MarkupSafe==1.1.1
matplotlib==3.1.1
more-itertools==7.1.0
msgpack==0.6.1
multidict==4.5.2
Naked==0.1.31
nodejs==0.1.1
npm==0.1.1
numpy==1.16.4
oauthlib==3.1.0
opencv-python==4.1.0.25
opencv-python-headless==4.1.0.25
openpyxl==2.6.2
optional-django==0.3.0
packaging==19.0
pandas==0.25.0
paramiko==2.6.0
ParamUnittest==0.2
parso==0.5.1
passlib==1.7.1
pbr==5.4.2
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
pipreqs==0.4.9
pluggy==0.12.0
ply==3.11
prettytable==0.7.2
prompt-toolkit==2.0.9
protobuf==3.9.0
ptyprocess==0.6.0
py==1.8.0
py4j==0.10.8.1
pyasn1==0.4.5
pyasn1-modules==0.2.6
pycodestyle==2.3.1
pycparser==2.19
pycrypto==2.6.1
pyecharts==1.3.1
PyExcelerate==0.8.0
PyExecJS==1.5.1
pyflakes==2.1.1
Pygments==2.4.2
PyJWT==1.7.1
pykafka==2.8.0
pymongo==3.8.0
PyMySQL==0.9.3
PyNaCl==1.3.0
pyparsing==2.4.0
pyRFC3339==1.1
pyrsistent==0.15.4
pyspark==2.4.3
pytesseract==0.2.7
pytest==5.0.1
python-dateutil==2.8.0
python-editor==1.0.4
python-engineio==3.9.2
python-ldap==3.2.0
python-socketio==4.3.0
pytz==2019.2
PyYAML==5.1.2
pyzmq==18.0.2
redis==3.3.5
reportlab==3.5.23
requests==2.22.0
requests-mock==1.6.0
requests-oauthlib==1.2.0
requests-toolbelt==0.9.1
retrying==1.3.3
rsa==4.0
rstr==2.2.6
ruamel.yaml==0.16.0
ruamel.yaml.clib==0.1.0
selenium==3.141.0
shellescape==3.4.1
simplejson==3.16.0
six==1.12.0
smmap2==2.0.5
soupsieve==1.9.2
speaklater==1.3
SQLAlchemy==1.3.6
sqlparse==0.3.0
stevedore==1.30.1
StringGenerator==0.3.3
tabulate==0.8.3
termcolor==1.1.0
TestLink-API-Python-client==0.8.0
threadpool==1.3.2
tornado==5.1.1
traitlets==4.3.2
tzlocal==2.0.0
urllib3==1.25.3
vine==1.3.0
wcwidth==0.1.7
wd==1.0.1
websocket==0.2.1
websocket-client==0.56.0
Werkzeug==0.15.5
wsaccel==0.6.2
WTForms==2.2.1
xeger==0.3.5
xlrd==1.2.0
XlsxWriter==1.1.8
xlutils==2.0.0
xlwt==1.3.0
yarg==0.1.9
yarl==1.3.0
zerorpc==0.6.3
zipp==0.5.2

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

The text was updated successfully, but these errors were encountered:

zhenjiazheng · 2019-10-21T07:09:32Z

[2019-10-21 15:07:21,714: DEBUG/MainProcess] Timer wake-up! Next ETA 0.99959993199991 secs.
[2019-10-21 15:07:22,715: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 15:07:23,715: DEBUG/MainProcess] Timer wake-up! Next ETA 0.8660087639996163 secs.
[2019-10-21 15:07:23,739: INFO/MainProcess] Received task: app.controllers.automation.testset.print_platform[22c634eb-320b-479d-acba-65c7c96831a3]
[2019-10-21 15:07:23,740: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x11efb9680> (args:('app.controllers.automation.testset.print_platform', '22c634eb-320b-479d-acba-65c7c96831a3', {'lang': 'py', 'task': 'app.controllers.automation.testset.print_platform', 'id': '22c634eb-320b-479d-acba-65c7c96831a3', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '22c634eb-320b-479d-acba-65c7c96831a3', 'parent_id': None, 'argsrepr': '(1, 10)', 'kwargsrepr': '{}', 'origin': 'gen40221@cn0014005695.local', 'reply_to': 'a7e6c528-c148-35fd-963c-a224a1e52e0b', 'correlation_id': '22c634eb-320b-479d-acba-65c7c96831a3', 'delivery_info': {'exchange': '', 'routing_key': 'query_platform_1_api', 'priority': None, 'redelivered': False}}, '[[1, 10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-10-21 15:07:23,741: DEBUG/MainProcess] Task accepted: app.controllers.automation.testset.print_platform[22c634eb-320b-479d-acba-65c7c96831a3] pid:40544
[2019-10-21 15:07:23,742: WARNING/ForkPoolWorker-10] Platform_id: 1
[2019-10-21 15:07:23,752: ERROR/ForkPoolWorker-10] Thread 'ResultHandler' crashed: ValueError('invalid file descriptor 12')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 478, in run
return self.body()
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 864, in body
for _ in self._process_result(1.0): # blocking
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 829, in _process_result
ready, task = poll(timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1329, in _poll_result
if self._outqueue._reader.poll(timeout):
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 287, in poll
return self._poll(timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 465, in _poll
r = wait([self], timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 998, in wait
return _poll(object_list, timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 978, in _poll
raise ValueError('invalid file descriptor %i' % fd)
ValueError: invalid file descriptor 12
[2019-10-21 15:07:24,171: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 15:07:24,540: ERROR/MainProcess] Process 'ForkPoolWorker-10' pid:40544 exited with 'exitcode 1'
[2019-10-21 15:07:24,553: DEBUG/ForkPoolWorker-12] Closed channel #1
[2019-10-21 15:07:24,587: DEBUG/MainProcess] Timer wake-up! Next ETA 0.1279656229999091 secs.

zhenjiazheng · 2019-10-21T09:08:13Z

(venv) zhengzhenjia@cn0014005695 backend-api % celery worker -A api.celery -l debug -c 2 -n localhost -Q query_platform_1_api
/Users/zhengzhenjia/Platform/backend-api/api.py:8: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See gevent/gevent#1016. Modules that had direct imports (NOT patched): ['aiohttp.connector (/usr/local/lib/python3.7/site-packages/aiohttp/connector.py)', 'aiohttp.web_runner (/usr/local/lib/python3.7/site-packages/aiohttp/web_runner.py)', 'urllib3.util.ssl_ (/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py)', 'aiohttp.worker (/usr/local/lib/python3.7/site-packages/aiohttp/worker.py)', 'aiohttp.client_reqrep (/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py)', 'aiohttp.client_exceptions (/usr/local/lib/python3.7/site-packages/aiohttp/client_exceptions.py)', 'urllib3.util (/usr/local/lib/python3.7/site-packages/urllib3/util/init.py)', 'aiohttp.client (/usr/local/lib/python3.7/site-packages/aiohttp/client.py)', 'aiohttp.web (/usr/local/lib/python3.7/site-packages/aiohttp/web.py)'].
monkey.patch_all()
[2019-10-21 17:05:35,295: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-10-21 17:05:35,298: DEBUG/MainProcess] | Worker: Building graph...
[2019-10-21 17:05:35,298: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, Beat, StateDB, Consumer}
[2019-10-21 17:05:35,308: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-10-21 17:05:35,309: DEBUG/MainProcess] | Consumer: Building graph...
[2019-10-21 17:05:35,363: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Mingle, Tasks, Control, Heart, Gossip, event loop}
/usr/local/lib/python3.7/site-packages/celery/backends/amqp.py:67: CPendingDeprecationWarning:
The AMQP result backend is scheduled for deprecation in version 4.0 and removal in version v5.0. Please use RPC backend or a persistent backend.

alternative='Please use RPC backend or a persistent backend.')

-------------- celery@localhost v4.3.0 (rhubarb)
---- **** -----
--- * *** * -- Darwin-19.0.0-x86_64-i386-64bit 2019-10-21 17:05:35
-- * - **** ---

** ---------- [config]
** ---------- .> app: app:0x11dbb5250
** ---------- .> transport: amqp://guest:**@172.20.25.197:5672/vhost
** ---------- .> results: amqp://
*** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> query_platform_1_api exchange=query_platform_1_api(direct) key=query_platform_1_api

[tasks]
. app.controllers.automation.testset.print_platform
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. coll_setup_teardown
. generate_and_send_collection_report
. run_api_collection
. run_api_collection_all_task
. run_api_setup
. run_api_teardown
. run_test_set

[2019-10-21 17:05:35,386: DEBUG/MainProcess] | Worker: Starting Pool
[2019-10-21 17:05:35,425: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:35,425: DEBUG/MainProcess] | Worker: Starting Consumer
[2019-10-21 17:05:35,426: DEBUG/MainProcess] | Consumer: Starting Connection
[2019-10-21 17:05:35,723: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@localhost', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See https://www.rabbitmq.com/', 'platform': 'Erlang/OTP 22.1.3', 'product': 'RabbitMQ', 'version': '3.8.0'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-10-21 17:05:35,800: INFO/MainProcess] Connected to amqp://guest:**@172.20.25.197:5672/vhost
[2019-10-21 17:05:35,800: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:35,801: DEBUG/MainProcess] | Consumer: Starting Events
[2019-10-21 17:05:35,815: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@localhost', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See https://www.rabbitmq.com/', 'platform': 'Erlang/OTP 22.1.3', 'product': 'RabbitMQ', 'version': '3.8.0'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-10-21 17:05:35,826: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:35,826: DEBUG/MainProcess] | Consumer: Starting Mingle
[2019-10-21 17:05:35,826: INFO/MainProcess] mingle: searching for neighbors
[2019-10-21 17:05:35,826: DEBUG/MainProcess] using channel_id: 1
[2019-10-21 17:05:35,833: DEBUG/MainProcess] Channel open
[2019-10-21 17:05:35,866: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@localhost', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See https://www.rabbitmq.com/', 'platform': 'Erlang/OTP 22.1.3', 'product': 'RabbitMQ', 'version': '3.8.0'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-10-21 17:05:35,880: DEBUG/MainProcess] using channel_id: 1
[2019-10-21 17:05:35,928: DEBUG/MainProcess] Channel open
[2019-10-21 17:05:37,739: INFO/MainProcess] mingle: all alone
[2019-10-21 17:05:37,739: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:37,739: DEBUG/MainProcess] | Consumer: Starting Tasks
[2019-10-21 17:05:39,821: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:39,822: DEBUG/MainProcess] | Consumer: Starting Control
[2019-10-21 17:05:39,822: DEBUG/MainProcess] using channel_id: 2
[2019-10-21 17:05:40,128: DEBUG/MainProcess] Channel open
[2019-10-21 17:05:40,144: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:40,145: DEBUG/MainProcess] | Consumer: Starting Heart
[2019-10-21 17:05:40,145: DEBUG/MainProcess] using channel_id: 1
[2019-10-21 17:05:40,149: DEBUG/MainProcess] Channel open
[2019-10-21 17:05:40,152: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:40,152: DEBUG/MainProcess] | Consumer: Starting Gossip
[2019-10-21 17:05:40,153: DEBUG/MainProcess] using channel_id: 3
[2019-10-21 17:05:40,153: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:40,157: DEBUG/MainProcess] Channel open
[2019-10-21 17:05:40,174: DEBUG/MainProcess] ^-- substep ok
[2019-10-21 17:05:40,174: DEBUG/MainProcess] | Consumer: Starting event loop
[2019-10-21 17:05:40,177: INFO/MainProcess] celery@localhost ready.
[2019-10-21 17:05:40,177: DEBUG/MainProcess] basic.qos: prefetch_count->8
[2019-10-21 17:05:41,158: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9944045720000005 secs.
[2019-10-21 17:05:42,156: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:43,156: DEBUG/MainProcess] Timer wake-up! Next ETA 0.999647714 secs.
[2019-10-21 17:05:43,301: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:05:43,302: INFO/MainProcess] Events of group {task} enabled by remote.
[2019-10-21 17:05:44,160: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9975191090000006 secs.
[2019-10-21 17:05:45,162: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9978837160000005 secs.
[2019-10-21 17:05:46,165: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:47,126: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:05:47,166: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9989655519999996 secs.
[2019-10-21 17:05:48,171: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:49,171: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9909507290000015 secs.
[2019-10-21 17:05:50,163: DEBUG/MainProcess] Timer wake-up! Next ETA 0.007922003000000899 secs.
[2019-10-21 17:05:50,173: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:51,175: DEBUG/MainProcess] Timer wake-up! Next ETA 0.997755042999998 secs.
[2019-10-21 17:05:52,061: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:05:52,174: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:53,178: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9955325370000025 secs.
[2019-10-21 17:05:54,179: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9843383330000002 secs.
[2019-10-21 17:05:55,166: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:56,170: DEBUG/MainProcess] Timer wake-up! Next ETA 0.008607946000001476 secs.
[2019-10-21 17:05:56,180: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:57,069: INFO/MainProcess] Received task: app.controllers.automation.testset.print_platform[e0fe9658-6c4e-4526-a8f1-66ff6b700696]
[2019-10-21 17:05:57,070: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x11dbec200> (args:('app.controllers.automation.testset.print_platform', 'e0fe9658-6c4e-4526-a8f1-66ff6b700696', {'lang': 'py', 'task': 'app.controllers.automation.testset.print_platform', 'id': 'e0fe9658-6c4e-4526-a8f1-66ff6b700696', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e0fe9658-6c4e-4526-a8f1-66ff6b700696', 'parent_id': None, 'argsrepr': '(1, 10)', 'kwargsrepr': '{}', 'origin': 'gen41567@cn0014005695.local', 'reply_to': '70f3c1b7-1eed-3865-9904-ba6e87922e64', 'correlation_id': 'e0fe9658-6c4e-4526-a8f1-66ff6b700696', 'delivery_info': {'exchange': '', 'routing_key': 'query_platform_1_api', 'priority': None, 'redelivered': False}}, '[[1, 10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-10-21 17:05:57,072: DEBUG/MainProcess] Task accepted: app.controllers.automation.testset.print_platform[e0fe9658-6c4e-4526-a8f1-66ff6b700696] pid:43426
[2019-10-21 17:05:57,072: WARNING/ForkPoolWorker-1] Platform_id: 1
[2019-10-21 17:05:57,183: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9976037319999982 secs.
[2019-10-21 17:05:57,369: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:05:58,184: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:05:59,185: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9812495550000015 secs.
[2019-10-21 17:06:00,170: DEBUG/MainProcess] Timer wake-up! Next ETA 0.013864099999999269 secs.
[2019-10-21 17:06:00,185: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:01,185: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9997381930000024 secs.
[2019-10-21 17:06:02,186: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:02,230: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:03,186: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9996455399999995 secs.
[2019-10-21 17:06:03,538: DEBUG/MainProcess] pidbox received method revoke(task_id='e0fe9658-6c4e-4526-a8f1-66ff6b700696', terminate=True, signal='SIGTERM') [reply_to:None ticket:None]
[2019-10-21 17:06:03,539: INFO/MainProcess] Terminating e0fe9658-6c4e-4526-a8f1-66ff6b700696 (Signals.SIGTERM)
[2019-10-21 17:06:03,736: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@localhost', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See https://www.rabbitmq.com/', 'platform': 'Erlang/OTP 22.1.3', 'product': 'RabbitMQ', 'version': '3.8.0'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-10-21 17:06:03,743: DEBUG/ForkPoolWorker-1] using channel_id: 1
[2019-10-21 17:06:03,748: DEBUG/ForkPoolWorker-1] Channel open
[2019-10-21 17:06:03,759: WARNING/ForkPoolWorker-1] task done: True
[2019-10-21 17:06:03,760: INFO/ForkPoolWorker-1] Task app.controllers.automation.testset.print_platform[e0fe9658-6c4e-4526-a8f1-66ff6b700696] succeeded in 6.687860780999998s: True
[2019-10-21 17:06:04,187: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9826785869999988 secs.
[2019-10-21 17:06:05,171: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:06,174: DEBUG/MainProcess] Timer wake-up! Next ETA 0.013081381999995756 secs.
[2019-10-21 17:06:06,191: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:07,195: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9965400670000051 secs.
[2019-10-21 17:06:07,312: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:08,197: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:09,199: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9713497069999946 secs.
[2019-10-21 17:06:10,171: DEBUG/MainProcess] Timer wake-up! Next ETA 0.026057154000000082 secs.
[2019-10-21 17:06:10,202: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:11,203: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9987305089999978 secs.
[2019-10-21 17:06:12,203: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:12,588: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:13,206: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9967872439999965 secs.
[2019-10-21 17:06:14,206: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9654150329999993 secs.
[2019-10-21 17:06:15,171: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:16,172: DEBUG/MainProcess] Timer wake-up! Next ETA 0.033868750999999975 secs.
[2019-10-21 17:06:16,207: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:16,364: INFO/MainProcess] Received task: app.controllers.automation.testset.print_platform[bbe5d96f-662f-46c4-bfa7-26f3db1afedc]
[2019-10-21 17:06:16,365: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x11dbec200> (args:('app.controllers.automation.testset.print_platform', 'bbe5d96f-662f-46c4-bfa7-26f3db1afedc', {'lang': 'py', 'task': 'app.controllers.automation.testset.print_platform', 'id': 'bbe5d96f-662f-46c4-bfa7-26f3db1afedc', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'bbe5d96f-662f-46c4-bfa7-26f3db1afedc', 'parent_id': None, 'argsrepr': '(1, 10)', 'kwargsrepr': '{}', 'origin': 'gen41567@cn0014005695.local', 'reply_to': '70f3c1b7-1eed-3865-9904-ba6e87922e64', 'correlation_id': 'bbe5d96f-662f-46c4-bfa7-26f3db1afedc', 'delivery_info': {'exchange': '', 'routing_key': 'query_platform_1_api', 'priority': None, 'redelivered': False}}, '[[1, 10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-10-21 17:06:16,367: DEBUG/MainProcess] Task accepted: app.controllers.automation.testset.print_platform[bbe5d96f-662f-46c4-bfa7-26f3db1afedc] pid:43427
[2019-10-21 17:06:16,368: WARNING/ForkPoolWorker-2] Platform_id: 1
[2019-10-21 17:06:17,207: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9994638140000021 secs.
[2019-10-21 17:06:17,728: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:18,212: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:19,217: DEBUG/MainProcess] Timer wake-up! Next ETA 0.954573418999999 secs.
[2019-10-21 17:06:20,173: DEBUG/MainProcess] Timer wake-up! Next ETA 0.03894081700000385 secs.
[2019-10-21 17:06:20,217: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:21,218: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9994084739999991 secs.
[2019-10-21 17:06:21,617: DEBUG/MainProcess] pidbox received method revoke(task_id='bbe5d96f-662f-46c4-bfa7-26f3db1afedc', terminate=True, signal='SIGTERM') [reply_to:None ticket:None]
[2019-10-21 17:06:21,618: INFO/MainProcess] Terminating bbe5d96f-662f-46c4-bfa7-26f3db1afedc (Signals.SIGTERM)
[2019-10-21 17:06:22,109: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:22,218: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:22,416: DEBUG/ForkPoolWorker-2] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@localhost', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See https://www.rabbitmq.com/', 'platform': 'Erlang/OTP 22.1.3', 'product': 'RabbitMQ', 'version': '3.8.0'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-10-21 17:06:22,518: DEBUG/ForkPoolWorker-2] using channel_id: 1
[2019-10-21 17:06:22,523: DEBUG/ForkPoolWorker-2] Channel open
[2019-10-21 17:06:22,535: WARNING/ForkPoolWorker-2] task done: True
[2019-10-21 17:06:22,535: INFO/ForkPoolWorker-2] Task app.controllers.automation.testset.print_platform[bbe5d96f-662f-46c4-bfa7-26f3db1afedc] succeeded in 6.167662094000001s: True
[2019-10-21 17:06:23,222: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9964623119999985 secs.
[2019-10-21 17:06:24,219: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9541233220000009 secs.
[2019-10-21 17:06:25,179: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:25,511: INFO/MainProcess] Received task: app.controllers.automation.testset.print_platform[3049f236-df17-415f-814c-41772f15abdf]
[2019-10-21 17:06:25,512: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x11dbec200> (args:('app.controllers.automation.testset.print_platform', '3049f236-df17-415f-814c-41772f15abdf', {'lang': 'py', 'task': 'app.controllers.automation.testset.print_platform', 'id': '3049f236-df17-415f-814c-41772f15abdf', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '3049f236-df17-415f-814c-41772f15abdf', 'parent_id': None, 'argsrepr': '(1, 10)', 'kwargsrepr': '{}', 'origin': 'gen41567@cn0014005695.local', 'reply_to': '70f3c1b7-1eed-3865-9904-ba6e87922e64', 'correlation_id': '3049f236-df17-415f-814c-41772f15abdf', 'delivery_info': {'exchange': '', 'routing_key': 'query_platform_1_api', 'priority': None, 'redelivered': False}}, '[[1, 10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-10-21 17:06:25,514: DEBUG/MainProcess] Task accepted: app.controllers.automation.testset.print_platform[3049f236-df17-415f-814c-41772f15abdf] pid:43426
[2019-10-21 17:06:25,513: WARNING/ForkPoolWorker-1] Platform_id: 1
[2019-10-21 17:06:26,179: DEBUG/MainProcess] Timer wake-up! Next ETA 0.03972443400000003 secs.
[2019-10-21 17:06:26,220: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:27,220: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9997170649999987 secs.
[2019-10-21 17:06:27,314: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:28,221: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:29,223: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9553012789999968 secs.
[2019-10-21 17:06:29,233: DEBUG/MainProcess] pidbox received method revoke(task_id='3049f236-df17-415f-814c-41772f15abdf', terminate=True, signal='SIGTERM') [reply_to:None ticket:None]
[2019-10-21 17:06:29,233: INFO/MainProcess] Terminating 3049f236-df17-415f-814c-41772f15abdf (Signals.SIGTERM)
[2019-10-21 17:06:29,692: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@localhost', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See https://www.rabbitmq.com/', 'platform': 'Erlang/OTP 22.1.3', 'product': 'RabbitMQ', 'version': '3.8.0'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-10-21 17:06:29,698: DEBUG/MainProcess] using channel_id: 1
[2019-10-21 17:06:29,701: DEBUG/MainProcess] Channel open
[2019-10-21 17:06:29,712: ERROR/MainProcess] Task handler raised error: Terminated(15)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1769, in _set_terminated
raise Terminated(-(signum or 0))
billiard.exceptions.Terminated: 15
[2019-10-21 17:06:29,722: DEBUG/ForkPoolWorker-3] Closed channel #1
[2019-10-21 17:06:29,723: DEBUG/ForkPoolWorker-3] Closed channel #1
[2019-10-21 17:06:30,184: DEBUG/MainProcess] Timer wake-up! Next ETA 0.03720110600000481 secs.
[2019-10-21 17:06:30,229: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:31,229: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9996820059999934 secs.
[2019-10-21 17:06:32,233: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:32,621: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:33,238: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9956817769999944 secs.
[2019-10-21 17:06:34,238: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9460516470000044 secs.
[2019-10-21 17:06:35,185: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:36,185: DEBUG/MainProcess] Timer wake-up! Next ETA 0.05293228200000044 secs.
[2019-10-21 17:06:36,240: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:37,055: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:37,246: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9947319669999928 secs.
[2019-10-21 17:06:38,245: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:39,246: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9381306079999945 secs.
[2019-10-21 17:06:40,185: DEBUG/MainProcess] Timer wake-up! Next ETA 0.06033249399999363 secs.
[2019-10-21 17:06:40,249: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:41,250: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9991482190000056 secs.
[2019-10-21 17:06:42,252: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:42,696: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:43,252: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9995253149999996 secs.
[2019-10-21 17:06:44,253: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9316984520000062 secs.
[2019-10-21 17:06:45,189: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:46,192: DEBUG/MainProcess] Timer wake-up! Next ETA 0.061758539999999584 secs.
[2019-10-21 17:06:46,256: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:47,261: DEBUG/MainProcess] Timer wake-up! Next ETA 0.995180359999992 secs.
[2019-10-21 17:06:47,521: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:47,522: INFO/MainProcess] Received task: app.controllers.automation.testset.print_platform[1a769cad-f25d-4efd-8556-38008868a715]
[2019-10-21 17:06:47,523: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x11dbec200> (args:('app.controllers.automation.testset.print_platform', '1a769cad-f25d-4efd-8556-38008868a715', {'lang': 'py', 'task': 'app.controllers.automation.testset.print_platform', 'id': '1a769cad-f25d-4efd-8556-38008868a715', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '1a769cad-f25d-4efd-8556-38008868a715', 'parent_id': None, 'argsrepr': '(1, 10)', 'kwargsrepr': '{}', 'origin': 'gen41567@cn0014005695.local', 'reply_to': '70f3c1b7-1eed-3865-9904-ba6e87922e64', 'correlation_id': '1a769cad-f25d-4efd-8556-38008868a715', 'delivery_info': {'exchange': '', 'routing_key': 'query_platform_1_api', 'priority': None, 'redelivered': False}}, '[[1, 10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-10-21 17:06:47,524: WARNING/ForkPoolWorker-2] Platform_id: 1
[2019-10-21 17:06:47,524: DEBUG/MainProcess] Task accepted: app.controllers.automation.testset.print_platform[1a769cad-f25d-4efd-8556-38008868a715] pid:43427
[2019-10-21 17:06:48,258: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:49,263: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9267479110000068 secs.
[2019-10-21 17:06:50,191: DEBUG/MainProcess] Timer wake-up! Next ETA 0.06641730100000132 secs.
[2019-10-21 17:06:50,263: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:50,822: DEBUG/MainProcess] pidbox received method revoke(task_id='1a769cad-f25d-4efd-8556-38008868a715', terminate=True, signal='SIGTERM') [reply_to:None ticket:None]
[2019-10-21 17:06:50,822: INFO/MainProcess] Terminating 1a769cad-f25d-4efd-8556-38008868a715 (Signals.SIGTERM)
[2019-10-21 17:06:51,173: DEBUG/MainProcess] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@localhost', 'copyright': 'Copyright (C) 2007-2019 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See https://www.rabbitmq.com/', 'platform': 'Erlang/OTP 22.1.3', 'product': 'RabbitMQ', 'version': '3.8.0'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
[2019-10-21 17:06:51,180: DEBUG/MainProcess] using channel_id: 1
[2019-10-21 17:06:51,184: DEBUG/MainProcess] Channel open
[2019-10-21 17:06:51,268: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9953308289999967 secs.
[2019-10-21 17:06:51,441: ERROR/MainProcess] Task handler raised error: Terminated(15)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1769, in _set_terminated
raise Terminated(-(signum or 0))
billiard.exceptions.Terminated: 15
[2019-10-21 17:06:51,454: DEBUG/ForkPoolWorker-4] Closed channel #1
[2019-10-21 17:06:52,063: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:52,267: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:53,268: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9989506820000003 secs.
[2019-10-21 17:06:54,269: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9227720749999975 secs.
[2019-10-21 17:06:55,197: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:55,947: INFO/MainProcess] Received task: app.controllers.automation.testset.print_platform[6193b159-4295-4151-b886-7a664e76200f]
[2019-10-21 17:06:55,948: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x11dbec200> (args:('app.controllers.automation.testset.print_platform', '6193b159-4295-4151-b886-7a664e76200f', {'lang': 'py', 'task': 'app.controllers.automation.testset.print_platform', 'id': '6193b159-4295-4151-b886-7a664e76200f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '6193b159-4295-4151-b886-7a664e76200f', 'parent_id': None, 'argsrepr': '(1, 10)', 'kwargsrepr': '{}', 'origin': 'gen41567@cn0014005695.local', 'reply_to': '70f3c1b7-1eed-3865-9904-ba6e87922e64', 'correlation_id': '6193b159-4295-4151-b886-7a664e76200f', 'delivery_info': {'exchange': '', 'routing_key': 'query_platform_1_api', 'priority': None, 'redelivered': False}}, '[[1, 10], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2019-10-21 17:06:55,951: DEBUG/MainProcess] Task accepted: app.controllers.automation.testset.print_platform[6193b159-4295-4151-b886-7a664e76200f] pid:43450
[2019-10-21 17:06:55,951: WARNING/ForkPoolWorker-3] Platform_id: 1
[2019-10-21 17:06:55,954: DEBUG/ForkPoolWorker-3] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:55,955: ERROR/ForkPoolWorker-3] Thread 'ResultHandler' crashed: ValueError('invalid file descriptor 12')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 504, in run
return self.body()
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 899, in body
for _ in self._process_result(1.0): # blocking
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 864, in _process_result
ready, task = poll(timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1370, in _poll_result
if self._outqueue._reader.poll(timeout):
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 285, in poll
return self._poll(timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 463, in _poll
r = wait([self], timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 996, in wait
return _poll(object_list, timeout)
File "/usr/local/lib/python3.7/site-packages/billiard/connection.py", line 976, in _poll
raise ValueError('invalid file descriptor %i' % fd)
ValueError: invalid file descriptor 12
[2019-10-21 17:06:56,199: DEBUG/MainProcess] Timer wake-up! Next ETA 0.0699374769999963 secs.
[2019-10-21 17:06:56,256: ERROR/MainProcess] Process 'ForkPoolWorker-3' pid:43450 exited with 'exitcode 1'
[2019-10-21 17:06:56,273: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:57,055: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:06:57,274: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9990880069999974 secs.
[2019-10-21 17:06:58,274: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:06:59,279: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9173235500000061 secs.
[2019-10-21 17:07:00,202: DEBUG/MainProcess] Timer wake-up! Next ETA 0.072229544999999 secs.
[2019-10-21 17:07:00,277: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.
[2019-10-21 17:07:01,282: DEBUG/MainProcess] Timer wake-up! Next ETA 0.9952956289999975 secs.
[2019-10-21 17:07:02,055: DEBUG/MainProcess] pidbox received method enable_events() [reply_to:None ticket:None]
[2019-10-21 17:07:02,281: DEBUG/MainProcess] Timer wake-up! Next ETA 1.0 secs.

auvipy · 2019-10-29T19:59:52Z

can you post your problem in celery-users mailing list?

RonRothman · 2019-11-15T18:34:02Z

Quick question: Is the celery-users mailing list the Source of Truth for Celery bug reports and their solutions? Or is it here on Github?

We're seeing this issue as well, and would like to follow the conversation. Thanks.

auvipy · 2019-11-16T02:12:14Z

Quick question: Is the celery-users mailing list the Source of Truth for Celery bug reports and their solutions? Or is it here on Github?

We're seeing this issue as well, and would like to follow the conversation. Thanks.

For bug report, GitHub is here, but many users mistakenly ask their questions here instead of the mailing list. the suggestion is for making sure the issue is really issue and not a question at all. and since more then on of you can verify this as a celery bug, we can continue discussin here.

wjkaola123 · 2020-01-06T02:41:29Z

I have got the same problem like "ValueError: invalid file descriptor 16" when the celery task running a few days.

auvipy · 2020-05-16T05:23:44Z

update on the latest master?

auvipy added the Status: Feedback Needed ✘ label Jul 28, 2020

auvipy added the Status: Needs Verification ✘ label Oct 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using -c 2 to start celery then revoke task twice, waiting for new worker create then start the same new task will happen "ValueError: invalid file descriptor 12" #5782

Using -c 2 to start celery then revoke task twice, waiting for new worker create then start the same new task will happen "ValueError: invalid file descriptor 12" #5782

zhenjiazheng commented Oct 21, 2019 •

edited by sync-by-unito bot

zhenjiazheng commented Oct 21, 2019

zhenjiazheng commented Oct 21, 2019

auvipy commented Oct 29, 2019

RonRothman commented Nov 15, 2019 •

edited

auvipy commented Nov 16, 2019

wjkaola123 commented Jan 6, 2020

auvipy commented May 16, 2020

Using -c 2 to start celery then revoke task twice, waiting for new worker create then start the same new task will happen "ValueError: invalid file descriptor 12" #5782

Using -c 2 to start celery then revoke task twice, waiting for new worker create then start the same new task will happen "ValueError: invalid file descriptor 12" #5782

Comments

zhenjiazheng commented Oct 21, 2019 • edited by sync-by-unito bot

Checklist

Mandatory Debugging Information

Optional Debugging Information

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Steps to Reproduce

Required Dependencies

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

zhenjiazheng commented Oct 21, 2019

zhenjiazheng commented Oct 21, 2019

auvipy commented Oct 29, 2019

RonRothman commented Nov 15, 2019 • edited

auvipy commented Nov 16, 2019

wjkaola123 commented Jan 6, 2020

auvipy commented May 16, 2020

zhenjiazheng commented Oct 21, 2019 •

edited by sync-by-unito bot

RonRothman commented Nov 15, 2019 •

edited