Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError in self.mempool_statuses[hashX] #567

Closed
SomberNight opened this issue Aug 9, 2018 · 2 comments

Comments

@SomberNight
Copy link
Contributor

commented Aug 9, 2018

Mainnet BTC server crashed.
On commit 9b22fd3.

{"log":"ERROR:root:task crashed: \u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.776608124Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.776640198Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.776642389Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.776644276Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 712, in notify\n","stream":"stderr","time":"2018-08-09T11:40:21.776646111Z"}
{"log":"    await self.notify_touched(touched)\n","stream":"stderr","time":"2018-08-09T11:40:21.776648722Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 674, in notify_touched\n","stream":"stderr","time":"2018-08-09T11:40:21.776651054Z"}
{"log":"    old_status = self.mempool_statuses[hashX]\n","stream":"stderr","time":"2018-08-09T11:40:21.776653821Z"}
{"log":"KeyError: b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb'\n","stream":"stderr","time":"2018-08-09T11:40:21.776656049Z"}
{"log":"ERROR:TaskGroup:task crashed: \u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.776789522Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.776800693Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.776803161Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.776805209Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.776806848Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.776808606Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 712, in notify\n","stream":"stderr","time":"2018-08-09T11:40:21.776810244Z"}
{"log":"    await self.notify_touched(touched)\n","stream":"stderr","time":"2018-08-09T11:40:21.776811985Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 674, in notify_touched\n","stream":"stderr","time":"2018-08-09T11:40:21.776813546Z"}
{"log":"    old_status = self.mempool_statuses[hashX]\n","stream":"stderr","time":"2018-08-09T11:40:21.77681521Z"}
{"log":"KeyError: b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb'\n","stream":"stderr","time":"2018-08-09T11:40:21.776816845Z"}
More log lines
{"log":"INFO:ElectrumX:[100856] SSL xx.xx.xx.xx:xx, 709 total\n","stream":"stdout","time":"2018-08-09T11:40:18.559036779Z"}
{"log":"INFO:BlockProcessor:flush #31,972 took 0.3s.  Height 535,928 txs: 333,647,507\n","stream":"stdout","time":"2018-08-09T11:40:19.449413683Z"}
{"log":"INFO:BlockProcessor:processed 1 block in 8.7s\n","stream":"stdout","time":"2018-08-09T11:40:19.449437343Z"}
{"log":"INFO:ElectrumX:[99839] notified of 1 address\n","stream":"stdout","time":"2018-08-09T11:40:19.566962734Z"}
{"log":"INFO:ElectrumX:[93693] notified of 1 address\n","stream":"stdout","time":"2018-08-09T11:40:19.567781746Z"}
{"log":"INFO:ElectrumX:[98248] notified of 2 addresses\n","stream":"stdout","time":"2018-08-09T11:40:19.569654529Z"}
{"log":"INFO:ElectrumX:[91953] notified of 2 addresses\n","stream":"stdout","time":"2018-08-09T11:40:19.57008459Z"}
{"log":"INFO:ElectrumX:[100380] notified of 2 addresses\n","stream":"stdout","time":"2018-08-09T11:40:19.572288621Z"}
{"log":"INFO:ElectrumX:[99792] notified of 4 addresses\n","stream":"stdout","time":"2018-08-09T11:40:19.573445767Z"}
{"log":"INFO:ElectrumX:[91973] notified of 4 addresses\n","stream":"stdout","time":"2018-08-09T11:40:19.573865699Z"}
{"log":"INFO:ElectrumX:[48671] notified of 2 addresses\n","stream":"stdout","time":"2018-08-09T11:40:19.601993121Z"}
{"log":"INFO:MemPool:13 stragglers\n","stream":"stdout","time":"2018-08-09T11:40:19.638086123Z"}
{"log":"INFO:MemPool:13 txs dropped\n","stream":"stdout","time":"2018-08-09T11:40:19.638226565Z"}
{"log":"ERROR:asyncio:SSL handshake failed\n","stream":"stderr","time":"2018-08-09T11:40:19.815686768Z"}
{"log":"protocol: \u003casyncio.sslproto.SSLProtocol object at 0x7fb926680390\u003e\n","stream":"stderr","time":"2018-08-09T11:40:19.815702316Z"}
{"log":"transport: \u003c_SelectorSocketTransport fd=195 read=polling write=\u003cidle, bufsize=0\u003e\u003e\n","stream":"stderr","time":"2018-08-09T11:40:19.815704798Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:19.815706741Z"}
{"log":"  File \"/usr/local/lib/python3.7/asyncio/sslproto.py\", line 625, in _on_handshake_complete\n","stream":"stderr","time":"2018-08-09T11:40:19.815708363Z"}
{"log":"    raise handshake_exc\n","stream":"stderr","time":"2018-08-09T11:40:19.815710087Z"}
{"log":"  File \"/usr/local/lib/python3.7/asyncio/sslproto.py\", line 189, in feed_ssldata\n","stream":"stderr","time":"2018-08-09T11:40:19.815725054Z"}
{"log":"    self._sslobj.do_handshake()\n","stream":"stderr","time":"2018-08-09T11:40:19.815726777Z"}
{"log":"  File \"/usr/local/lib/python3.7/ssl.py\", line 763, in do_handshake\n","stream":"stderr","time":"2018-08-09T11:40:19.815728268Z"}
{"log":"    self._sslobj.do_handshake()\n","stream":"stderr","time":"2018-08-09T11:40:19.815729893Z"}
{"log":"ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1045)\n","stream":"stderr","time":"2018-08-09T11:40:19.815731386Z"}
{"log":"ERROR:asyncio:SSL error in data received\n","stream":"stderr","time":"2018-08-09T11:40:19.815963183Z"}
{"log":"protocol: \u003casyncio.sslproto.SSLProtocol object at 0x7fb926680390\u003e\n","stream":"stderr","time":"2018-08-09T11:40:19.815970152Z"}
{"log":"transport: \u003c_SelectorSocketTransport closing fd=195 read=idle write=\u003cidle, bufsize=0\u003e\u003e\n","stream":"stderr","time":"2018-08-09T11:40:19.81597258Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:19.815974608Z"}
{"log":"  File \"/usr/local/lib/python3.7/asyncio/sslproto.py\", line 526, in data_received\n","stream":"stderr","time":"2018-08-09T11:40:19.815976216Z"}
{"log":"    ssldata, appdata = self._sslpipe.feed_ssldata(data)\n","stream":"stderr","time":"2018-08-09T11:40:19.815977929Z"}
{"log":"  File \"/usr/local/lib/python3.7/asyncio/sslproto.py\", line 189, in feed_ssldata\n","stream":"stderr","time":"2018-08-09T11:40:19.815979474Z"}
{"log":"    self._sslobj.do_handshake()\n","stream":"stderr","time":"2018-08-09T11:40:19.815981169Z"}
{"log":"  File \"/usr/local/lib/python3.7/ssl.py\", line 763, in do_handshake\n","stream":"stderr","time":"2018-08-09T11:40:19.815982686Z"}
{"log":"    self._sslobj.do_handshake()\n","stream":"stderr","time":"2018-08-09T11:40:19.815991014Z"}
{"log":"ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1045)\n","stream":"stderr","time":"2018-08-09T11:40:19.815992721Z"}
{"log":"INFO:ElectrumX:[100857] SSL xx.xx.xx.xx:xx, 706 total\n","stream":"stdout","time":"2018-08-09T11:40:21.731083942Z"}
{"log":"ERROR:root:task crashed: \u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.776608124Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.776640198Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.776642389Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.776644276Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 712, in notify\n","stream":"stderr","time":"2018-08-09T11:40:21.776646111Z"}
{"log":"    await self.notify_touched(touched)\n","stream":"stderr","time":"2018-08-09T11:40:21.776648722Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 674, in notify_touched\n","stream":"stderr","time":"2018-08-09T11:40:21.776651054Z"}
{"log":"    old_status = self.mempool_statuses[hashX]\n","stream":"stderr","time":"2018-08-09T11:40:21.776653821Z"}
{"log":"KeyError: b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb'\n","stream":"stderr","time":"2018-08-09T11:40:21.776656049Z"}
{"log":"ERROR:TaskGroup:task crashed: \u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.776789522Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.776800693Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.776803161Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.776805209Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.776806848Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.776808606Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 712, in notify\n","stream":"stderr","time":"2018-08-09T11:40:21.776810244Z"}
{"log":"    await self.notify_touched(touched)\n","stream":"stderr","time":"2018-08-09T11:40:21.776811985Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 674, in notify_touched\n","stream":"stderr","time":"2018-08-09T11:40:21.776813546Z"}
{"log":"    old_status = self.mempool_statuses[hashX]\n","stream":"stderr","time":"2018-08-09T11:40:21.77681521Z"}
{"log":"KeyError: b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb'\n","stream":"stderr","time":"2018-08-09T11:40:21.776816845Z"}
{"log":"ERROR:root:task crashed: \u003cTask finished coro=\u003cMemPool._refresh_hashes() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:130\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e])\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.777451275Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.777458607Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.777473989Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.777480833Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py\", line 142, in _refresh_hashes\n","stream":"stderr","time":"2018-08-09T11:40:21.777482542Z"}
{"log":"    await self.notifications.on_mempool(touched, height)\n","stream":"stderr","time":"2018-08-09T11:40:21.777484407Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/controller.py\", line 65, in on_mempool\n","stream":"stderr","time":"2018-08-09T11:40:21.77748607Z"}
{"log":"    await self._maybe_notify()\n","stream":"stderr","time":"2018-08-09T11:40:21.777487816Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/controller.py\", line 58, in _maybe_notify\n","stream":"stderr","time":"2018-08-09T11:40:21.777489309Z"}
{"log":"    await notify_func(height, touched)\n","stream":"stderr","time":"2018-08-09T11:40:21.777490968Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 468, in _notify_sessions\n","stream":"stderr","time":"2018-08-09T11:40:21.777492445Z"}
{"log":"    await group.spawn(session.notify(height, touched))\n","stream":"stderr","time":"2018-08-09T11:40:21.777494102Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 264, in __aexit__\n","stream":"stderr","time":"2018-08-09T11:40:21.777495599Z"}
{"log":"    await self.join(wait=self._wait)\n","stream":"stderr","time":"2018-08-09T11:40:21.777497209Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 238, in join\n","stream":"stderr","time":"2018-08-09T11:40:21.777498685Z"}
{"log":"    raise TaskGroupError(bad_tasks)\n","stream":"stderr","time":"2018-08-09T11:40:21.777500317Z"}
{"log":"aiorpcx.curio.TaskGroupError: TaskGroupError(KeyError)\n","stream":"stderr","time":"2018-08-09T11:40:21.777501798Z"}
{"log":"ERROR:TaskGroup:task crashed: \u003cTask finished coro=\u003cMemPool._refresh_hashes() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:130\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e])\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.7777188Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.777725732Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.777728222Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.777730575Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.777732122Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.777733738Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py\", line 142, in _refresh_hashes\n","stream":"stderr","time":"2018-08-09T11:40:21.777735236Z"}
{"log":"    await self.notifications.on_mempool(touched, height)\n","stream":"stderr","time":"2018-08-09T11:40:21.777736921Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/controller.py\", line 65, in on_mempool\n","stream":"stderr","time":"2018-08-09T11:40:21.777738431Z"}
{"log":"    await self._maybe_notify()\n","stream":"stderr","time":"2018-08-09T11:40:21.777740091Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/controller.py\", line 58, in _maybe_notify\n","stream":"stderr","time":"2018-08-09T11:40:21.777741577Z"}
{"log":"    await notify_func(height, touched)\n","stream":"stderr","time":"2018-08-09T11:40:21.777743327Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/session.py\", line 468, in _notify_sessions\n","stream":"stderr","time":"2018-08-09T11:40:21.777744783Z"}
{"log":"    await group.spawn(session.notify(height, touched))\n","stream":"stderr","time":"2018-08-09T11:40:21.777749534Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 264, in __aexit__\n","stream":"stderr","time":"2018-08-09T11:40:21.777751196Z"}
{"log":"    await self.join(wait=self._wait)\n","stream":"stderr","time":"2018-08-09T11:40:21.777752845Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 238, in join\n","stream":"stderr","time":"2018-08-09T11:40:21.777754571Z"}
{"log":"    raise TaskGroupError(bad_tasks)\n","stream":"stderr","time":"2018-08-09T11:40:21.777757181Z"}
{"log":"aiorpcx.curio.TaskGroupError: TaskGroupError(KeyError)\n","stream":"stderr","time":"2018-08-09T11:40:21.777759152Z"}
{"log":"ERROR:root:task crashed: \u003cTask finished coro=\u003cMemPool.keep_synchronized() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:237\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cMemPool._refresh_hashes() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:130\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e])\u003e])\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.778093712Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.778101555Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.778103205Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.778104878Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py\", line 250, in keep_synchronized\n","stream":"stderr","time":"2018-08-09T11:40:21.778106419Z"}
{"log":"    await group.spawn(self._log_stats())\n","stream":"stderr","time":"2018-08-09T11:40:21.77810813Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 264, in __aexit__\n","stream":"stderr","time":"2018-08-09T11:40:21.778109742Z"}
{"log":"    await self.join(wait=self._wait)\n","stream":"stderr","time":"2018-08-09T11:40:21.778111403Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 238, in join\n","stream":"stderr","time":"2018-08-09T11:40:21.778112916Z"}
{"log":"    raise TaskGroupError(bad_tasks)\n","stream":"stderr","time":"2018-08-09T11:40:21.77811457Z"}
{"log":"aiorpcx.curio.TaskGroupError: TaskGroupError(TaskGroupError)\n","stream":"stderr","time":"2018-08-09T11:40:21.778116093Z"}
{"log":"ERROR:TaskGroup:task crashed: \u003cTask finished coro=\u003cMemPool.keep_synchronized() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:237\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cMemPool._refresh_hashes() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:130\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e])\u003e])\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.778330122Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.778337316Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.778339365Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.778341046Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.778342644Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.778347617Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py\", line 250, in keep_synchronized\n","stream":"stderr","time":"2018-08-09T11:40:21.778349248Z"}
{"log":"    await group.spawn(self._log_stats())\n","stream":"stderr","time":"2018-08-09T11:40:21.778351065Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 264, in __aexit__\n","stream":"stderr","time":"2018-08-09T11:40:21.778352593Z"}
{"log":"    await self.join(wait=self._wait)\n","stream":"stderr","time":"2018-08-09T11:40:21.77835472Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 238, in join\n","stream":"stderr","time":"2018-08-09T11:40:21.778357066Z"}
{"log":"    raise TaskGroupError(bad_tasks)\n","stream":"stderr","time":"2018-08-09T11:40:21.77835981Z"}
{"log":"aiorpcx.curio.TaskGroupError: TaskGroupError(TaskGroupError)\n","stream":"stderr","time":"2018-08-09T11:40:21.778362197Z"}
{"log":"INFO:SessionManager:closing down RPC, TCP, SSL listening servers\n","stream":"stdout","time":"2018-08-09T11:40:21.778714504Z"}
{"log":"INFO:BlockProcessor:flushing to DB for a clean shutdown...\n","stream":"stdout","time":"2018-08-09T11:40:21.785439057Z"}
{"log":"ERROR:root:task crashed: \u003cTask finished coro=\u003cController.serve() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/controller.py:79\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cMemPool.keep_synchronized() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:237\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cMemPool._refresh_hashes() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:130\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e])\u003e])\u003e])\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.78580498Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.785817107Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.785819221Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.785821087Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/controller.py\", line 113, in serve\n","stream":"stderr","time":"2018-08-09T11:40:21.785822689Z"}
{"log":"    serve_externally_event.set()\n","stream":"stderr","time":"2018-08-09T11:40:21.785824397Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 264, in __aexit__\n","stream":"stderr","time":"2018-08-09T11:40:21.785825919Z"}
{"log":"    await self.join(wait=self._wait)\n","stream":"stderr","time":"2018-08-09T11:40:21.78582759Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 238, in join\n","stream":"stderr","time":"2018-08-09T11:40:21.785829135Z"}
{"log":"    raise TaskGroupError(bad_tasks)\n","stream":"stderr","time":"2018-08-09T11:40:21.785830786Z"}
{"log":"aiorpcx.curio.TaskGroupError: TaskGroupError(TaskGroupError)\n","stream":"stderr","time":"2018-08-09T11:40:21.785832326Z"}
{"log":"ERROR:TaskGroup:task crashed: \u003cTask finished coro=\u003cController.serve() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/controller.py:79\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cMemPool.keep_synchronized() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:237\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cMemPool._refresh_hashes() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/mempool.py:130\u003e exception=TaskGroupError([\u003cTask finished coro=\u003cElectrumX.notify() done, defined at /usr/local/lib/python3.7/site-packages/electrumx/server/session.py:692\u003e exception=KeyError(b'\\xfa\\xa3\\x9e\\x8f\\xc3\\x1c?\\x8e\\xc6\\x82\\xeb')\u003e])\u003e])\u003e])\u003e\n","stream":"stderr","time":"2018-08-09T11:40:21.786028886Z"}
{"log":"Traceback (most recent call last):\n","stream":"stderr","time":"2018-08-09T11:40:21.786045447Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.786048214Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.7860509Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/util.py\", line 118, in check_task\n","stream":"stderr","time":"2018-08-09T11:40:21.786053275Z"}
{"log":"    task.result()\n","stream":"stderr","time":"2018-08-09T11:40:21.786055712Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/electrumx/server/controller.py\", line 113, in serve\n","stream":"stderr","time":"2018-08-09T11:40:21.786058074Z"}
{"log":"    serve_externally_event.set()\n","stream":"stderr","time":"2018-08-09T11:40:21.786060338Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 264, in __aexit__\n","stream":"stderr","time":"2018-08-09T11:40:21.786061897Z"}
{"log":"    await self.join(wait=self._wait)\n","stream":"stderr","time":"2018-08-09T11:40:21.786063564Z"}
{"log":"  File \"/usr/local/lib/python3.7/site-packages/aiorpcx/curio.py\", line 238, in join\n","stream":"stderr","time":"2018-08-09T11:40:21.786065107Z"}
{"log":"    raise TaskGroupError(bad_tasks)\n","stream":"stderr","time":"2018-08-09T11:40:21.786066761Z"}
{"log":"aiorpcx.curio.TaskGroupError: TaskGroupError(TaskGroupError)\n","stream":"stderr","time":"2018-08-09T11:40:21.786068278Z"}
{"log":"INFO:ElectrumX:[29186] disconnected.  Sent 1,959,642 bytes in 29,455 messages\n","stream":"stdout","time":"2018-08-09T11:40:21.844880084Z"}
{"log":"INFO:ElectrumX:[74648] disconnected.  Sent 1,494,550 bytes in 2,769 messages\n","stream":"stdout","time":"2018-08-09T11:40:21.849947145Z"}
{"log":"INFO:ElectrumX:[86170] disconnected.  Sent 1,322,855 bytes in 2,302 messages\n","stream":"stdout","time":"2018-08-09T11:40:21.878142095Z"}
{"log":"INFO:ElectrumX:[98854] disconnected whilst throttled.  Sent 9,808,212 bytes in 453 messages\n","stream":"stdout","time":"2018-08-09T11:40:21.891803357Z"}
{"log":"INFO:ElectrumX:[86230] disconnected.  Sent 1,207,630 bytes in 10,885 messages\n","stream":"stdout","time":"2018-08-09T11:40:21.902380481Z"}
{"log":"INFO:ElectrumX:[99839] disconnected whilst throttled.  Sent 9,642,335 bytes in 109 messages\n","stream":"stdout","time":"2018-08-09T11:40:21.975097654Z"}
{"log":"INFO:ElectrumX:[100273] disconnected whilst throttled.  Sent 9,639,610 bytes in 94 messages\n","stream":"stdout","time":"2018-08-09T11:40:22.013019565Z"}
{"log":"INFO:ElectrumX:[77109] disconnected.  Sent 1,282,337 bytes in 23,781 messages\n","stream":"stdout","time":"2018-08-09T11:40:22.034476124Z"}
{"log":"ERROR:PeerManager:[bitnode.pw:57081 SSL] marking bad: (bad height 535,940 (ours: 535,928))\n","stream":"stdout","time":"2018-08-09T13:48:12.729016751Z"}
{"log":"ERROR:PeerManager:[bitnode.pw:57081 SSL] marking bad: (bad height 535,940 (ours: 535,928))\n","stream":"stdout","time":"2018-08-09T13:58:13.294950661Z"}
{"log":"ERROR:PeerManager:[bitnode.pw:57081 SSL] marking bad: (bad height 535,940 (ours: 535,928))\n","stream":"stdout","time":"2018-08-09T14:18:13.728271904Z"}
{"log":"INFO:PeerManager:forgetting bad peer: bitnode.pw\n","stream":"stdout","time":"2018-08-09T14:18:13.728513858Z"}
@kyuupichan

This comment has been minimized.

Copy link
Owner

commented Aug 10, 2018

Your logs are really hard to read...

@kyuupichan

This comment has been minimized.

Copy link
Owner

commented Aug 10, 2018

Oh wow there's a race condition there now because of the await and I actually made this properly concurrent. Fix coming up...

shyba added a commit to lbryio/electrumx that referenced this issue Oct 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.