db: db_flushed_event: fix thread-safety for asyncio.Event()#376
Merged
Conversation
had an exception that did not get logged anywhere...
---
before:
```
DEBUG:Daemon:new height: 8887
DEBUG:MemPool:waiting for DB to sync
INFO:Prefetcher:new block height 8,887 hash 6496d4c0774e2d804167c491d97855ce7988d459994e288628bae1b4aac3cb94
DEBUG:BlockProcessor:new height: 8887
INFO:DB:flush took 0.0s. Height 8,887 txs: 9,519 (+1)
INFO:Prefetcher:cancelled; prefetcher stopping
INFO:SessionManager:stop listening on servers, so no new sessions can be created
INFO:SessionManager:closing down server for rpc://127.0.0.1:8000
INFO:SessionManager:closing down server for tcp://127.0.0.1:51001
INFO:SessionManager:closing down server for ssl://127.0.0.1:51002
INFO:SessionManager:closing down server for tcp://[::1]:51001
INFO:SessionManager:closing down server for ssl://[::1]:51002
INFO:SessionManager:closing 0 active sessions
INFO:SessionManager:waiting for all server's resources to close
<hang>
```
after:
```
DEBUG:Daemon:new height: 8886
INFO:Prefetcher:new block height 8,886 hash 4662fcd5308e94cbbe82754d94a59123942c4009d62e88a653d5dabca48423dd
DEBUG:BlockProcessor:new height: 8886
INFO:DB:flush took 0.0s. Height 8,886 txs: 9,518 (+1)
ERROR:BlockProcessor:_process_prefetched_blocks got exception: Non-thread-safe operation invoked on an event loop other than the current one
INFO:BlockProcessor:_process_prefetched_blocks exiting
INFO:Prefetcher:cancelled; prefetcher stopping
ERROR:BlockProcessor:fetch_and_process_blocks taskgroup died.
Traceback (most recent call last):
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 884, in fetch_and_process_blocks
async with OldTaskGroup() as group:
~~~~~~~~~~~~^^
File "/home/user/wspace/aiorpcX/aiorpcx/curio.py", line 304, in __aexit__
await self.join()
File "/home/user/wspace/electrumx/./src/electrumx/lib/util.py", line 375, in join
task.result()
~~~~~~~~~~~^^
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 833, in _process_prefetched_blocks
await self.check_and_advance_blocks(blocks)
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 264, in check_and_advance_blocks
await self._maybe_flush()
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 415, in _maybe_flush
await self.flush(True)
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 409, in flush
await self.run_in_thread_with_lock(flush)
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 242, in run_in_thread_with_lock
return await asyncio.shield(run_in_thread_locked())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 241, in run_in_thread_locked
return await run_in_thread(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/wspace/aiorpcX/aiorpcx/curio.py", line 57, in run_in_thread
return await get_event_loop().run_in_executor(None, func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/concurrent/futures/thread.py", line 86, in run
result = ctx.run(self.task)
File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/concurrent/futures/thread.py", line 73, in run
return fn(*args, **kwargs)
File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 404, in flush
self.db.flush_dbs(
~~~~~~~~~~~~~~~~~^
flush_data=self.flush_data(),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
flush_utxos=flush_utxos,
^^^^^^^^^^^^^^^^^^^^^^^^
estimate_txs_remaining=self.estimate_txs_remaining,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/user/wspace/electrumx/./src/electrumx/server/db.py", line 319, in flush_dbs
self.db_flushed_event.set() #
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/asyncio/locks.py", line 192, in set
fut.set_result(True)
~~~~~~~~~~~~~~^^^^^^
File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/asyncio/base_events.py", line 829, in call_soon
self._check_thread()
~~~~~~~~~~~~~~~~~~^^
File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/asyncio/base_events.py", line 866, in _check_thread
raise RuntimeError(
"Non-thread-safe operation invoked on an event loop other "
"than the current one")
RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one
INFO:BlockProcessor:fetch_and_process_blocks taskgroup stopped.
INFO:SessionManager:stop listening on servers, so no new sessions can be created
INFO:SessionManager:closing down server for rpc://127.0.0.1:8000
INFO:SessionManager:closing down server for tcp://127.0.0.1:51001
INFO:SessionManager:closing down server for ssl://127.0.0.1:51002
INFO:SessionManager:closing down server for tcp://[::1]:51001
INFO:SessionManager:closing down server for ssl://[::1]:51002
INFO:SessionManager:closing 0 active sessions
INFO:SessionManager:waiting for all server's resources to close
INFO:SessionManager:all servers closed
<hang>
```
regression from spesmilo@34d7111 Was visible when running with: export PYTHONASYNCIODEBUG=1 export PYTHONDEVMODE=1 ``` DEBUG:Daemon:new height: 8886 INFO:Prefetcher:new block height 8,886 hash 4662fcd5308e94cbbe82754d94a59123942c4009d62e88a653d5dabca48423dd DEBUG:BlockProcessor:new height: 8886 INFO:DB:flush took 0.0s. Height 8,886 txs: 9,518 (+1) ERROR:BlockProcessor:_process_prefetched_blocks got exception: Non-thread-safe operation invoked on an event loop other than the current one INFO:BlockProcessor:_process_prefetched_blocks exiting INFO:Prefetcher:cancelled; prefetcher stopping ERROR:BlockProcessor:fetch_and_process_blocks taskgroup died. Traceback (most recent call last): File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 884, in fetch_and_process_blocks async with OldTaskGroup() as group: ~~~~~~~~~~~~^^ File "/home/user/wspace/aiorpcX/aiorpcx/curio.py", line 304, in __aexit__ await self.join() File "/home/user/wspace/electrumx/./src/electrumx/lib/util.py", line 375, in join task.result() ~~~~~~~~~~~^^ File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 833, in _process_prefetched_blocks await self.check_and_advance_blocks(blocks) File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 264, in check_and_advance_blocks await self._maybe_flush() File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 415, in _maybe_flush await self.flush(True) File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 409, in flush await self.run_in_thread_with_lock(flush) File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 242, in run_in_thread_with_lock return await asyncio.shield(run_in_thread_locked()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 241, in run_in_thread_locked return await run_in_thread(func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/wspace/aiorpcX/aiorpcx/curio.py", line 57, in run_in_thread return await get_event_loop().run_in_executor(None, func, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/concurrent/futures/thread.py", line 86, in run result = ctx.run(self.task) File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/concurrent/futures/thread.py", line 73, in run return fn(*args, **kwargs) File "/home/user/wspace/electrumx/./src/electrumx/server/block_processor.py", line 404, in flush self.db.flush_dbs( ~~~~~~~~~~~~~~~~~^ flush_data=self.flush_data(), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ flush_utxos=flush_utxos, ^^^^^^^^^^^^^^^^^^^^^^^^ estimate_txs_remaining=self.estimate_txs_remaining, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/home/user/wspace/electrumx/./src/electrumx/server/db.py", line 319, in flush_dbs self.db_flushed_event.set() # ~~~~~~~~~~~~~~~~~~~~~~~~~^^ File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/asyncio/locks.py", line 192, in set fut.set_result(True) ~~~~~~~~~~~~~~^^^^^^ File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/asyncio/base_events.py", line 829, in call_soon self._check_thread() ~~~~~~~~~~~~~~~~~~^^ File "/opt/cpython-versions/3.14.6_nogil/lib/python3.14t/asyncio/base_events.py", line 866, in _check_thread raise RuntimeError( "Non-thread-safe operation invoked on an event loop other " "than the current one") RuntimeError: Non-thread-safe operation invoked on an event loop other than the current one INFO:BlockProcessor:fetch_and_process_blocks taskgroup stopped. INFO:SessionManager:stop listening on servers, so no new sessions can be created INFO:SessionManager:closing down server for rpc://127.0.0.1:8000 INFO:SessionManager:closing down server for tcp://127.0.0.1:51001 INFO:SessionManager:closing down server for ssl://127.0.0.1:51002 INFO:SessionManager:closing down server for tcp://[::1]:51001 INFO:SessionManager:closing down server for ssl://[::1]:51002 INFO:SessionManager:closing 0 active sessions INFO:SessionManager:waiting for all server's resources to close INFO:SessionManager:all servers closed <hang> ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
see individual commits