perf: multi-threading experiments (--disable-gil?)

[SomberNight]

Split block processing work across a thread-pool.

There are two kinds of performance we should consider:

1. wall clock time for first sync from genesis
2. wall clock time it takes to process a single block when already at the tip
   • check the logs: grep BlockProcessor, e.g.:

     DEBUG:BlockProcessor:new height: 955368
     INFO:BlockProcessor:processed 1 block size 1.72 MB in 2.4s
     

   • on master (and for the past several years), there has been a lot of variance for how long this takes. On my server, it is typically around 2 seconds, but 10+ seconds is not that rare either. The max I saw was 30 seconds at one point! That's a lot.

---

I have been experimenting running ElectrumX with a free-threaded cpython (compiled with --disable-gil), and running with PYTHON_GIL=0 at runtime. Normally Python has a Global Interpreter Lock, which means in practice only one thread can do work on the CPU, i.e. single core CPU performance is often the bottleneck for many workloads.

Since Python 3.13, it is possible to run cpython without the GIL.

---

On master, the BlockProcessor is completely single-threaded. It processes blocks in order.
For each block, it processes txs in order. For each tx:

• it processes the tx inputs in order: spending utxos (calling spend_utxo)
• it processes the tx outputs in order: funding utxos

Part of the reason for the huge variance in time needed for (2), is spend_utxo hitting the disk. spend_utxo does a blocking read in RocksDB (or LevelDB), which might hit the RocksDB cache, or might hit the OS cache, or finally might have to go and read the disk. For a single block with 5k txs, each with 5 inputs, spend_utxo might hit the disk 25k times, sequentially. (though such an extreme example is unlikely: the caches will get warmed)

This PR changes the block processing logic to process txs out-of-order within a single block:

• first, concurrently process all the outputs, funding utxos
• second, concurrently process all the inputs, spending utxos
• we disregard the possibility that this way a utxo might be spent before it gets created: we trust bitcoind to have already validated consensus rules

As the spend_utxo codepath is disk-read-bottlenecked, this should improve the performance regardless of the GIL.

---

In case the GIL is disabled, this PR also parallelises a lot of the CPU-bottlenecked calculations.

However, I have found that doing this unconditionally makes performance of (1) significantly slower in the GIL-enabled traditional scenario (tx/sec since genesis: 7,923 (height: 785,897), see later). Hence, we maintain two distinct code paths and branch at runtime depending on whether the GIL is enabled.

---

Measured performance of this PR compared to master (using RocksDB 9):

• sync from genesis (1):
  • master, GIL enabled: tx/sec since genesis: 10,282 (height 955,147)
  • this PR, GIL enabled: tx/sec since genesis: 9,766 (height 783,693)
  • this PR, GIL disabled: tx/sec since genesis: 22,413 (height 955,333)
• single block at tip (2):
  • master, GIL enabled: (min=0.2 sec, avg=1.275 sec, max=3.4 sec, num_blocks=222)
  • this PR*, GIL enabled: (min=0.1 sec, avg=0.633 sec, max=2.4 sec, num_blocks=221)
    • (tested using a previous version of this PR that should be similar enough)
  • this PR, GIL disabled: (min=0.2 sec, avg=0.514 sec, max=1.9 sec, num_blocks=44)

Memory usage: during sync from genesis, with the GIL disabled, due to the massive parallelization significantly more RAM is needed. I was testing with the default CACHE_MB=1200: with the GIL, memory usage hovers around 2 GB, while without it is around 8 GB.

---

For now, running ElectrumX with the GIL disabled is to be considered experimental. However, it seemed to work for me :)

---

generic perf baseline from master for my own reference (using SomberNight@b3ff9f9):

DEBUG:BlockProcessor:new height: 712588
DEBUG:BlockProcessor:total times:
ch_adv_bl=48254.05 (
  block_deser=12296.57,
  adv_bl=26595.92 (
      adv_tx=26512.74 (
          spend_utxo=16470.37,
      ),
  ),
  flush=8220.05,
),
prefetch_blocks=4578.71,