Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Batched graph updates #3367
This PR introduces batched operations for channel graph ingestion in order to speed up IGD. Currently a single db transaction is used to process each
With this PR we bring this down to linear in the duration it takes the sync the graph, as we'll no make at most 2 db transactions/sec. OMM I was able to get a 4 minute graph sync using this, implying only about 500 db transactions were made in total.
So far I'm seeing a 2x speedup in IGD. The CPU and memory usage should be much lower, but I'm still in the process of tuning and gathering exact performance metrics.
joostjager left a comment
It will indeed be interesting to see how fast the IGD is on low powered devices. I saw my rpi3 take 5 hours for sync on a 100 mbit fiber connection.
My main question related to this PR is whether this is the right optimization path to take.
How would this PR compare to a different, quite extreme, approach: The graph only exists in memory. It is updated in memory and queried from memory (which we want at some point anyway to speed up path finding). Existing caches can be removed / merged with the in-memory graph.
Then there is a background process that periodically (maybe once a minute) serializes all of it into a byte array and writes it in a single bbolt key (or file even). On startup, that value is deserialized into memory again. Because the graph is non critical data, it is not a problem if we don't have all the very latest data on disk.
valentinewallace left a comment
Great feature, so excited to get this in for mobile users!!
Mostly for feedback I just have some questions. Looking forward to reading what the testing looks like, but time-based batching scheduler seems like the perfect starting point for batching db txs like these