Batch tx requests #1759
Batch tx requests #1759
Conversation
)" This reverts commit 28bc035.
andrewtoth
force-pushed
the
batch-tx-requests
branch
from
eb2cd37
to
57b0f79
Compare
andrewtoth
force-pushed
the
batch-tx-requests
branch
from
57b0f79
to
c9d270b
Compare
There was a problem hiding this comment.
@raphjaph @casey Reviewed conceptually and talked to @andrewtoth - this approach looks consistent with what we discussed yesterday. Does not introduce new communication protocol, and massively speeds up the indexing process for when we have inscriptions (from a day+ to less than an hour).
Let's discuss and see if you also ACK this approach, then @andrewtoth can complete this PR.
|
Concept ACK from @veryordinally, @casey, and @raphjaph (reviewing together on a Discord call). This seems like the right approach. It doesn't require any config changes and doesn't change our security model. Would be great to get the MVP in! Should be as simple as possible, with no configuration or additional knobs. Do not worry about polishing too much. Just get full test coverage, undraft, and request review. |
|
@veryordinally @raphjaph @casey fixed broken test, ready for review. Also, #1637 and #1731 are relevant to this as well. Since we're using hyper, the json-rpc |
There was a problem hiding this comment.
Reviewed by the peanut gallery in Discord: We are confident this fixes the issue but we would love any comments that will help us understand and maintain this code. If you could do that in addition to our review comments (if they are easy), we can smash merge ❤️
src/index/tx_fetcher.rs
Outdated
| .unwrap() | ||
| .cmp(&b.id.parse::<usize>().unwrap()) | ||
| }); | ||
|
|
There was a problem hiding this comment.
add an assert if it's critical that it's sorted
There was a problem hiding this comment.
Sorry, not sure what I would be asserting here? It needs to be sorted before we return, but this is sorting it so we know it's sorted?
|
@raphjaph addressed most of your feedback, asked some questions for clarification on some things. |
andrewtoth
requested review from
veryordinally and
raphjaph
and removed request for
raphjaph and
veryordinally
|
|
@kvnn running pop_os with Intel® Core™ i7-10870H CPU @ 2.20GHz × 16, with nvme drive. Bitcoind was running locally on the same machine. Did a test with the same machine but connecting over wifi to a bitcoind on a separate machine in the local network. Took 178 minutes to get to the same block. |
Fixes https://github.com/casey/ord/issues/1364 by reverting #1357.
This PR uses
hyperto fetch raw transactions in the background with batched JSON-RPC requests sent 12 at a time concurrently. This saturates the default 4 rpc threads but doesn't exceed the default rpcworkqueue of 16. I managed to sync to block 776774 in 41 minutes.This does not require any configuration changes. It uses the json-rpc endpoint, but more efficiently. All missing inputs are sent to a separate thread before indexing, and the resulting input values are sent back via a channel in order.