Make transaction processing more scalable

[bbedward]

This has a couple of main components

1. Moving RPC send and receive actions to a separate process. Investigate using persistent queues to communicate between processes. In current setup the secondary process should not interact with the database at all due to multi write limitations of sqlite. It should be strictly for work/send/receiving and should communicate the results back to the main process which would then update our database.

This would increase thoroughput quite dramatically due to limitations with python threading and the GIL

2. Figure a way to separate work_generate from the send/receive action and distribute among multiple peers. This is a little tricky as there's so much activity the frontiers frequently change.

[meltingice]

Ok, here's my proposed solution. Would love to hear your thoughts on this. We can introduce a separate process that handles sending and receiving, and is backed by Redis/Sidekiq.

1. TipBot transaction is created. We store the transaction in SQLite like we already do.
2. After writing the transaction to the db, we build a Sidekiq job and push it into the send queue. Doing so is trivial from Python.
3. Over in the new worker process, the job is picked up and the transaction is sent to the node RPC.
4. Once the transaction is successfully sent, the job schedules a corresponding receive job for the destination address with the address and block in the payload.
5. The receive job is picked up by the worker, and it tells the node RPC to receive the block for the account. This is idempotent, so if the RPC fails for whatever reason, it will be automatically retried until it succeeds.

This solution is highly scalable because you can increase the number of workers and threads as needed. As the workers scale, the bottleneck will become the node, so we'll have to handle that separately if needed (but probably not for awhile).

One missing piece is that we probably want a way to update the database and tell it the transaction has been processed. Might require moving to Postgres, or we can set up some kind of interprocess communication between the two.

[bbedward]

Sounds about perfect to me. Besides not really needing to worry about the receives being retried. We don't care about the result of that job as much as we do the result of the send job .

Sending could use the retry mechanism, and an easy solution to the sqlite multi processing problem is probably just switching to postgres so maybe issues #3 and #5 could be prerequisites to this one.

[tundak]

@renesq I need some support (paid also fine) for an issue, can i have your email ? or can you drop me contact on sonu5650@gmail.com