bot: Force refresh the blockhash more often while sending txs

[joncinque]

While running the stake-o-matic for testnet stake pools, I kept running into the same issue often reported on buildkite:

[2021-09-10T01:47:00.354078368Z WARN  solana_stake_o_matic::rpc_client_utils] Failed to send transaction: ClientError { request: Some(SendTransaction), kind: RpcError(RpcResponseError { code: -32002, message: "Transaction simulation failed: Blockhash not found", data: SendTransactionPreflightFailure(RpcSimulateTransactionResult { err: Some(BlockhashNotFound), logs: Some([]), accounts: None }) }) }

So as a little patch / Ugly Thing ™️ , this force refreshes the blockhash.

This is not intended as a replacement for the work happening in #143, but as a band-aid. Note that the number of 100 was chosen based on my local machine. 200 still resulted in some blockhashes expiring, so I went with this safe value, which consistently manages to push all transactions out. We may be able to bump it up for the actual runs.

[t-nelson]

Doesn't this break the blockhash expiration checks below since we now have multiple live blockhashes in play, but only account for the most recent

[joncinque]

Kind of -- I've been confused about that logic for a bit.

The blockhash check is done and transactions are resent before we've actually checked the transaction status. Which means that you can take time to send 100 transactions, and if the blockhash expires right after that, you'll just resend them all without ever checking if the transactions even errored. The resend logic should go after the check, and only if the status is status.err.unwrap() == TransactionError::BlockhashNotFound, no?

As I was thinking about better ways to do this, I thought: "oh, then we'd probably need a hashmap. Trent probably already thought that, which means that it's probably in his PR. I'll just read his PR"

[joncinque]

So this does a Slightly Nicer Thing ™️. Now, we actually refresh the blockhash and retry a transaction if the simulation or transaction has a blockhash error. So far, this has done well in testing until getting timeouts from testnet. I'll keep testing this to see how well it recovers from blockhash expirations.

[joncinque]

This has changed a lot, so I'll close and re-open

[mvines]

I feel like we should just flush send_and_confirm_transactions () entirely and use TpuClient!

send_and_confirm_messages_with_spinner() is still nestled in the cli (https://github.com/solana-labs/solana/blob/24fd47a32d5d65bed361bdaab8cafb3387ba84a6/cli/src/program.rs#L2200), and not yet available in solana-client, but a stand-alone version of send_and_confirm_messages_with_spinner() with some improvements is available in the spl-token bench ... commands I recently added:
a. https://github.com/solana-labs/solana-program-library/blob/master/token/cli/src/rpc_client_utils.rs
b. also check out send_messages() at https://github.com/solana-labs/solana-program-library/blob/21ae18dbd0458f9e5f18b1407a626670f6475438/token/cli/src/bench.rs#L428

[joncinque]

Yeah I was thinking about that too... I do have the changes done using rpc client, so maybe we go the TpuClient route next time everything falls over?

[mvines]

Up to you. I found moving to the TpuClient pretty easy, should be the same here if you just lift my work from spl-token bench ... that I referenced. It makes sending thousands of transactions so much more reliable!

[joncinque]

All right sounds good, I'll give it a shot!