bridge/pkg/solana: retry VAA submission on transient errors

[leoluk]

Stack from ghstack:

• bridge: log submission for cross-submissions to Solana #54 bridge: log submission for cross-submissions to Solana
• bridge: do not resubmit submitted VAAs during aggregation #53 bridge: do not resubmit submitted VAAs during aggregation
• bridge: do not log errors for duplicate VAA submissions #52 bridge: do not log errors for duplicate VAA submissions
• bridge: have all nodes submit VAAs to Solana #51 bridge: have all nodes submit VAAs to Solana
• bridge/pkg/solana: retry VAA submission on transient errors #48 bridge/pkg/solana: retry VAA submission on transient errors
• agent: return gRPC Internal error on submission failure #47 agent: return gRPC Internal error on submission failure
• bridge: move initial guardian set fetching to pkg/ethereum/watcher.go #46 bridge: move initial guardian set fetching to pkg/ethereum/watcher.go
• bridge: propagate panics from runnables #45 bridge: propagate panics from runnables
• bridge: in-place debugging using dlv #44 bridge: in-place debugging using dlv

In particular, this fixes a race condition where the Solana devnet would
take longer to deploy than the ETH devnet to deploy and we'd end up
with an outdated guardian set on Solana.

We currently create a Goroutine for every pending resubmission, which
waits and blocks on the channel until solwatch is processing requests
again. This is effectively an unbounded queue. An alternative approach
would be a channel with sufficient capacity plus backoff.

Test Plan: Deployed without solana-devnet, waited for initial guardian
set change VAA to be requeued, then deployed solana-devnet.

The VAA was successfully submitted once the transient error resolved:

[...]
21:08:44.712Z	ERROR	wormhole-guardian-0.supervisor	Runnable died	{"dn": "root.solwatch", "error": "returned error when NODE_STATE_HEALTHY: failed to receive message from agent: EOF"}
21:08:44.712Z	INFO	wormhole-guardian-0.supervisor	rescheduling supervised node	{"dn": "root.solwatch", "backoff": 0.737286432}
21:08:45.451Z	INFO	wormhole-guardian-0.root.solwatch	watching for on-chain events
21:08:50.031Z	ERROR	wormhole-guardian-0.root.solwatch	failed to submit VAA	{"error": "rpc error: code = Canceled desc = stream terminated by RST_STREAM with error code: CANCEL", "digest": "79[...]"}
21:08:50.031Z	ERROR	wormhole-guardian-0.root.solwatch	requeuing VAA	{"error": "rpc error: code = Canceled desc = stream terminated by RST_STREAM with error code: CANCEL", "digest": "79[...]"}
21:09:02.062Z	INFO	wormhole-guardian-0.root.solwatch	submitted VAA	{"tx_sig": "4EKmH[...]", "digest": "79[...]"}