Use cached state for continuous deployment instances

[teor2345]

Motivation

If we want to deploy instances on every main branch push, they need to use a persistent state disk, or a copy of the CI cached state.

Otherwise, they will put a lot of load on the network by syncing all the time. And they won't be very useful for testing.

Doing this for the release and manual deploy instances also makes them available faster, and puts less load on the network. (Changing the state path will disable the cache for manual deploys, that's an existing input parameter.)

This would be useful to have before we tag the first stable release, but it's not a blocker.

Other Changes

We should re-enable deployments on main branch pushes as part of this PR, by reverting PR #6895.

Testing

Check the Google Cloud logs of a manually deployed instance, and look for the block height.

[mpguerra]

@gustavovalverde let's discuss when we can schedule this in

[mpguerra]

Hey team! Please add your planning poker estimate with Zenhub @arya2 @gustavovalverde @oxarbitrage @teor2345 @upbqdn

[gustavovalverde]

Even though this is marked as high-priority, I've removed it from the following sprint as we have some big tickets which we've been moving around for a while and are causing other pipeline changes harder to tackle. We should plan this for late September.

[teor2345]

Even though this is marked as high-priority, I've removed it from the following sprint as we have some big tickets which we've been moving around for a while and are causing other pipeline changes harder to tackle. We should plan this for late September.

If we can delay it until late September, maybe it's not actually a high priority 🙂
What are we losing by not having these instances relaunch on every commit to main?

I think the Zebra team are mostly doing fine without it, but we might need to be slightly more careful to run full sync tests as needed.

Edit: we already have coverage for a lot of what's tested by these deployments in our CI sync to tip tests. But they only run for a short time. So bugs that happen after hours or days of runtime aren't currently being tested for.

[mpguerra]

I'll put it in sprint 19 for now with the thought that it may move to Sprint 20