ci: use self-hosted runners for CI pipelines

[gustavovalverde]

Motivation

Some of our tests, mainly our full sync test, are way above GitHub-hosted runners limits, being the 6 hours timeout the main limitation.

We might want to start testing GitHub self-hosted runners to raise the limits.

Designs

• Deploy a Self-hosted runner to GCP using terraform
• Confirm how the cached state is handled with a self-hosted runner (SHR)
• Validate how SSH connections will be handled in a SHR
• Move the full-sync job to this runner
• Validate time, reliability and cost of hosting the runner

Related Work

Another approach would be using #5551, if we get through their beta waitlist. But this might won't get rid from the 6 hours limit.

[mpguerra]

Hey team! Please add your planning poker estimate with Zenhub @arya2 @dconnolly @gustavovalverde @oxarbitrage @teor2345

[teor2345]

@gustavovalverde @mpguerra can we be more specific with the scope here? What specific changes are we going to try?

It's hard to estimate this ticket without a list of tasks or a list of requirements.

[arya2]

Larger runners are still limited to 6 hours, but this may bypass ssh issues if done the hard way.

https://github.com/actions-runner-controller/actions-runner-controller
https://github.com/jonico/awesome-runners

[dconnolly]

@gustavovalverde @mpguerra can we be more specific with the scope here? What specific changes are we going to try?

It's hard to estimate this ticket without a list of tasks or a list of requirements.

Yeah we might want to scope this first issue as 'try setting up /a/ self-hosted runner for one simple workflow/action'

[gustavovalverde]

I added some steps

[teor2345]

Thanks, that looks good!

• Add other sync jobs to the runner

I think moving all the sync jobs might be out of scope for an initial ticket. There's no particular urgency, they seem to be working ok.

Or did you want this to become a tracking issue?

[gustavovalverde]

I removed that one.

[aktech]

By the way, if you want a no-maintenance method to achieve that, you can use cirun.io for the same.

[mpguerra]

unscheduling for now