Get GCP quota approvals to increase pipeline capacity by 10x

[rviscomi]

[pmeenan]

I just sent a quota increase request for 2400 CPU's in each of the 8 US GCP regions which would give us a little over 10x increase. I don't expect it will get approved but hopefully it at least kicks off an email thread where we can see what the options are and if there is a balance we can do across regions to get that kind of capacity.

[pmeenan]

Ahh, the fun of automated bots/systems. Got approval for 1/2 of one of the quotas but everything was auto-declined with a suggestion to reach out to sales. Now starts the path of seeing if we can find the actual team to talk to.

[pmeenan]

Looks like the requests get auto-approved if they are no more than ~2x what you are currently using for each quota. The current server can't handle a full 10x thruput but can probably handle ~1.5x so I'm incrementally increasing the areas where we have really small quotas (like 12 VM's in most of the regions) so we'll be starting from a much higher base when we're ready to go full 10x.

We'll likely need to get the new pipeline working first and then complete the quota requests once we have a pipeline that won't crumble with that many tests. I'm more optimistic than I was before about getting the approvals though since we got the 2400 CPU approval for Iowa (where we were running the bulk of the infrastructure before).

[rviscomi]

Something to also keep in mind is that we'll be running the Dataflow pipeline during the crawl to process test results and write them to BQ on the fly, and that would eat into some of those shared resources.

Including @giancarloaf here in case there's any coordination that needs to be done to run the Dataflow pipeline at reduced capacity or possibly in a different region with separate quota. Not sure, so I'll leave that to you both.

[pmeenan]

With the pipeline running on GCP now, we are pushing usage hard enough that I could put in for the full quota increases (and have no doubt we can add quota as needed). We have 2400 CPU's in most of the regions and 3000 in us-central1 (using around 60% of that right now while we fix bottlenecks).

Quota-wise we're above 10x.

[rviscomi]

Amazing news thanks @pmeenan!