Establish a strategy to prevent or mitigate quota-related flakes in Beam integration tests.

[damccorm]

Every now and then our postcommit tests fail due to lack of quota in apache-beam-testing, see [1].

Filing this as an umbrella issue to establish a strategy for preventing quota-related failures in Apache Beam tests.

Potential ideas:

• improve resilience of test infrastructure to retry tests that fail due to lack of quota, or delay the test execution when quota limits fall below a threshold
• monitor & procure the quota usage in apache-beam-testing
• tune the parameters affecting the number of concurrent test jobs according to available quota, and document best practices.
• stop previous jenkins job that were triggered on a PR when a new job is started.

[1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20test-failures%20AND%20text%20~%20quota

Imported from Jira BEAM-7749. Original Jira may contain additional context.
Reported by: tvalentyn.