Implement a solution to cleanup GCP resources in beam testing projects

[damccorm]

FR to create a solution to clean up lingering GCE, VMs, Dataproc clusters, etc.

Some sort of perodic job which runs and deletes leaked GCE VMs would prevent the resource concerns caused by failed tests as in: 

https://issues.apache.org/jira/browse/BEAM-12898

Possible solutions:

• Name all the resources with a convention that represents a TTL. Then the resources can be deleted by a cron/periodic job if the TTL has expired
• Dataproc cluster may have a TTL configuration option itself which the flink on dataproc test can use.
• Similar to how we delete old Dataflow jobs in https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_CancelStaleDataflowJobs.groovy

 

Imported from Jira BEAM-12903. Original Jira may contain additional context.
Reported by: ajamato@google.com.

[elink21]

.take-issue

[elink21]

Hi team, the new cron-job is now running: https://ci-beam.apache.org/job/Cleanup%20Dataproc%20Resources/8/console , now in case that Flink resources get leaked those will be monitored and cleaned periodically ,we think this task can be closed now, regards.