[BEAM-9889] Populate local instance of InfluxDB with data

[kamilwu]

The goal of these changes is to make the process of creating and modifying Grafana dashboards easier by populating InfluxDB database with real data.

Two workflows have been implemented:

• a cronjob that makes a full backup of InfluxDB and sends it to a public GCS bucket. The bucket has object versioning[1] enabled. It also has a lifecycle rule[2] that keeps only the 14 most recent versions of the backup.
• a docker-compose service that starts just after InfluxDB, downloads the latest version of the backup and restore its content.

[1] https://cloud.google.com/storage/docs/object-versioning
[2] https://cloud.google.com/storage/docs/lifecycle

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Dataflow	Flink	Samza	Spark	Twister2
Go		---		---		---
Java
Python				---		---
XLang	---	---		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[kamilwu]

R: @Ardagan Will you find a moment to make a code review?

cc: @iemejia

[Ardagan]

Nice addition that can make for local development easier.

[Ardagan]

Are these backups available to non-committers?

[kamilwu]

Everyone on the Internet has read-only access to the bucket. Grafana dashboards are already public, so it doesn't change anything in terms of security or privacy

[Ardagan]

As I can understand, we overwrite same backup each time. Given daily creation of backups and noone is paged when something goes wrong, this backup can be easily corrupted as well.

Though it is ok for current PR, it's best to add some rotation to backups, keeping 1-2 week old backups available.

[kamilwu]

Yes, we overwrite the same backup each time, but the bucket has object versioning enabled. According to the doc:

You enable Object Versioning for a bucket. Once enabled:

Cloud Storage creates a noncurrent version of an object each time you perform an overwrite or delete of the live version

If you run gsutil ls -la gs://apache-beam-testing-metrics, you will see all versions of the backup that were created so far.

[Ardagan]

I'm not well familiar with influx DB operations. It might be worth another pair of eyes taking a look at overall backup approach.

[kamilwu]

It might be worth another pair of eyes taking a look at overall backup approach.

R: @iemejia, I think you were interested in this change. Could you verify if everything is fine?

[Ardagan]

I guess we can merge this since there're no more comments for quite some time.

[kamilwu]

Yeah, let's merge it then. Thanks @Ardagan