bigtable-backup: Tool for creating and managing periodic tables stored in Bigtable #729

sandeepsukhani · 2019-07-09T09:07:53Z

Relies on this tool: https://github.com/grafana/bigtable-backup

It has 2 commands:

backup-active-periodic-table: For backing up the active periodic table
ensure-backups: It helps ensure right tables are backed up and retain only single backup for non-active periodic tables. It also ensures we have a backup created after table becomes non-active i.e after the last possible timestamp in the period for which table was active.

Usage:

python3 bigtable-backup.py --help
usage: bigtable-backup.py [-h] --bigtable-project-id BIGTABLE_PROJECT_ID
                          --bigtable-instance-id BIGTABLE_INSTANCE_ID
                          --bigtable-table-id-prefix BIGTABLE_TABLE_ID_PREFIX
                          --destination-path DESTINATION_PATH --temp-prefix
                          TEMP_PREFIX --periodic-table-duration
                          PERIODIC_TABLE_DURATION
                          {backup-active-periodic-table,ensure-backups} ...

positional arguments:
  {backup-active-periodic-table,ensure-backups}
                        commands
    backup-active-periodic-table
                        Backup active periodic table
    ensure-backups      Ensure backups of right tables exist

optional arguments:
  -h, --help            show this help message and exit
  --bigtable-project-id BIGTABLE_PROJECT_ID
                        The ID of the GCP project of the Cloud Bigtable
                        instance
  --bigtable-instance-id BIGTABLE_INSTANCE_ID
                        The ID of the Cloud Bigtable instance that contains
                        the tables
  --bigtable-table-id-prefix BIGTABLE_TABLE_ID_PREFIX
                        Prefix to build IDs of the tables using periodic-
                        table-duration
  --destination-path DESTINATION_PATH
                        GCS path where data should be written. For example,
                        gs://mybucket/somefolder/
  --temp-prefix TEMP_PREFIX
                        Path and filename prefix for writing temporary files.
                        ex: gs://MyBucket/tmp
  --periodic-table-duration PERIODIC_TABLE_DURATION
                        Periodic config set for loki tables in days

Makefile

cyriltovena

LGTM to me but I don't have enough knowledge to approve. I think @gouthamve should have a look.

gouthamve

Looks good, but I think this is super brittle. We need some checks and monitoring here. When will this exit unsuccessfully?

gouthamve · 2019-07-30T10:36:22Z

tools/bigtable-backup/bigtable-backup.py

+
+    while True:
+        line = popen.stdout.readline()
+        if i == 0:


Can be a bool. But this seems hacky, can we add a json output to the tool and we parse json?

bigtable-backup .... -ojson? And we parse the json here.

gouthamve · 2019-07-30T10:46:36Z

tools/bigtable-backup/bigtable-backup.py

+
+    backup_active_periodic_table_parser = subparser.add_parser('ensure-backups',
+                                                               help="Ensure backups of right tables exist")
+    backup_active_periodic_table_parser.add_argument('--period-from', required=True, type=valid_date)


Can we also have a time-range, like 90 days? --period-for? And make sure only one is set.

gouthamve · 2019-07-30T11:00:39Z

tools/bigtable-backup/bigtable-backup.py

+        if last_timestamp_from_table_number > timestamps[-1]:
+            # Checking whether backup is created after last timestamp of tables period.
+            create_backup(table_id, args)
+        elif len(timestamps) > 1:


No need of an elif here. We need both checks always.

sandeepsukhani · 2019-08-02T06:34:33Z

@gouthamve Thanks for the review! All the requested changes are done.

gouthamve

Minor nits, other than that good to go!

gouthamve · 2019-08-02T06:49:36Z

tools/bigtable-backup/bigtable-backup.py

+registry = CollectorRegistry()
+bigtable_backup_job_last_run_unixtime = Gauge('bigtable_backup_job_last_run_unixtime', 'Last time a bigtable backup job ran at', registry=registry)
+bigtable_backup_job_last_success_unixtime = Gauge('bigtable_backup_job_last_success_unixtime', 'Last time a bigtable backup job successfully finished', registry=registry)
+bigtable_backup_job_runtime_seconds =  Gauge('bigtable_backup_job_runtime_seconds', 'Runtime of last successfully finished bigtable backup job', registry=registry)


Can we add 2 more metrics: backups_created and backups_existing?

gouthamve · 2019-08-02T06:49:59Z

tools/bigtable-backup/bigtable-backup.py

+from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
+
+registry = CollectorRegistry()
+bigtable_backup_job_last_run_unixtime = Gauge('bigtable_backup_job_last_run_unixtime', 'Last time a bigtable backup job ran at', registry=registry)


Should be _seconds and not _unixtime.

It has 2 commands: 1. backup-active-periodic-table: For backing up active periodic table 2. ensure-backups: It helps ensure right tables are backed up and retain only single backup for non active periodic tables

sandeepsukhani requested review from tomwilkie and gouthamve July 9, 2019 09:07

cyriltovena reviewed Jul 15, 2019

View reviewed changes

Makefile Outdated Show resolved Hide resolved

cyriltovena reviewed Jul 15, 2019

View reviewed changes

gouthamve reviewed Jul 30, 2019

View reviewed changes

tomwilkie removed their request for review July 30, 2019 13:08

gouthamve approved these changes Aug 2, 2019

View reviewed changes

sandeepsukhani requested a review from gouthamve August 2, 2019 07:39

Tool for creating and managing periodic tables stored in Bigtable

8e8893a

It has 2 commands: 1. backup-active-periodic-table: For backing up active periodic table 2. ensure-backups: It helps ensure right tables are backed up and retain only single backup for non active periodic tables

sandeepsukhani force-pushed the loki-bigtable-backup branch from d1b5d2c to 8e8893a Compare August 2, 2019 09:05

update parent image for building bigtable backup docker image

7d763ed

gouthamve approved these changes Aug 2, 2019

View reviewed changes

renamed docker image to loki-bigtable-backup

dd70087

sandeepsukhani merged commit 9930f70 into grafana:master Aug 2, 2019

Tikam02 mentioned this pull request Jun 2, 2020

Loki-bigtable backup and restore #2157

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bigtable-backup: Tool for creating and managing periodic tables stored in Bigtable #729

bigtable-backup: Tool for creating and managing periodic tables stored in Bigtable #729

sandeepsukhani commented Jul 9, 2019

cyriltovena left a comment

gouthamve left a comment

gouthamve Jul 30, 2019

gouthamve Jul 30, 2019

gouthamve Jul 30, 2019

sandeepsukhani commented Aug 2, 2019

gouthamve left a comment

gouthamve Aug 2, 2019

gouthamve Aug 2, 2019

bigtable-backup: Tool for creating and managing periodic tables stored in Bigtable #729

bigtable-backup: Tool for creating and managing periodic tables stored in Bigtable #729

Conversation

sandeepsukhani commented Jul 9, 2019

cyriltovena left a comment

Choose a reason for hiding this comment

gouthamve left a comment

Choose a reason for hiding this comment

gouthamve Jul 30, 2019

Choose a reason for hiding this comment

gouthamve Jul 30, 2019

Choose a reason for hiding this comment

gouthamve Jul 30, 2019

Choose a reason for hiding this comment

sandeepsukhani commented Aug 2, 2019

gouthamve left a comment

Choose a reason for hiding this comment

gouthamve Aug 2, 2019

Choose a reason for hiding this comment

gouthamve Aug 2, 2019

Choose a reason for hiding this comment