Skip to content
This repository has been archived by the owner on Aug 25, 2023. It is now read-only.

Commit

Permalink
YACHT-1014: small update in SLI_README
Browse files Browse the repository at this point in the history
  • Loading branch information
marcin-kolda committed Sep 17, 2018
1 parent 21bdd98 commit e6c400d
Showing 1 changed file with 10 additions and 6 deletions.
16 changes: 10 additions & 6 deletions SLI_README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,26 @@

This metric is measuring number of tables which are not backed up since X days from last observed change.

Metric is implemented as a BigQuery query, which uses data about modifications of tables (collected by Census) and data about existing backups.
Metric is implemented as a BigQuery query, which uses data about modifications of tables (collected by [GCP Census](https://github.com/ocadotechnology/gcp-census)) and data about existing backups.

### Number of tables not modified since 3 days and which have no equal backup

This metric is measuring number of tables that were not modified since 3 days and which last available backup does not contain the same data as source.
This metric is measuring number of tables that were not modified since 3 days and which last available backup does not contain the same metadata as source.

Metric is implemented as a BigQuery query, which list all tables where last modification was over 3 days ago and joins them with last available backup using [Cloud Datastore export](./SETUP.md#cloud-datastore-export) and then compare the number of bytes and rows between source table and its backup (collected by Census).
Metric is implemented as a BigQuery query, which list all tables where last modification was over 3 days ago and joins them with last available Datastore backups metadata using [Cloud Datastore export](./SETUP.md#cloud-datastore-export) and then compare the number of bytes and rows between source table and its backup (collected by [GCP Census](https://github.com/ocadotechnology/gcp-census)).

## SLI setup

To measure SLI, please follow all the steps below:
1. Install [GCP Census](https://github.com/ocadotechnology/gcp-census) application that periodical collects metadata about BigQuery tables.
1. Configure [Cloud Datastore export](./SETUP.md#cloud-datastore-export),
which periodically exports backup metadata and stores it in BigQuery,
1. Install [Terraform](https://www.terraform.io/) if it's not available,
1. Create [Terraform backend](https://www.terraform.io/docs/backends/) GCS bucket, which will be used to store TF state:
```bash
export TERRAFORM_STATE_BUCKET_ID="<your-bucket-id-to-store-terraform-state>"
gsutil mb -p ${BBQ_PROJECT_ID} gs://${TERRAFORM_STATE_BUCKET_ID}/
```
1. Export BBQ, BBQ restoration project and GCP Census project ids:
```bash
export TF_VAR_bbq_project=${BBQ_PROJECT_ID}
Expand All @@ -27,8 +33,6 @@ which periodically exports backup metadata and stores it in BigQuery,
```
1. Create a bucket to store remotely infrastructure state and then create all views/tables using [Terraform](https://www.terraform.io/) by running the following commands:
```bash
export TERRAFORM_STATE_BUCKET_ID="<your-bucket-id-to-store-terraform-state>"
gsutil mb -p ${BBQ_PROJECT_ID} gs://${TERRAFORM_STATE_BUCKET_ID}/
terraform init -backend-config="bucket=${TERRAFORM_STATE_BUCKET_ID}"
terraform apply
```
Expand All @@ -52,4 +56,4 @@ which periodically exports backup metadata and stores it in BigQuery,

## SLI results

All delayed backups will be streamed to ``SLI_history.SLI_backup_creation_latency`` table and ``SLI_backup_quality_views.SLI_quality`` view.
All SLI results can be queried from ``SLI_history.SLI_backup_creation_latency_view`` and ``SLI_backup_quality_views.SLI_quality`` views.

0 comments on commit e6c400d

Please sign in to comment.