BigQuery Tools: Dig into the costs of Google BigQuery projects
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bqdb
bqscrape
googlelogin
integrationtests/bqscrape_test
templates
.gitignore
LICENSE
README.md
bqcost.go
bqcost.yaml
bqcost_test.go
check.sh

README.md

BigQuery Tools

A tool to figure out what costs money in a Google BigQuery project. Try it at https://bigquery-tools.appspot-preview.com/ Send feedback to ej@evanjones.ca.

Deploying

  1. Get a BigQuery OAuth Client ID: https://console.cloud.google.com/apis/credentials . Add https://yourdomain.appspot-preview.com/oauth2callback as the OAuth2 callback URL. If you want to test this locally, you should also add localhost:8080.

  2. Generate a secure cookie hash key: openssl rand -hex 64 and encryption key: openssl rand -hex 32

  3. Save these values as constants in credentials.go:

package main

const googleOAuthClientID = "..."
const googleOAuthClientSecret = "..."

var cookieHashKey = mustDecodeHex("...")
var cookieEncryptionKey = mustDecodeHex("...")
  1. Create a Cloud SQL instance. Edit bqcost.go and bqcost.yaml to reference the correct name.

  2. Edit bqcost.go to reference the correct host name that will serve your app.

  3. Deploy: aedeploy gcloud app deploy --project=(PROJECT) bqcost.yaml

Running locally

You can run a local copy against cloud SQL with go run bqcost.go credentials.go --cloudSQLProxy=true

You can also run a local copy using SQLite, but I need to figure out a way to make this work without breaking deploys to App Engine Flexible.

Known Issues

  • Downloading the state of all the tables is incredibly slow. It needs to be parallelized.
  • The "get table data from BigQuery" job is just a goroutine. If the instance restarts, the job is stuck forever and you will never be able to access the project. This should be split into smaller chunks that get retried.