Skip to content
A docker image that runs a script to export Leanplum data into BigQuery
Python Makefile Dockerfile Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
bin
leanplum_data_export
requirements
tests
.flake8
Dockerfile
Makefile
README.md
docker-compose.yml
setup.py

README.md

CircleCI

Leanplum Data Export

This repository is the job to export Mozilla data from Leanplum and into BQ.

It's dockerized to run on GKE. To run locally:

pip install .
leanplum-data-export export-leanplum \
  --app-id $LEANPLUM_APP_ID \
  --client-key $LEANPLUM_CLIENT_KEY \
  --date 20190101 \
  --bucket gcs-leanplum-export \
  --table-prefix leanplum \
  --bq-dataset dev_external \
  --prefix dev

Doing it this way will, by default, use your local GCP credentials. GCP only allows you to do this a few times

Alternatively, run in Docker.

First, create a service account with access to GCS and BigQuery. Download a JSON key file and make it available in your environment as GCLOUD_SERVICE_KEY. Then run:

bq mk leanplum
make run COMMAND="leanplum-data-export export-leanplum \
  --app-id $LEANPLUM_APP_ID \
  --client-key $LEANPLUM_CLIENT_KEY \
  --date 20190101 \
  --bucket gcs-leanplum-export \
  --table-prefix leanplum \
  --bq-dataset leanplum \
  --prefix dev"

That will create the dataset in BQ, download the files, and make them available in BQ in that dataset as external tables.

Development and Testing

While iterating on development, we recommend using virtualenv to run the tests locally.

Run tests locally

Install requirements locally:

python3 -m virtualenv venv
source venv/bin/activate
make install-requirements

Run tests locally:

pytest tests/

Run tests in docker

You can run the tests just as CI does by building the container and running the tests.

make clean && make build
make test

Deployment

This project deploys automatically to Dockerhub. The latest release is used to run the job.

You can’t perform that action at this time.