Skip to content

Tool for extracting Google Analytics data suitable for migrating to other platforms/databases

Notifications You must be signed in to change notification settings

MartinHeinz/ga-extractor

Repository files navigation

Google Analytics Extractor

PyPI version

A CLI tool for extracting Google Analytics data using Google Reporting API. Can be also used to transform data to various formats suitable for migration to other analytics platforms.

Also see - Goodbye, Google Analytics - Why and How You Should Leave The Platform for more context.


If you find this useful, you can support me on Ko-Fi (Donations are always appreciated, but never required):

ko-fi

Setup

You will need Google Cloud API access for run the CLI:

  • Navigate to Cloud Resource Manager and click Create Project

    • alternatively create project with gcloud projects create $PROJECT_ID
  • Navigate to Reporting API and click Enable

  • Create credentials:

    • Go to credentials page

    • Click Create credentials, select Service account

    • Give it a name and make note of service account email. Click Create and Continue

    • Open Service account page

    • Select previously created service account, Open Keys tab

    • Click Add Key and Create New Key. Choose JSON format and download it. (store this securely)

  • Give SA permissions to GA - guide

    • email: SA email from earlier
    • role: Viewer

Alternatively see following setup.

To install and run:

pip install ga-extractor
ga-extractor --help

Running

ga-extractor --help
# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...
# ...

# Create config file:
ga-extractor setup \
  --sa-key-path="analytics-api-24102021-4edf0b7270c0.json" \
  --table-id="123456789" \
  --metrics="ga:sessions" \
  --dimensions="ga:browser" \
  --start-date="2022-03-15" \
  --end-date="2022-03-19"
  
cat ~/.config/ga-extractor/config.yaml  # Optionally, check config

ga-extractor auth  # Test authentication
# Successfully authenticated with user: ...

ga-extractor setup --help  # For options and flags
  • Value for --table-id can be found in GA web console - Click on Admin section, View Settings and see View ID field
  • All configurations and generated extracts/reports are stored in ~/.config/ga-extractor/...
  • You can also use metrics and dimensions presets using --preset with FULL or BASIC, if you're not sure which data to extract

Extract

ga-extractor extract
# Report written to /home/some-user/.config/ga-extractor/report.json

extract perform raw extraction of dimensions and metrics using the provided configs

Migrate

You can directly extract and transform data to various formats. Available options are:

  • JSON (Default option; Default API output)
  • CSV
  • SQL (compatible with Umami Analytics PostgreSQL backend)
ga-extractor migrate --format=CSV
# Report written to /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv

head /home/user/.config/ga-extractor/02c2db1a-1ff0-47af-bad3-9c8bc51c1d13_extract.csv
# path,browser,os,device,screen,language,country,referral_path,count,date
# /,Chrome,Android,mobile,1370x1370,zh-cn,China,(direct),1,2022-03-18
# /,Chrome,Android,mobile,340x620,en-gb,United Kingdom,t.co/,1,2022-03-18

ga-extractor migrate --format=UMAMI
# Report written to /home/user/.config/ga-extractor/cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql

# IMPORTANT: Verify the data and check test database before inserting into production instance 
# To insert into DB (This should be run against clean database):
cat cee9e1d0-3b87-4052-a295-1b7224c5ba78_extract.sql | psql -Upostgres -a some-db

You can verify the data is correct in Umami web console and GA web console:

Note: Some data in GA and Umami web console might be little off, because GA displays many metrics based on sessions (e.g. Sessions by device), but data is extracted/migrated based on page views. You can however confirm that percentage breakdown of browser or OS usage does match.

Development

Setup

Requirements:

  • Poetry (+ virtual environment)
poetry install
python -m ga_extractor --help

Testing

pytest

Building Package

poetry install
ga-extractor --help

# Usage: ga-extractor [OPTIONS] COMMAND [ARGS]...
# ...

About

Tool for extracting Google Analytics data suitable for migrating to other platforms/databases

Topics

Resources

Stars

Watchers

Forks

Languages