👩🚀 📈 🤖
The burnham project is an end-to-end test suite that aims to automatically verify that Glean-based products correctly measure, collect, and submit non-personal information to the GCP-based Data Platform and that the received telemetry data is then correctly processed, stored to the respective tables and made available in BigQuery.
Blog post about the proof of concept: https://raphael.codes/blog/automated-end-to-end-tests-for-glean/
For Mozilla, getting reliable data from our products is critical to inform our decision making. Glean is a product analytics & telemetry solution that provides a consistent experience and behavior across all of our products.
Find out more in our Data Documentation.
We have developed a command-line application based on the Glean SDK Python bindings for producing test data as part of the automated end-to-end test suite. The burnham application submits Glean pings to the Data Platform which validates and stores these pings in BigQuery tables.
You can find the code for the burnham application in the application
We also developed a test suite based on the pytest framework that
dynamically generates tests. Each test runs a specific query on BigQuery to
verify a certain test scenario. After the test session finished, we then
store the results in a designated BigQuery table with ID
The test suite code is located in the bigquery directory.
We build and push Docker images for both burnham and burnham-bigquery on CI
for pushes to the
main branch of this repository. The end-to-end test suite
is configured as a DAG on telemetry-airflow on the Data
Platform and scheduled to run daily. It runs several instances of a burnham
Docker container to produce Glean telemetry, uses Airflow sensors to wait for
the data to be available in the various burnham live tables, and then runs
burnham-bigquery to verify the results.
Please see the burnham DAG for more information.
We created two scheduled queries that read the results from the
burnham_derived.test_results_v1 table from the past 4 days. The queries run
daily at 02:00 UTC after telemetry-airflow ran the burnham DAG.
Mozilla employees can find this information in the burnham dashboard.
We successfully completed the proof of concept and are now running burnham
and burnham-bigquery in production. We are now actively working on developing
additional test scenarios.
Both burnham and burnham-bigquery run on Python 3.7 on Debian.
The burnham project has integration points with the Glean SDK team, the Data Platform team and the Ecosystem Test Engineering team.
Community Participation Guidelines
This repository is governed by Mozilla's code of conduct and etiquette. Please read the burnham Code of Conduct.
The project theme is inspired by Michael Burnham, the fictional protagonist
on the web television series Star Trek: Discovery portrayed by Sonequa
Martin-Green. Burnham is a science specialist on the Discovery. She and her
crew do research on spore drive technology and complete missions in outer
space and these themes of scientific exploration and space travel are a
perfect fit for this project.