Skip to content

A webservice that produces Mozilla code repository telemetry.


Notifications You must be signed in to change notification settings


Repository files navigation

Mozilla Commit Telemetry Service

A simple queue worker that produces Mozilla code repository telemetry.

The bin/process-queue-messages script reads messages about Mozilla source code pushes from the Mozilla Pulse messaging service. It adds some data about the code review system used by the commit author and submits the data to where we can build nifty dashboards.


You will need to create an account on Mozilla Pulse to collect messages about hgpush events.

These programs were designed to run on Heroku and follow the Heroku Architectural Principles. They read their settings from environment variables.

See the file dotenv.example.txt in the project root for possible values. The values that must be present in your local and/or heroku execution environments:

$ cp dotenv.example.txt .env
$ vim .env
# Add your personal environment's configuration

Run the following command to check that everything works. It won't send any data:

$ PYTHONPATH=. bin/process-queue-messages --no-send

Local Usage

$ PYTHONPATH=. bin/process-queue-messages

Read all push messages from the hgpush event queue, figure out which review system was used for each, and send the result to

Use --help for full command usage info.

Use --debug for full command debug output.

Use --no-send to gather all the data and build a payload, but do not send any real pings. All push event messages remain in queues, too. This is great for testing changes or diagnosing problems against a live queue.

$ PYTHONPATH=. bin/dump-telemetry SOME_COMMIT_SHA1

Calculate and print the ping that would have been sent to for a given changeset ID. This command does not send any data to Useful for debugging troublesome changesets and testing service connectivity.

Use --help for full command usage info.

Use --debug for full command debug output.


Read the Mercurial repository pushlog at REPO_URL, fetch all pushes from STARTING_PUSHID to ENDING_PUSHID, then calculate and publish their telemetry. This can be used to back-fill pushes missed by service gaps.

Use --help for full command usage info.

Use --debug for full command debug output.

Use --no-send to gather all the data and build a payload, but do not send any real pings.


Environment setup

Use pyenv to install the same python version listed in the project's .python-version file:

$ pyenv install

Set up a virtual environment (e.g. with pyenv virtualenv) and install the project development dependencies:

$ pip install -r requirements.txt -r dev-requirements.txt


Code formatting is done with black.

requirements.txt and dev-requirements.txt are updated using hashin.

Push event messages are read from a Pulse message queue. You can inspect a live hgpush message queue with Pulse Inspector.

Messages use the hgpush message format.

Push events are generated from the mercurial repo pushlogs.

Pings (telemetry data) are sent to TMO using the hgpush ping schema. Make sure you match the schema or your pings will be dropped!


Automated tests

The unit test suite can be run with py.test.

Manual tests

Manual testing can be done with:

$ PYTHONPATH=. bin/dump-telemetry --debug  <SOME_CHANGESET_SHA>


$ PYTHONPATH=. bin/process-queue-messages --no-send --debug

If you need a message queue with a lot of traffic for testing you may want to listen for messages on integration/mozilla-inbound. To switch the message queue set the following environment variables:


Testing ping schema changes

After deploying a schema change check these monitors:

You can also write custom monitors using hand-crafted CEP dashboards.

Ask in #datapipeline on IRC if you need help with this.


A webservice that produces Mozilla code repository telemetry.



Code of conduct





No releases published


No packages published