Skip to content
Code to gather data and generate graphs for timing measurements
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
measuring_ci
one_offs
requirements
.gitignore
.isort.cfg
.travis.yml
CODE_OF_CONDUCT.md
Dockerfile
README.md
cost_by_taskgraph.py
create_lambda_func.sh
create_lambda_func_in_docker.sh
gather_tc_aws_costs.py
graph_analyzer.py
nightlies.yml
nightly_scanner.py
parquet_collator.py
pushlog_backfill.py
pushlog_scanner.py
pushlog_scanner_lambda_backfill.py
releases.yml
releases_scanner.py
scanner.yml
tox.ini
update_costs.yml

README.md

Measuring CI

The Mozilla Release Engineering team is producing some estimates of our Continous Integration costs.

This repository contains a work-in-progress for the scripts used to gather and process the data.

Current Architecture

pushlog_scanner.py and its dependencies are turned into a .zip file for use in an AWS Lambda function. This gives us a place to run the script periodically without worrying about other infrastructure. The lambda function's role gives it the appropriate access to an S3 bucket for the results.

The S3 is connected to Athena, which allows use of SQL-style queries against it. Re:dash has a read-only api key to allow it access, and from there we can make queries and dashboards.

Updating the Lambda Function

  1. Clone the repository
  2. Ensure Python 3.6 is installed (AWS Lambda currently only supports 2.7 and 3.6)
  3. Run create_lambda_func_in_docker.sh If running on Linux, create_lambda_func.sh will do. Some modules require compilation, and the target platform is Linux, so installing natively on Mac/Windows is not going to work.
  4. Copy the resulting zip file to S3: aws s3 cp measuring_ci.zip s3://mozilla-releng-metrics/measuring_ci.zip
  5. Visit the Lambda function's config page (If needed: ARN - arn:aws:lambda:us-east-1:314336048151:function:measuring_ci_parquet_update)
  6. Under 'Function code' choose a 'Code entry type' of 'Upload a file from Amazon S3' and paste the above s3 url into the box.
  7. Ensure the Handler is set correctly if not using lambda_handler() in lambda_function.py
  8. Under 'Basic Settings' ensure the Memory usage is at 512Mb and Timeout is at least 2 minutes.
  9. Click 'Save' at the top of the page
  10. Test the lambda function using the 'Test' button. The test event itself doesn't matter as we're not using its data. If a test event is not defined, the basic 'Hello world' template will do.

Planned Updates

The direct querying of a parquet file in S3 is a short-term step in order to get data visibility. Longer term we will want to use the generic data ingestion method provided by the data team.

You can’t perform that action at this time.