Store test run results along with report 📝

[hackebrot]

We need to store the test report and test run information from the burnham-bigquery Docker image somewhere.

Let's discuss a good approach for that.

[jklukas]

Do you have a good sense of what the test output will look like? My go-to solution is that we provision a destination table in BQ for test results and the burnham-bigquery image can be responsible for writing there. Even if the output is basically a block of text, we can dump that into a STRING field.

[hackebrot]

• When we run the tests on Airflow, we will generate a UUID for the test run and pass it to burnham, which stores it in metrics.uuid.test_run, and also pass it to burnham-bigquery to query data for this run
• Every test scenario will generate a test item with a unique name, which is stored in pings under metrics.string.test_name, and that test item has an outcome (passed, failed, error etc.) and output
• It might be useful to also keep track of the Glean SDK version used to produce the data for a test run

We can produce test reports in Markdown using https://github.com/hackebrot/pytest-md

[hackebrot]

What do you think about this table schema?

This would be a row for each individual test.

Field name	Type	Description
test_run	STRING	ID of the current test run
test_name	STRING	Name of the current test.
test_outcome	STRING	Outcome of the current test. (see below)
test_duration	FLOAT64	Duration of the current test.
test_report	STRING	Report for the current test for error or failed tests

Possible outcomes:

• ERROR
• FAILED
• PASSED
• SKIPPED
• XFAILED
• XPASSED

Does that make sense or do you think it would be better to write one row for the entire test run and use a record to store results for individual tests?

[jklukas]

I think the flatter structure you propose will be easier to work with than a nested one. The data size will be small, so we don't have to worry about making the representation efficient. This looks good to me.

[hackebrot]

@jklukas can you provision a destination table for this in BigQuery, please?

[jklukas]

Proposed table structure in mozilla/bigquery-etl#1220

[hackebrot]

Thank you @jklukas! I now realize that we probably also need to link to the Airflow job or the logs for the burnham operators, so that we can diagnose client-side issues for failed test runs. 🤔

[jklukas]

we probably also need to link to the Airflow job or the logs for the burnham operators, so that we can diagnose client-side issues for failed test runs

I think it would make sense to include a test_log_url field. We should have all the relevant data via Airflow variables. Links look like:

https://workflow.telemetry.mozilla.org/log?task_id=verify_data&dag_id=burnham&execution_date=2020-08-06T00%3A00%3A00%2B00%3A00

We should be able to pull this using Airflow macros as {{ task_instance.log_url }} and pass it into the task. We just need to make sure it's passed as a parameter that's going to have templating applied.

[hackebrot]

Perfect! 👍