results database

[amcnicho]

The adopted "dependency-injection" method of installing casatasks results in a record identified by different environments for each wheel tested, but it’s all the same package. Earlier usage stripped the path and build numbers from some parts of the output JSON, but kept the major and minor revision number in the environment name.

Example

The relevant part of the output file results/ab8e0751-virtualenv-py3.6-pip+casatasks6.4.0.2.json generated by an asv run configured with "include": [{"python":"3.6", "pip+casatasks":"6.4.0.2"},] specified inside asv.conf.json:

"params": {"arch": "x86_64", "cpu": "Intel(R) Xeon(R) CPU E5-1660 v4 @ 3.20GHz", "machine": "NRAO workstation", "num_cpu": "16", "os": "Linux 3.10.0-1160.15.2.el7.x86_64", "ram": "32GB", "python": "3.6", "pip+casatasks": "6.4.0.2"}, 
"requirements": {"pip+casatasks": "6.4.0.2"},

changed to

"params": {"arch": "x86_64", "cpu": "Intel(R) Xeon(R) CPU E5-1660 v4 @ 3.20GHz", "machine": "NRAO workstation", "num_cpu": "16", "os": "Linux 3.10.0-1160.15.2.el7.x86_64", "ram": "32GB", "python": "3.6", "pip+casatasks": "6.4.0"}, 
"requirements": {"pip+casatasks": "6.4.0"},

to allow asv publish to recognize the samples as forming a continuous trend for a single code base.

When the package is benchmarked for different branches and platforms (and actual combinations of dependencies!) this will quickly become unwieldy. The asv publish (used for generating long-term trends and producing plots/HTML from results database) and asv compare (used for calculating deltas with some threshold between results) commands also require different dictionaries from the results JSON to be sanitized to facilitate this easy representation.

As the number of test results grows it is also possible that the collection of results will exceed repository size limitations, so it may eventually be necessary to manage and maintain a separate results database instead of storing them in the VCS.