[SR-873] Python Benchmark Driver needs unit tests

[gottesmm]

Previous ID	SR-873
Radar	None
Original Reporter	@gottesmm
Type	Task

Additional Detail from JIRA

Votes	0
Component/s	Compiler
Labels	Task, StarterBug
Assignee	None
Priority	Medium

md5: 9053a294b75b04073c72f457567c4ddd

Issue Description:

In https://bugs.swift.org/browse/SR-871, we discovered that the upper level python benchmark driver was using lexicographical comparison in order to calculate the minimum/max value/etc.

We should add some basic sanity checking here in terms of a nose test or something like that.

[gottesmm]

Luke, I am going to unassign this in case any one else wants to try this before you get to it.

[twhitt14]

I think this can be closed. The test suite already includes tests that exercise min/max logic in the benchmark driver, especially in the test_benchmark_runtime_range and related tests in TestBenchmarkDoctor.

These tests:

• Create fake benchmark results with various min_value values (including edge cases like 0, 3, 200, 1001, 500000, etc.).
• Assert that the runtime analysis and reporting logic in the driver works as expected.
  Use numeric values that would expose lexicographical comparison bugs (e.g., 3 vs 200 vs 1001).