Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
testing: investigate possible bias due to time estimation #27168
In #24735 (comment) I wrote:
This issue is to follow up on this concern, independently of #24735, which is really about something else.
referenced this issue
Aug 23, 2018
Still needs investigation. I think the theoretical concern is real; the real question is whether it matters in practice.
While I'm thinking of it, another simple fix for this would be to accept benchmark execution times that are with a range of the goal (say +/- 10%) rather than requiring that they be strictly >= the goal. That'd also reduce pressure to overestimate b.N (see also CL 112155 and #27217) and make benchmarks run faster overall.