A tool to check benchmark is within threshold up or down OR a tool for no change

[kealjones-wk]

See this comment for details on the idea for a tool within threshold up or down.

I was curious if it would be possible to add a tool for no change?

The use case is that I want to ensure that this benchmark is unchanged up or down and if there is ANY change at all in either direction I want an alert and fail.

Basically if the Ratio isn't 1 then alert/fail.

customNoChange? customEqualIsBetter?

[TomAFrench]

This sounds like more of a test than a benchmark surely?

[kealjones-wk]

I mean... sure it is a "test", but isn't that the point of having the feature of "fail on warning"?

But fair ok, maybe "No Change" is unreasonable, i just thought it might be easier to implement considering how it is coded right now. Maybe something more useful from a benchmark perspective would be a tool that would warn if it exceeds a threshold, up or down. To me, it seems like it would be helpful if changes in either direction of a certain amount could indicate some kind of abnormality that should be brought up.

Use case: We are benchmarking the amount of chunks in our js bundle over time. Smaller is indeed better but if it drops from like 54 to 2 that probably indicates that something is wrong and should at least be addressed.

[ktrz]

hey @kealjones-wk

I don't understand why you would need a new tool for that. Wouldn't it be sufficient to set alert-threshold to 100?