2x Regression in string join benchmark on NanoSoldier

[Zentrik]

The last couple of commits have seen a ~2x regression on NanoSoldier in min wall time for the string join benchmark, graph. Locally, I actually see an improvement, but given the regression has persisted for a few runs I'm inclined to think it's real.

[giordano]

The last couple of commits

It'd be good to leave a permanent reference to the commit when you have it handy, otherwise someone who wants to look into this will have a hard time to figure out exactly what were the offending commits you were referring to. A range could be guessed by checking the time of your message and comparing it with the history of the repository, provided that there aren't many commits on the same day and one doesn't get lost between the different timezones, but if you have the information it's just much easier for everyone if you can share it

[Zentrik]

Last good commit was 5c7d244. First detected bad commit was 8e7751c and the following commits also show the regression, 9ae7eab, 0f04b33, 2775c9a, 9df47f2 and 5c891de.

[jishnub]

I'm curious why 8e7751c leads to this regression. In any case, should we revert that to see if the regression gets resolved?

[Zentrik]

Sorry to be clear the offending commit is probably after 5c7d244 up to and including 8e7751c. We don't run NanoSoldier on every commit so we don't know which commit actually caused the regression just that 8e7751c is the first commit benchmarked including the regression.

[giordano]

In other words (or links), somewhere in 5c7d244...8e7751c

[adienes]

per the graphing tool above the specific benchmark seems to have been fixed since early November, but it would probably be good to add a test to make sure it doesn't re-regress