Regression in Rff Code due to OVA + AveragedPerceptron

[Anipik]

I ran a benchmark provided to me By @justinormont that uses Rff Transformer.
I ran it too on master as well as on 7d9660a
and 8b19930

The time taken by the benchmark for both of the last two commit was around 850 ms for both netstandard as well as netcoreapp 3.0 implementation where as it was around 1.3 sec for the master branch.

I am working on adding the benchmark into the repo.
The rsp being used here is

cc @adamsitnik @eerhardt @danmosemsft @shauheen

[sfilipi]

Has any of the default parameters changed?

[Anipik]

The rsp being used here

[justinormont]

Do we currently have a dashboard of the runtimes of the current benchmarks?

Perhaps we should add check-in gating based on performance tests? @markusweimer

[adamsitnik]

@Anipik you can use the EtwProfiler to profile before and after and then use PerfView to compare both trace files for regressions.

1. Run the benchmark without the changes with --profiler ETW, save the path to the trace file
2. Run the benchmark with the changes with --profiler ETW, save the path to the trace file
3. Open both trace files in PerfView
4. Filter both trace files to single benchmark iteration like described here to have apples-to-apples comparison
5. Disable the module grouping (GroupPats = [no grouping])
6. Load the symbols: go to the By name tab, and press Ctrl+A (select all) and then Alt+S (load symbols).
7. Once you have more than one trace file opened in PerfView, a new menu item appears in the "Regression".
   
8. Go to the Window with the trace file for the slower result, choose Regression => With Baseline (it should point to the faster result)
9. You should get a table with regressions
   
10. To get better understaning of what all these numbers mean go to Help->User guide and hit Ctrl+F and search for "Regression Investigation with Overweight Analysis "

[Anipik]

There are 2 regressions in the code the first one is #1512 and the other is 435a63b

the first one changed the benchmark value from 850 ms to 1.4 s
The second one changed to value for, 900ms to 3 sec

cc @abgoswam @wschin @danmosemsft @eerhardt

[adamsitnik]

@Anipik great job on finding the root cause of the regressions!

[abgoswam]

@Anipik There is one more bug fix for the OVA + AveragedPerceptron scenario, related to the data being used for calibration. Please check out #1805

[Anipik]

@abgoswam I ran the benchmark for #1805 The particular commit actually decreased the time from 1.3sec to 900 ms. which is almost same as 850ms. The regression due is about 6% due to this.

@danmosemsft @eerhardt should we keep this open ? or is it okay to close them. I have opend the separate issue for the cache mechanism regression

cc @yaeldekel