New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MannWhitneyTest fails when comparing statistics of different sample size #950

Closed
inkel opened this Issue Nov 9, 2018 · 9 comments

Comments

Projects
None yet
3 participants
@inkel

inkel commented Nov 9, 2018

Hi, everyone.

I've a simple benchmarks suite, where I'm trying to compare the speed of JSON serialization using three different providers (Newtonsoft.Json, Jil and ServiceStack.Text) and I consistently get the following exception when trying to run the whole suite:

Unhandled Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at BenchmarkDotNet.Mathematics.StatisticalTesting.MannWhitneyTest.PValueForSmallN(Int32 n, Int32 m, Double u)
   at BenchmarkDotNet.Mathematics.StatisticalTesting.MannWhitneyTest.IsGreater(Double[] x, Double[] y, Threshold threshold)
   at BenchmarkDotNet.Mathematics.StatisticalTesting.StatisticalTestHelper.CalculateTost[T](IOneSidedTest`1 test, Double[] baseline, Double[] candidate, Threshold threshold)
   at BenchmarkDotNet.Mathematics.RankHelper.AreSame(Statistics x, Statistics y)
   at BenchmarkDotNet.Mathematics.RankHelper.GetRanks(Statistics[] stats)
   at BenchmarkDotNet.Columns.RankColumn.GetValue(Summary summary, BenchmarkCase benchmarkCase)
   at System.Linq.Enumerable.SelectArrayIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at System.Linq.Enumerable.SelectArrayIterator`2.ToArray()
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Reports.SummaryTable..ctor(Summary summary, ISummaryStyle style)
   at BenchmarkDotNet.Reports.Summary..ctor(String title, IList`1 reports, HostEnvironmentInfo hostEnvironmentInfo, IConfig config, String resultsDirectoryPath, TimeSpan totalTime, ValidationError[] validationErrors)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String rootArtifactsFolderPath, StartedClock& runChronometer)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(BenchmarkRunInfo[] benchmarkRunInfos, IConfig commonSettingsConfig)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(BenchmarkRunInfo benchmarkRunInfo)
   at Benchmarks.Program.Main(String[] args) in C:\Users\inkel\source\repos\Foo\test\Benchmarks\Program.cs:line 19

Funny thing is that if I comment out any benchmark, leaving only two of them, I don't get the error anymore. Any ideas?

@wojtpl2

This comment has been minimized.

Collaborator

wojtpl2 commented Nov 9, 2018

It is the same problem like in #948
In version 0.11.2, MannWhitneyTest class has been added.
in the previous version 0.11.1 everything worked fine.

@inkel

This comment has been minimized.

inkel commented Nov 9, 2018

@wojtpl2 thanks. I'll try downgrading my installation and re-running the suite.

@inkel

This comment has been minimized.

inkel commented Nov 9, 2018

So far I can tell that running the suite is significantly faster and I haven't seen the same error. I'll run it a few more times so I can confirm things are fixed.

@wojtpl2

This comment has been minimized.

Collaborator

wojtpl2 commented Nov 9, 2018

I've created PR #951 with failing test.

@inkel

This comment has been minimized.

inkel commented Nov 9, 2018

@AndreyAkinshin AndreyAkinshin self-assigned this Nov 14, 2018

@AndreyAkinshin AndreyAkinshin added this to the v0.11.3 milestone Nov 14, 2018

@AndreyAkinshin

This comment has been minimized.

Member

AndreyAkinshin commented Nov 14, 2018

Should be fixed now.

@inkel

This comment has been minimized.

inkel commented Nov 14, 2018

@AndreyAkinshin any ideas when a new version will be released?

@adamsitnik adamsitnik changed the title from Unhandled exception System.IndexOutOfRangeException when running 3 benchmarks to MannWhitneyTest fails when comparing statistics of different sample size Nov 16, 2018

@AndreyAkinshin

This comment has been minimized.

Member

AndreyAkinshin commented Nov 20, 2018

@inkel v0.11.3 is already on nuget.org

@inkel

This comment has been minimized.

inkel commented Nov 20, 2018

@AndreyAkinshin awesome! I saw your announcement on Twitter today, that's great! Thanks 😸

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment