Skip to content
This repository

Int overflow(?) in wilcoxonMatchedPairCriticalValue #18

Open
Shimuuar opened this Issue December 09, 2011 · 2 comments

1 participant

Aleksey Khudyakov
Aleksey Khudyakov
Collaborator

There is a bug in calculation of critical value for Wilcoxon-T test.

λ wilcoxonMatchedPairCriticalValue 35 0.05
Just 213
λ  wilcoxonMatchedPairCriticalValue 36 0.05
Just 2147483647   --  Which equal to Just (2^31-1) 

It looks like at some point integer overflow occurs.

Also it cannot handle samples larger than 1023 at all since it uses 2**sampleSize in calculation of critical value which overflows to +∞ for sampleSize > 1023

Test probably should fall back to normal approximation at some point.

Aleksey Khudyakov
Collaborator

Indeed on 64-bit machine critical point is at N=67

Aleksey Khudyakov
Collaborator

Use of Integers solves problem with integer overflows (commit e04da30) but wilcoxonMatchedPairCriticalValue should revert to normal approximation at some point because of double overflow and since algorithm is really slow for large N.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.