-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The HodgesLehmann mean appears to have a bug #97
Comments
This is indeed a strange behaviour. The HL-estimator in DescTools uses the estimate generated by the base R function wilcox.test and this function yields exactly this result.
I have to investigate this further... |
This is doubly strange if it uses a base R function! Thank you very much for looking into it. In the meanwhile I am using the corresponding function from rt.test.
Sincerely
Tom
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: Andri Signorell ***@***.***>
Sent: Saturday, January 14, 2023, 3:33 AM
To: AndriSignorell/DescTools ***@***.***>
Cc: tkpmep ***@***.***>; Author ***@***.***>
Subject: Re: [AndriSignorell/DescTools] The HodgesLehmann mean appears to have a bug (Issue #97)
This is indeed a strange behaviour. The HL-estimator in DescTools uses the estimate generated by the base R function wilcox.test and this function yields exactly this result.
wilcox.test(c(0.7, 0.5, 0.5), exact=F, conf.int=T, conf.level = 0.95)
I have to investigate this further...
—
Reply to this email directly, view it on GitHub<#97 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADKDO3Z6333ETN6XBBJSLG3WSJQE3ANCNFSM6AAAAAASW6OXKI>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Hello Andri, Sincerely Tom |
Not yet, sorry, I keep it open... |
I did some literature research. It turns out, that existing of tied values forces wilcox.test to switch to the approximated algorithm that returns strange results. I found the following alternative implementations for the HL-estimator:
where rQCC turns out to perform best, the exact wsrTest() has massive overhead for just computing the estimator and so has wilcox.test. So I will change the implementation to one of the candidates, resp. try to add some engineering. So, stay tuned, or better yet, collaborate! ;-) |
Hello Andri, Separately, there is an robust t-test package, rt.test, available at CRAN (https://cran.r-project.org/web/packages/rt.test/index.html) and it would be worth looking at their implementation as well. Sincerely Tom 616.txt |
This is an excellent presentation, thank you! I also did some research myself, Monahan's algorithm seems to be the only available suggestion. |
Oh, btw https://cran.r-project.org/web/packages/rt.test authors use the known O(N^2) outer(x, y, "-") approach. |
I'm glad you found it useful. I'm afraid I have not ever used C++, but even so, I'm happy to work with you on implementing Monahan's algorithm. Sincerely Tom |
Hello Andri, Sincerely Tom |
@tkpmep, @CyrilFMoser: Motion regarding Monahan's algorithm. The algorithm was implemented in version 0.99.52 by Cyril in C++. In a further step we will try to complete the hodges-lehmann-sen estimator for 2 samples including the respective confidence intervals. |
Thank you very much Andri and Cyril. I updated all the packages in my R 4.3.1 installation (DescTools is now at 0.99.52), and tested it with the original reprex, but got an error message: DescTools::HodgesLehmann(c(0.7, 0.5, 0.5)) If it is any help, here is what sessionInfo() gives:
Matrix products: default locale: time zone: Asia/Calcutta attached base packages: other attached packages: loaded via a namespace (and not attached): Would you kindly look into this? Many thanks in advance Tom |
Ohh, what a mess! An untested last minute change causes that, sorry for the inconvenience! I will fix that as soon as possible with CRAN. |
Just tried it again and it works perfectly with y=1 and the error message makes perfect sense for y=2. Thank you very much and I look forward to the two sample test as well. By the way, Sebastian Krantz is building a set of fast C++ packages for data processing called the fastverse (https://fastverse.github.io/fastverse/) that is built around data.table, collapse (which i have used, see https://sebkrantz.github.io/collapse/), kit and magrittr, and he might be happy to get a copy of your C++ implementation to add a powerful new robust estimator of location to collapse and kit. Would you be OK with me telling him to reach out to you for some help with the code etc.? One stated objective of the fastverse is stable syntax (the tidyverse tends to change syntax over time, requiring maintenace). The attached pdf goes back to 2003, and has the asymptotic relative efficiency of the Hodges-Lehmann Estimator (i.e. relative to the MLE) as a measure of location for t distributions with varying degrees of freedom, all the way from Cauchy to Normal (see the table on the last page). Its performance is truly remarkable, and, in fact, I routinely use the Hodges-Lehmann Mean in place of the sample mean as its efficiency is almost always higher than that of the sample mean, and it gives more than adequate (1-1/sqrt(2) ~ 29%) protection against outliers as a bonus. Sincerely Tom |
…sion 0.99.52 DescTools 0.99.52 (2023-11-30) ------------------------------ NEW FUNCTIONS ADDED: * dTri(), pTri(), qTri() and rTri() return the specific values for a triangular distribution. UPDATED FUNCTIONS: * New C++ port of the Hodges-Lehman estimator following Monahan's algorithm https://dl.acm.org/doi/10.1145/1271.319414. (Credits go to Cyril Flurin Moser) (Two sample case and confidence intervals are still pending.) (AndriSignorell/DescTools#97) BUGFIXES: (NEWS truncated at 15 lines)
Cyril's impressively productive! We now have the 2-sample estimator:
|
Terrific! Thank you very much for fixing this. It appears the second argument (i.e. 1 or 2) is not needed any longer. I assume this will be on GitHub in a day or two and I will test it again. It might be worth sharing this with the Base R team as well, as wilcox.test is so widely used. |
yes, on its way to github... |
Hello Andri, Sincerely Tom |
Hello Andri Tom |
Well done, we currently work on the confidence intervals and may come to a solution soon. If we get there this year we will include it, in any case there will be a new DescTools version this year fixing the H-L estimator. |
@mmaechler: @CyrilFMoser has done an excellent job. Referring to @tkpmep's comment above, this might be a good addition to wilcox.test, especially as the calculation is actually really fast. |
Hello Andri,
A possible improvement for the Mann-Whitney test implementation is using the Edgeworth expansion: Sincerely Tom While using the Hodges Lehmann Mean in DescTools (DescTools::HodgesLehmann), I found that it generated incorrect answers (see #97). The error is driven by the existence of tied values forcing wilcox.test in Base R to switch to an approximate algorithm that returns incorrect results - see https://aakinshin.net/posts/r-hodges-lehmann-problems/ for a detailed exposition of the issue. Andri Signorell and Cyril Moser have a new C++ implementation of DescTools::HodgesLehmann using a O(N log(N)) algorithm due to Monahan, but wilcox.test in Base R appears to be still broken. Will someone kindly bring this observation, as well as the existence of a solution, to the attention of the relevant person(s) in the Base R development team? The paper by Mohanan, as well as the original Fortran implementation of the algorithm are linked to from #97. Inefficient O(N^2) algorithms for the Hodges-Lehmann mean are known and are implemented in a variety of packages. For example, the authors of rt.test (https://cran.r-project.org/web/packages/rt.test) use the O(N^2) approach. I suspect that Andri and Cyril will be more than happy to assist with fixing wilcox.test in Base R with their implementation of Monahan’s fast algorithm. Sincerely Thomas Philips |
Just updated to 0.99.53 from CRAN. Thanks as always! |
Hey all--Andreas Loeffler brought this discussion to my attention today following up on our previous discussions on improving the base I'll make sure that the discussion gets forwarded onto R-devel with participants here cc'd; I really don't know why it didn't make it to the mailing list. I'm also planning to spend the next week or two working on continuing to patch the (numerous) issues with wilcox-family functions that Andrey has comprehensively outlined on his blog (the same list linked above by Thomas). A C++ implementation won't be directly usable in base R, but I'm sure we could port the Mohanan implementation to R/C for it. Worst case, that O(N^2) implementation isn't awful, I'm just concerned about exhausting vector memory in the -Aidan |
Hello Aidan,
Thanks a mill for taking this forward, and thank you Andri for sharing the work that Cyril and you did. I have no idea why my original post did not get forwarded to r-devel: it may have fallen through the cracks or it may have been a reflection of the fact that I am not a developer. In any event, thanks for bridging the gap.
Separately, I'm pretty upbeat about Monahan's O(N log N) algorithm for three reasons;
1. Faster speed,
2. Reduced memory usage, and
3. Enhanced numerical stability / accuracy on account of fewer arithmetic computations.
Sincerely
Tom
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: Aidan Lakshman ***@***.***>
Sent: Monday, February 12, 2024 1:11:02 PM
To: AndriSignorell/DescTools ***@***.***>
Cc: tkpmep ***@***.***>; Mention ***@***.***>
Subject: Re: [AndriSignorell/DescTools] The HodgesLehmann mean appears to have a bug (Issue #97)
Hey all--Andreas Loeffler brought this discussion to my attention today following up on our previous discussions on improving the base *wilcox functions<https://bugs.r-project.org/show_bug.cgi?id=18655>. As far as I'm aware, I haven't seen any discussion of this on R-devel or activity on Bugzilla regarding these issues, so I'm pretty sure it hasn't been brought up.
I'll make sure that the discussion gets forwarded onto R-devel with participants here cc'd; I really don't know why it didn't make it to the mailing list. I'm also planning to spend the next week or two working on continuing to patch the (numerous) issues with wilcox-family functions that Andrey has comprehensively outlined on his blog (the same list linked above<#97 (comment)> by Thomas).
A C++ implementation won't be directly usable in base R, but I'm sure we could port it the Mohanan implementation to R/C for it. Worst case, that O(N^2) implementation isn't awful, I'm just concerned about exhausting vector memory in the outer() call.
-Aidan
—
Reply to this email directly, view it on GitHub<#97 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADKDO36GMLN3BHR7DUEI5KLYTJLLNAVCNFSM6AAAAAASW6OXKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZZGI3TONBQHE>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Hello @ahl27, Thanks for your interest in my blog posts! It would be awesome if we could extend the *wilcox family with the Edgeworth approximation. At the moment, I can't rely on the standard
I will be happy to help with brainstorming, implementing, and testing. Don't hesitate to reach out to me at andrey.akinshin@gmail.com. Speaking of the documentation, I can also publish an arXiv preprint that covers all the details so that we can refer to it (the blog posts just briefly describe the idea, but they lack context and examples). |
Sounds good! I'm in the process of putting some patches together--I have a small github repo opened here just to organize my own thoughts for the time being. I was able to quickly put together a patch to implement the Edgeworth approximation for 2-sample I'll email R-devel momentarily so we can start a discussion on the points you're raising--I think they're all valid and worth consideration. I think going forward we can move discussion to the R-devel mailing list and Bugzilla, since it'll be easier to loop in other people in R. |
@ahl27 awesome, thanks for the update! Could you please include me in these discussions? (I've never participated in the R-devel mailing list before and I don't have an R Bugzilla account) |
The HodgesLehmann function appears to give the wrong answer, even in simple cases:
Reprex:
Desctools::HodgesLehmann(c(0.7, 0.6, 0.5)) = 0.6. This is correct - it is the median of (0.7, 0.6, 0.5, 0.65, 0.6, 0.55), which is 0.6 (average of 0.6 and 0.6).
But
Desctools::HodgesLehmann(c(0.7, 0.5 0.5)) = 0.5828427. This is incorrect - it should be the median of (0.7, 0.6, 0.6, 0.5, 0.5, 0.5), which is 0.55 (average of 0.6 and 0.5).
The text was updated successfully, but these errors were encountered: