Change σ² calculation #25
Conversation
After removing the superfluous square root from the σ² calculation, the resulting CI based on that should be more in line with what is commonly calculated. The example in README.md should also be updated. The bootstrap and normal distribution-based CI's should very likely be different, but not by as much as they are now. Adding the option to use a (noncentral) t-distribution might be good as well (e.g. Hedges LV; Distribution Theory for Glass's Estimator of Effect Size and Related Estimators; https://doi.org/10.2307/1164588 or Cumming G, Finch S; A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions; https://doi.org/10.1177/0013164401614002)
The example CI changes because of the new σ² calculation. Notably the estimate is also different, likely due to changes in the random number generator/seeding mechanism. The current example is produced with Julia 1.7.1 on macOS and on Linux.
Use SpecialFunctions.jl, to be able to use the gamma function in the correction calculation
Use the gamma function "gamma(df / 2) / (√(df / 2) * gamma((df - 1) / 2))" when n < 40 (df = n - 2 for two independent groups) instead of the approximation "1−(3/(4*df−9))"
SpecialFunctions supplies the gamma function for the correction factor calculation
Cohens D -> no correction Hedges g -> correction factor
Updated the example after the modification of the estimation functions.
|
Thank you for this PR.
How can I verify that the new calculation is better than the old one? What reference did you use?
Extending the features of this package is not a priority for me. I'm generally more fan of Bayesian statistics so rather focus on that. However, if you create super solid PRs which are easy to verify and hard to refute, then I'm more than happy to merge |
|
Thank you for your quick response. I wondered why the bootstrap differed so much from the estimate in the example in README.md. That's why I looked a bit further at this. I used the calculations mentioned on https://en.wikipedia.org/wiki/Effect_size and verified them as far as I could with the references mentioned there (I know... but still) and I also looked at the Stata documentation (https://www.stata.com/manuals/resize.pdf). After the modifications, I compared a few estimates between EffectSizes.jl, Stata , and the R packages effectsize and effsize. They appear to match better after the modifications (the confidence intervals do not match for various reasons). Still not perfect, but I believe it is an improvement. |
|
Okay, thank you for providing additional information. Could you make sure that the tests pass? The CI run indicates that there is a syntax error in the PowerAnalyses.jl module. |
Test fails as pooledstd1 and pooledstd2 are not available anymore after changes
Update numbers after changes
The changes correction factor calculation results in -Inf. probably a bit dirty to not handle this in the input to the function. This should be improved at some point.
For some reason the test output differs slightly from the output when executing the command from the REPL environment.
The changes correction factor calculation results in -Inf. probably a bit dirty to not handle this in the input to the function. This should be improved at some point.
Change test for correction function
|
The tests should work now. The handling of a very low sample size should be reviewed again at some point. |
|
Apparently they do not... Locally it passed all tests... Unfortunately I do not have a lot of experience with Julia/testing/github... |
For me the tests work too. I've set it to Julia 1.7 now. Maybe that was the culprit. |
|
Thank you for your work! :) |
|
Looks like that had to do with it indeed. Maybe the range allowed SpecialFunctions versions should be a bit extended (don't know if that could have to do with the error) |
|
Nice to be able to practice Julia a bit, and hopefully improve this package in doing so. But I suppose especially concerning the confidence intervals a lot of work still has to be done. https://cran.r-project.org/web/packages/effectsize/ and https://cran.r-project.org/web/packages/effsize/ might be good inspiration. |
I think that you used some Julia 1.7 specific syntax which wasn't valid in 1.6.
Yes I know that this package is far from perfect. Luckily, based on the stars almost no one is using it in the first place. At least things should be slightly better after today :) |
It looks like there was one square root to much in the calculation of the confidence interval.
An additional thing to contemplate: is it worth the effort to add the option to use a (noncentral) t-distribution (e.g. Hedges LV; Distribution Theory for Glass's Estimator of Effect Size and Related Estimators; https://doi.org/10.2307/1164588 or Cumming G, Finch S; A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions; https://doi.org/10.1177/0013164401614002) in the calculation?
I additionally propose to calculate Cohen's D without any correction factor, and Hedges G with the approximation by Hedges or the gamma function.