[Bug]: CI for Krippendorff's alpha

[ulrikesenftleben]

JASP Version

0.18.0

Commit ID

No response

JASP Module

Reliability

What analysis are you seeing the problem on?

Krippendorff's alpha

What OS are you seeing the problem on?

Windows 10

Bug Description

I've noticed that the confidence interval values for Krippendorff's alpha don't make any sense (i.e., don't include alpha) for my analysis.

I think the issue is that I have my data formatted so that raters are in rows and variables in columns and I have selected the corresponding option in the Krippendorff's alpha sub menu. If I select "raters are in columns" instead, the value of alpha changes, but the CI stays virtually the same (aside from minor variations due to bootstrapping).

Here is my original analysis with raters in rows:

Here is the same analysis with raters in columns (note how alpha changes, but SE and CI stay the same):

This leads me to believe that the calculation of the CI ignores the "raters in rows" argument and always assumes that raters are in columns.

Expected Behaviour

I would expect the SE and CI to change when going from "raters in columns" to "raters in rows" in the Krippendorff's alpha sub-menu.

Steps to Reproduce

1. run Krippendorff's alpha on a dataset that is formatted with raters in rows and select the option "raters in rows"

2. repeat the same analysis with "raters in column"

I've created a made up test data set with 4 raters and 14 items to recreate this issue (the first output is "raters in columns", the second output is "raters in rows", otherwise the data and analyses are the same):

testdata_krippendorffs_alpha.zip
testdata.csv

Log (if any)

No response

Final Checklist

• I have included a screenshot showcasing the issue, if possible.
• I have included a JASP file (zipped) or data file that causes the crash/bug, if applicable.
• I have accurately described the bug, and steps to reproduce it.

[juliuspfadt]

hi @ulrikesenftleben, thanks for reporting this bug. This is indeed an error in the code. I appreciate your advice. Should be fixed in the next release.

[juliuspfadt]

this is what I get with the fixed code for your analysis:

[ulrikesenftleben]

Hi @juliuspfadt, ok great, thanks for looking into this! The CI with the fixed code looks as expected, so that's great.

Any idea on when the next release will be available? This analysis is part of an undergraduate thesis, so it'd be good to know if it makes sense for my student to wait for the next release.

[juliuspfadt]

So if the PR is reviewed quickly, the revised code will make it into a nightly built, which is a sort of beta version that can be downloaded here. https://static.jasp-stats.org/Nightlies/ That should be fine for the thesis.

[ulrikesenftleben]

That's brilliant, thanks!