Question about correlated agreement statistics when the subjects are different in the two periods

[bzki]

Thanks for your work on this package and the methodology. My understanding is that these functions work when you have the same raters in 2 periods and the same subjects being rated in both periods (as the name of the package implies, analogous to a paired t-test). Are you aware of methods or software implementations that can handle the following extension:

• Different subjects being rated in the two periods.
• Not every subject being rated by every rater. For example, raters 1, 2 and 3 rating subject A and B, raters 4, 5 and 6 rating subject C and D, and so forth.
• 2 groups of raters of different sizes, with overlap in the 2 periods (e.g., maybe due to attrition, we have 10 raters in the first period and 8 in the second, and the 8 are in common to the two periods). This aspect is less important than the above two points.

One can estimate agreement statistics in 1 period using Edwards/Nelson GLMM-model based agreement stats or Gwet's AC1/AC2 (e.g., using your package irrCAC or the SAS macro MAGREE). But I'm not sure there is a way to estimate correlation of the agreement statistics in 2 periods in order to estimate a standard error and confidence interval for the difference. Do you have any suggestions or ideas for this?