Python implementation of the Cross-Replication Reliability Measure, an IRR score assessing reliability and agreement between responders, specifically developed for crowd sourced answers.
Based on Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability.
See demo.ipynb for explaination.
Official implementation (in Java)