Krippendorff Alpha implementation for std-ruby Matrix library
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
ext
lib
test
.gitignore
Gemfile
README.md
Rakefile
krippendorff_alpha.gemspec

README.md

Krippendorff Alpha

An implementation of the Krippendorff Alpha coefficient calculation for the Ruby standard Matrix library. Krippendorff's algorithm is used extensively in content analysis to determine inter-rater reliability.

https://en.wikipedia.org/wiki/Krippendorff%27s_alpha

Implementation

This implementation assumes the ratio difference function, though any function can be passed through the diff_func parameter as a Proc object accepting c and k values, as mentioned in the Wikipedia article above.

Example

m = Matrix[
   [ nil, nil, nil, nil, nil, 3, 4, 1, 2, 1, 1, 3, 3, nil, 3 ],    # coder 1
   [ 1, nil, 2, 1, 3, 3, 4, 3, nil, nil, nil, nil, nil, nil, nil], # coder 2
   [ nil, nil, 2, 1, 3, 4, 4, nil, 2, 1, 1, 3, 3, nil, 4 ]         # coder 3
]

puts m.krippendorff_alpha

Optimization

Since Matrix objects can get quite large, and some of the calculations for the Alpha score can have a significant computation cost associated with the multiple levels of recursion, we've implemented an FFI module in order to enjoy a profound speedup in performance.

Given the following:

require 'benchmark'
require_relative 'lib/krippendorff_alpha'

m = Matrix.build(35, 100) { |row, col| ((rand * 2) + 1).round }
puts Benchmark.measure { 10.times { m.krippendorff_alpha }}

We observe the following speedup:

Method user system total real
Ruby 84.350000 0.090000 84.440000 85.125188
FFI 0.010000 0.000000 0.010000 0.010492

References