Repository navigation
STEP 5: INTERCODER RELIABILITY TESTING
Sample coding of select texts by two experienced coders. Iterative refinement of coding instructions until agreement is reached among expert coders. Determine which elements are subject to intercoder reliability testing and document reasons for decision (for example, is identifying institutional statements part of intercoder reliability or not; what about constitutive/regulative?)
After codebook development and after independently coding a sample transcript, coders on the coding team should assess intercoder agreement of their coding using simple percent agreement. Current recommendations suggest that any agreement between 80-85% is considered acceptable, although agreement in the lower range might warrant further codebook revisions and testing until higher agreement is reached. Once acceptable agreement is reached, coders commence coding all texts.
Study PI continues to randomly check intercoder agreement among coding team to ensure percent agreement remains within range during entire coding phase.
Once coding is complete, intercoder reliability testing on a random sample percentage (we recommend 20-30% of total coded statements) of the entire coded dataset is conducted using a test that accounts for chance (Cohen’s kappa, Krippendorff’s alpha).