Skip to content

contentvalidR 0.3.0

Choose a tag to compare

View all tags
@JUhalt JUhalt released this 15 Sep 12:33
v0.3.0
6831ea6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Third public release, and the first prepared for CRAN.

v0.3.0 is built on evidence. Every addition rests on published, verifiable methodology, and one design rule runs through the release: where more than one published method exists, you choose through an argument whose default is the best-supported option. Methods with published evidence against them stay available so earlier work can be reproduced, but they are never the default, and choosing one prints the critique.

The package still declares Imports: stats only, so it installs without a compiler toolchain.

Install

install.packages(
  "contentvalidR",
  repos = c("https://juhalt.r-universe.dev", "https://cloud.r-project.org")
)

Two defaults change results

  • qfactor_content() now chooses the number of factors by parallel analysis. Use retention = "kaiser" to reproduce earlier results.
  • expert_validity() relevance mode now bootstraps an interval for panel agreement. Set seed when printed output must be reproducible.

Other existing results are unchanged. The new interval and agreement columns are additions.

Intervals for I-CVI and Psa

I-CVI and Psa are proportions from small panels, and until now they appeared as bare numbers. cvi() and compute_psa() now report an interval for each, and expert_validity() and sort_validity() carry it through. The Wilson score interval is the default, following Newcombe (1998); Agresti-Coull and Clopper-Pearson exact intervals are available when a study needs to match earlier work.

Output names the method and level, and says what an interval cannot tell you: it reflects how few ratings an item received, not whether the right judges were chosen. With eight experts, an item every expert rated relevant still has a lower limit well below 1.

Panel-level agreement

panel_agreement() reports one coefficient for how consistently a panel rated the whole item set, and expert_validity() relevance mode now includes it. Modified kappa still describes each item; this is the panel-level view it never gave.

  • Krippendorff's alpha is the default, at a measurement level you choose: ordinal (default), nominal, or interval. It handles any number of experts and missing ratings, which Zapf et al. (2016) recommend it for.
  • Gwet's AC1 is available but never the default. Vach and Gerke (2023) show it rises as ratings concentrate in one category even when agreement does not change, and choosing it prints that critique.
  • The interval resamples whole items with all of their ratings, following Zapf et al. (2016), who found Krippendorff's original bootstrap reached only about 60% coverage.

Alpha can be low on a panel that agrees closely. When nearly every rating is the same value, chance predicts very little disagreement, so a few dissenting ratings pull alpha down. The output reports the share of identical rating pairs next to alpha and says plainly that a low alpha with many identical pairs is not by itself evidence of a poor panel. In the README's three-item example, alpha is near zero while 71% of rating pairs are identical.

Factor retention for Q-factor content adequacy

qfactor_content(), which implements the content-adequacy approach of Schriesheim et al. (1993, 1999), used Kaiser's eigenvalue-greater-than-1 rule to pick the number of factors. Zwick and Velicer (1986) found that rule severely overestimates the number of components. The default is now Horn's (1965) parallel analysis, simulated with the same size and missing cells as your ratings and reproducible with seed. signal_detection() now cites the Anderson and Gerbing (1991) validation design it follows.

Verification

  • Krippendorff's alpha reproduces the worked example in Krippendorff (2011) at all three measurement levels (.743, .815, .849) and matches irr::kripp.alpha() to 1e-10.
  • AC1 reproduces the Wongpakaran et al. (2013) table, and its handling of missing ratings matches Gwet's own implementation.
  • Wilson and exact intervals match base R's prop.test() and binom.test().
  • Parallel analysis recovers a simulated two-factor structure; on pure noise, the Kaiser rule retains on average more than three factors beyond it.
  • R CMD check --as-cran is clean across Windows, macOS, and Linux on R release, oldrel and devel, including a job that treats any NOTE as a failure. A clean-library install of the built tarball runs the examples for every help topic, all seven vignettes, and the citation.

Notes

CRAN submission follows this release and is tracked in #14. Evidence-reviewed methods that did not meet the bar for v0.3.0, including Bayesian extensions, COSMIN appraisal, Delphi consensus, and the 95th-percentile parallel analysis criterion, are recorded with their reasons in the Future research milestone.

Full changelog: https://juhalt.github.io/contentvalidR/news/index.html

Assets 2
Loading