contentvalidR 0.8.0

Eighth public release. v0.8.0 is about reading the output and the literature
correctly. Every printed result now reads as a report in APA style, every
source is cited where its method is used, and the earlier published rules can
be shown beside each decision for teaching, each with its shortcomings stated.
One criterion now matches its source, and the last deprecation is completed,
so nothing deprecated is carried into 1.0.

The package continues to declare Imports: stats only.

Results that change. Values computed by 0.7.0 and 0.8.0 were compared on
25 analyses spanning every workflow. The only differences are these:

• Lynn's criterion at nine experts. An item endorsed by 7 of 9 experts now
  meets the I-CVI criterion, as Lynn's (1986) Table 2 says; 0.7.0 required 8.
  cvi_criterion now shows each panel size's exact cutoff, such as .875 for
  eight experts, where it showed .78 throughout. Beyond ten experts, where her
  table stops, the package holds her 7 of 9. That lowers the requirement by
  one expert at every multiple of nine (14 of 18, 21 of 27, 28 of 36, and so
  on) and at some panels above 58. Verdicts at other panel sizes are
  unchanged.
• judge_validity()'s influence check. It removes one judge at a time, so
  a ten-judge panel was checked at nine, where 0.7.0 applied the wrong
  criterion. On a ten-judge panel this could flag every judge as influential.
  It now applies Lynn's.
• A label. A judge whose fit falls below the range is now Too predictable, not Erratic. The status is Review either way.
• One removed column. anova_content() no longer returns posthoc_pass;
  read contrast_pass, which holds the same value.
• Handoff rule text reads as counts and APA numbers. The schema is
  unchanged.

One display default changes. Print methods now default to digits = 2, as
APA reports statistics; print(x, digits = 3) restores the previous
precision. Stored values are unaffected.

New arguments: sort_validity(legacy = , n_constructs = ),
expert_validity(legacy = ), and legacy in their print() methods.

Published on GitHub and R-universe. 0.4.0 is still in CRAN's review queue, and
CRAN policy asks that no further version be submitted while one is pending.

The last deprecation is completed before 1.0

• Breaking change: anova_content() no longer returns posthoc_pass. It
  was a duplicate of contrast_pass, documented as deprecated in 0.7.0. The
  written policy requires one minor release of notice and allows removal in a
  minor release only before 1.0, so this is the last release that can remove
  it without waiting for 2.0. Read contrast_pass, which holds the same
  value. It was the last column, so no other column moves.
• No deprecation is in progress now, so nothing deprecated is carried into
  1.0. ?contentvalidR records both of anova_content()'s completed cycles.

Earlier methods, for comparison

For teaching, the way a methods text reports eta-squared beside omega-squared,
sort_validity() and expert_validity() now compute the earlier published
rules beside their own. legacy = TRUE prints them under "Earlier methods, for
comparison (not used for the decision)", and print(fit, legacy = TRUE) shows
them for any fit. They are stored in details$earlier_methods and never change
a decision. Each follows its source as published, and the tests check each
against the numbers its source prints.

• Item sort.
  • Anderson and Gerbing's (1991) critical Csv, from their Equations 5 and 6:
    .50 for 20 judges at .05. The printout explains why it is not used to
    decide: it assumes every judge who misses the target picks the same rival,
    so when they spread across several constructs it can pass an item the
    exact test does not.
  • Yao, Wu and Yang's (2008) cutoffs, Psa and Csv both at least .30, set for a
    four-domain sort.
  • A labeled package extension of Yao et al.'s rule to other numbers of
    constructs: chance plus .05, so Psa and Csv both at least 1/k + .05,
    which is their .30 at four. It prints in its own column marked * and is
    described as "a contentvalidR extension, not a published rule". The new
    n_constructs argument states how many constructs judges were offered;
    by default the package counts the ones they used, and says so.
  • A count of how often each earlier rule agrees with the decision, and a
    note that pooling every rival into Csv gives a different index.
• Expert essentiality. Lawshe's (1975) Table 1 minimum CVR, the
  recalculation by Wilson, Pan and Schumsky (2012), z / sqrt(N), and Lawshe's
  content validity index, the mean CVR of the items his table retains. A panel
  size Lawshe did not tabulate gets no minimum rather than an interpolated one.
  When a CVR prints equal to a cutoff it misses, such as 8 of 9 (.778) against
  Lawshe's .78, the printout says so.
• Expert relevance. Fleiss' (1971) kappa on the relevant/not-relevant
  decision, computed by the package and checked against the irr package, and
  the benchmarks of Polit and Beck (2006): .90 for S-CVI/Ave, which is the
  average congruency percentage, and .80 for S-CVI/UA.
• Expert relevance: the content validity coefficient (Ccv) of
  Hernández-Nieto (2002), beside Aiken's V, read from the full text of the
  book's chapter.
  It is shown because it is widely cited, and never informs a decision,
  because of shortcomings the printout, ?expert_validity and the
  expert-panel vignette all set out:
  • It uses only each item's mean, so it cannot reflect agreement, although
    the book says it does. The book's own Table 7 gives ratings of 1, 3, 4,
    5, 2 and of 3, 3, 3, 3, 3 the same .60.
  • Its chance correction, (1/J)^J, depends only on the number of judges. It
    is .00032 for five of them.
  • On a scale starting at 0 it is Aiken's V; on a scale starting at 1 it
    cannot fall below one over the maximum.
  • Its .80 and .90 bands are not derived, and the book's own Example 11
    calls .7968 acceptable.
• The item-sort vignette section that explained why Anderson and Gerbing's rule
  was "not exposed" now shows it for comparison, using the shipped example
  where it and the exact test disagree. The expert-panel vignette shows
  Lawshe's anomaly at nine experts.

Every vignette lists what it cites

• Four vignettes cited works without a reference list, and the expert-panel
  vignette cited Lynn (1986) without listing it. Each now has an APA reference
  list, and vignette("reading-output"), which explains the APA number rules,
  cites the Publication Manual. Yao et al. (2008) joins the README references.

Printed output reads as a report, in APA style

No statistic changes. Only printed output changes, and results still holds
every value at full precision.

• Numbers follow the APA Publication Manual, seventh edition. A statistic
  that cannot exceed 1 prints without a leading zero (Psa = .90, p = .021,
  kappa = -.13). One that can exceed 1 keeps it (a chi-square of 0.31, a
  severity of -0.27 rating points). Values have two decimals, and p values
  have three, with anything below .001 printed as < .001 rather than 0.000
  (Section 6.36).
  Intervals print as 95% CI [.70, .97] in a column headed with their level
  (Section 6.43). digits now defaults to 2 in every print method.
• Every print opens with its verdict. A sentence says how many items met
  the criterion and names the ones flagged, before any table. The decision
  appears beside each item. Anything that is the same on every row, such as the
  I-CVI criterion for a panel of one size, is stated once in a sentence instead
  of repeated down a column.
• No more walls of numbers. Tables are laid out to fit an 80-column
  console (long item names can still widen them), and prose wraps to the
  console width. The summary() methods show flagged items in the same columns
  as print(), where the expert-panel summary used to print its whole results
  table. An explanation shared by several items, such as a
  Delphi note or a judge's flag, is printed once with every item it applies
  to.
• Planning tables have one row per panel size. expert_power() and
  sort_power() put each assumed probability in its own column, so ten panel
  sizes take ten rows instead of thirty.
• The key names columns by the heading you see (Psa, 95% CI, kappa), not
  by the name of the column in results.
• Headings are words. n_target and n print together as judges (18/20),
  severity(pts) as severity, differentiation as scale use, and the
  recommendation column as decision.
• delphi_validity() prints the degrees of freedom beside a chi-square, as APA
  reports it. judge_validity() prints infit beside outfit, since either can
  flag a judge, and states the rules that flag one.
• The README and vignettes follow the same rules, and the vignette on reading
  output explains them.

Two labels now say what they mean

• judge_validity() no longer calls a too-predictable judge "Erratic". A
  fit mean square above the range means ratings noisier than the model expects;
  one below it means ratings more predictable than it expects, the opposite
  problem. Both were labeled Erratic. The second is now Too predictable,
  with its own interpretation. The status is Review either way, and which
  judges are flagged does not change. Code that matched recommendation == "Erratic" to find every misfitting judge should also match "Too predictable", or use status.
• The handoff states the I-CVI criterion as a count. Its rule text read
  I-CVI >= 0.8333333. It now reads at least 5 of 6 experts rate the item relevant (I-CVI >= .83; Lynn, 1986), since the decision compares counts, and
  the handoff now cites Lynn (1986). Rule text is prose for a person and
  outside the frozen schema, which is unchanged.

The I-CVI criterion now matches Lynn's table

• Changed result, with nine experts only. An item endorsed by 7 of 9 experts
  now meets the I-CVI criterion; before, it needed 8. The criterion was stored
  as ".78", which is Lynn's (1986) 7 of 9 rounded, and 7/9 = .778 fell short of
  the rounded value. Lynn's Table 2 requires 7 of 9, and Polit and Beck (2006)
  say the same in words. Every other panel size from three to ten was already
  right. This affects expert_validity() verdicts, expert_power() planning
  probabilities, and judge_validity()'s check of whether one judge moves an
  item.
• cvi_criterion now shows Lynn's cutoff for each panel size, such as .833
  for six experts and .778 for nine, instead of .78 throughout. Verdicts at six,
  seven, eight and ten experts are unchanged, but the printed criterion now
  matches the verdict beside it: an item is supported exactly when its I-CVI
  reaches the number shown. A test checks that for every panel of 3 to 15
  experts and every count of agreeing experts.
• Beyond ten experts, the criterion is now labeled an extension. Lynn's
  table stops at ten. The package holds her lowest tabled proportion, 7 of 9,
  and says so, where before it applied .78 without comment. That also lowers
  the requirement by one expert at every multiple of nine, where 7/9 falls
  exactly on a whole count (14 of 18, 21 of 27, 28 of 36, and so on), and at
  some panels above 58, where .78 and 7/9 straddle a whole count.
• The rule lives in one place, stored as Lynn's counts. The three functions
  compare counts of agreeing experts, so no rounding can move an item.

Attribution

• Every source the package cites is now on its public reference list. The
  reference list in the README was missing eight sources that function help
  pages cite, including Krippendorff (2011), whose alpha is the default panel
  coefficient, and Schriesheim et al. (1993, 1999), behind qfactor_content().
• Four foundational sources are now cited where their methods are used, each
  checked against the full text:
  • cvi() and expert_validity() cite Lynn (1986), the source of the I-CVI
    and of the panel-size criterion expert_validity() applies, and Polit and
    Beck (2006), who named S-CVI/UA and S-CVI/Ave. ?cvi now explains where
    each index comes from and why both scale-level versions should be reported.
  • cvr() cites Wilson, Pan and Schumsky (2012) and explains why the exact
    values replace Lawshe's original table: Wilson et al. found the table
    unexplained, non-monotonic at eight experts, and closer to a two-tailed
    than the one-tailed test it was labeled as.
  • agreement_summary() credited nobody. It now cites Fleiss (1971) for the
    statistic and the irr package's authors for the code it calls.
• sort_power() and simulate_csv_power() cite Howard and Melloy (2016), whose
  test they plan for. Two incomplete entries are completed.
• Three instructions that did not work are fixed. print() on a handoff,
  vignette("handoff-to-empirical-validation"), and the README all told
  readers to pass the handoff object to nomologR::nomo_screen(), which cannot
  read it until nomologR#46.
  All three now pass handoff$items.