effectcheck

IMPORTANT: Development Version. This package is under heavy development and has not yet been fully validated. Results should be independently verified before use in any consequential context (e.g., peer review, retractions, editorial decisions). Use of this package is at the sole responsibility of the user. We welcome contributions, verification reports, and bug reports at https://github.com/giladfeldman/escicheck/issues or by contacting Gilad Feldman (giladfel@gmail.com).

EffectCheck is a conservative, assumption-aware statistical consistency checker for published research results. It parses statistical results across multiple citation styles (APA, Harvard, Frontiers, PLOS ONE, Scientific Reports, Nature Human Behaviour, PeerJ, eLife, PNAS, and more), uses type-matched comparison to verify reported effect sizes against computed variants of the same type, explicitly flags all assumptions and uncertainty, and validates internal consistency between test statistics, effect sizes, and confidence intervals.

Reference

This package implements effect size computations following the Guide to Effect Sizes and Confidence Intervals by Jané et al. (2024). The effectsize R package is used as the primary computation engine when available.

Citation for the Guide:

Jané, M.B., Xiao, Q., Yeung, S., et al. (2024). Guide to Effect Sizes and Confidence Intervals. http://dx.doi.org/10.17605/OSF.IO/D8C4G

Features

Statistical Coverage

Mean Differences: Cohen's d, Hedges' g (independent); dz, dav, drm (paired)
ANOVA / F-tests: Eta², partial eta², generalized eta², omega², Cohen's f
Correlations: Pearson r, Spearman ρ, Kendall τ
Contingency Tables: Phi (φ), Cramer's V
Regression / GLM: Standardized β, partial r, semi-partial r, f², R²
Ratio Measures: OR, RR, IRR
Nonparametric: Rank-biserial r, Cliff's delta, Kendall's W, ε²

Key Capabilities

Type-matched comparison: Compares reported effect sizes against computed variants of the SAME type (e.g., reported d compared to d variants, not g)
Ambiguity handling: When design is unclear, computes ALL variants and finds closest match among same-type variants
Alternative suggestions: Shows other effect size types as suggestions (e.g., Hedges' g as alternative to Cohen's d)
Ultra-robust parsing with Unicode normalization, locale-aware decimals, and PDF artifact handling
Comprehensive CI computation with method tracking and fallback strategies
Uncertainty framework that tracks all assumptions and design inferences
Context window extraction for design inference (±2 sentences)
Decision error detection (significance reversals) similar to statcheck
statcheck-compatible API for batch processing of files and directories
Multiple export formats: HTML (with expandable sections), CSV, JSON
S3 class with methods: print, summary, plot, ec_identify, get_variants

Installation

From Source

# Install dependencies
install.packages(c("devtools", "stringr", "stringi", "dplyr", "purrr", 
                   "tibble", "tidyr", "xml2", "rvest",
                   "DT", "shiny", "shinythemes", "testthat", "jsonlite"))

# Install effectcheck
devtools::install_local("effectcheck")

Optional Dependencies

For enhanced CI computation:

MBESS - Noncentral t-distribution CIs for Cohen's d
effectsize - Additional effect size computations

Quick Start

Basic Usage

library(effectcheck)

# Check text directly
text <- "t(28) = 2.21, p = .035, d = 0.80, 95% CI [0.12, 1.48]"
result <- check_text(text)
print(result)
summary(result)

# Check a single file
result <- check_file("paper.pdf")

# Check multiple files
results <- check_files(c("paper1.docx", "paper2.html"))

# Check a directory (recursively)
results <- check_dir("manuscripts/")

# Export results
render_report(result, "report.html")
export_csv(result, "results.csv")
export_json(result, "results.json")

statcheck-Compatible API

# Check PDF files (like statcheck's checkPDF)
results <- checkPDF(c("paper1.pdf", "paper2.pdf"))

# Check HTML files (like statcheck's checkHTML)
results <- checkHTML(c("paper1.html", "paper2.html"))

# Check directory of PDFs (like statcheck's checkPDFdir)
results <- checkPDFdir("manuscripts/")

# Check directory of HTML files (like statcheck's checkHTMLdir)
results <- checkHTMLdir("manuscripts/")

Working with Results

# Get summary statistics
summary(result)

# Plot results
plot(result, type = "status")      # Status distribution
plot(result, type = "uncertainty") # Uncertainty levels
plot(result, type = "all")         # All plots

# Filter problematic results
errors <- ec_identify(result, "errors")
warnings <- ec_identify(result, "warnings")
decision_errors <- get_decision_errors(result)
high_uncertainty <- filter_by_uncertainty(result, "high")

# Filter by test type
t_tests <- filter_by_test_type(result, "t")

# Count by category
count_by(result, "status")
count_by(result, "test_type")

Working with Variants (New in v2.0)

# Get all variants for a specific row
variants <- get_variants(result, row_index = 1)

# Get same-type variants only
same_type <- get_same_type_variants(result, row_index = 1)

# Get alternative suggestions
alternatives <- get_alternatives(result, row_index = 1)

# Format variants for display
cat(format_variants(result, row_index = 1))

# Compare reported value to all variants
comparison <- compare_to_variants(result, row_index = 1)
print(comparison)

# Get metadata for a specific variant type
metadata <- get_variant_metadata("dz")
print(metadata$when_to_use)

Shiny App

# Run the interactive Shiny app
shiny::runApp("effectcheck-demo")

The Shiny app provides:

Upload/Input tab: File upload, text paste, encoding selection
Parsing Diagnostics: Highlighted statistics, context windows
Results tab: Interactive table with status summary and export options
Uncertainty Review: Filtered view of WARN/ERROR cases
Settings tab: Configurable tolerances, CI levels, r-grid
Documentation tab: Complete user guide and effect size definitions

Understanding Results

Status Levels

PASS: Computed effect size matches reported value within tolerance
WARN: Minor mismatch (rounding, CI method difference, minor assumptions)
ERROR: Significant mismatch (incompatible values)
INSUFFICIENT_DATA: Not enough information to compute effect sizes

Uncertainty Levels

Low: All required information present, no assumptions
Medium: Minor assumptions made (e.g., equal group sizes)
High: Major assumptions or missing critical information

Output Columns

Key columns in results:

Reported Values:

reported_type: The effect size type user reported (d, g, r, eta2, etc.)
effect_reported: Reported effect size value
effect_reported_name: Original name as parsed from text

Type-Matched Comparison (New in v2.0):

matched_variant: Which same-type variant matched best
matched_value: The computed value of that variant
delta_effect: Absolute difference between reported and matched
ambiguity_level: "clear", "ambiguous", or "highly_ambiguous"
ambiguity_reason: Why comparison was ambiguous (if applicable)
all_variants: JSON structure with all computed variants and alternatives

Legacy Columns (backward compatible):

test_type: Type of test (t, F, r, chisq, z)
closest_method: Alias for matched_variant
delta_effect_abs: Alias for delta_effect

P-value and Decision Errors:

p_reported: Reported p-value
p_computed: Computed p-value from test statistic
decision_error: TRUE if significance reversal detected

Status and Uncertainty:

status: PASS/WARN/ERROR/INSUFFICIENT_DATA
uncertainty_level: Low/medium/high
uncertainty_reasons: Specific sources of uncertainty
assumptions_used: All assumptions made during computation
design_inferred: Inferred study design (paired, independent, etc.)
assumptions_used: All assumptions made during computation
design_inferred: Best guess at experimental design
variants_tested: All effect size variants computed
source: Source file name (or "text" for direct input)

Examples

Example 1: Independent t-test

text <- "t(28) = 2.21, p = .035, d = 0.80, 95% CI [0.12, 1.48], N = 30, n1 = 15, n2 = 15"
result <- check_text(text)

Example 2: ANOVA

text <- "F(2, 27) = 4.56, p < .05, η² = 0.25"
result <- check_text(text)
# Computes: eta², partial eta², omega², Cohen's f

Example 3: Correlation

text <- "r(198) = .34, p < .001, 95% CI [.21, .45]"
result <- check_text(text)

Example 4: Multiple Statistics

text <- "t(28) = 2.21, d = 0.80. Another test: F(2, 27) = 4.56, η² = 0.25."
result <- check_text(text)
# Returns one row per statistic detected

Advanced Usage

Custom Tolerances

result <- check_text(
  text,
  tol_effect = list(d = 0.02, r = 0.005, phi = 0.02, V = 0.02, eta2 = 0.01),
  tol_ci = 0.02
)

Paired t-test r-grid

result <- check_text(
  text,
  paired_r_grid = seq(0.1, 0.9, by = 0.1)  # Default
)
# Shows dav range across correlation assumptions

CI Level Detection

EffectCheck automatically detects CI level from text:

95% CI [0.12, 0.45] → uses 95%
CI [0.12, 0.45] → defaults to 95% (flagged as assumption)

File Format Support

.pdf: PDF files (via pdftotext from poppler-utils, with optional table extraction and OCR)
.docx: Microsoft Word documents (via pandoc; requires pandoc on PATH)
.html: HTML files (via rvest package)
.txt: Plain text files

PDF Options

# Enable table extraction from PDFs
results <- checkPDF("paper.pdf", try_tables = TRUE)

# Enable OCR for scanned PDFs
results <- checkPDF("scanned_paper.pdf", try_ocr = TRUE)

Philosophy

EffectCheck follows these principles:

Conservative by Design: When ambiguous, compute all variants rather than guessing
Transparency First: All assumptions visible in output
Never Silent: Always flag uncertainty and assumptions
Comprehensive: Compute all plausible effect size variants

Limitations

Some effect sizes require additional information (e.g., generalized eta² for within-subjects)
CI methods may differ from those used in original papers (flagged as WARN)
Design inference from context is heuristic-based
OCR for scanned PDFs requires tesseract and magick packages

API Reference

Core Functions

Function	Description
`check_text(text, ...)`	Check plain text for statistical consistency
`check_file(path, ...)`	Check a single file
`check_files(paths, ...)`	Check multiple files
`check_dir(dir, ...)`	Check all files in a directory

statcheck-Compatible Functions

Function	Description
`checkPDF(files, ...)`	Check PDF files
`checkHTML(files, ...)`	Check HTML files
`checkPDFdir(dir, ...)`	Check directory of PDFs
`checkHTMLdir(dir, ...)`	Check directory of HTML files
`checkDOCXdir(dir, ...)`	Check directory of Word documents

S3 Methods

Method	Description
`print(x)`	Print formatted summary
`summary(x)`	Comprehensive summary statistics
`plot(x, type)`	Visualize results
`ec_identify(x, what)`	Filter problematic results

Filter Functions

Function	Description
`get_errors(x)`	Get ERROR status results
`get_warnings(x)`	Get WARN status results
`get_decision_errors(x)`	Get significance reversals
`filter_by_test_type(x, types)`	Filter by test type
`filter_by_uncertainty(x, levels)`	Filter by uncertainty level
`filter_by_source(x, files)`	Filter by source file
`filter_by_delta(x, min, max)`	Filter by effect size delta
`count_by(x, by)`	Count results by category

Export Functions

Function	Description
`render_report(x, out)`	Generate HTML report
`export_csv(x, out)`	Export to CSV
`export_json(x, out)`	Export to JSON

Citation

If you use EffectCheck in your research, please cite:

EffectCheck: Statistical Consistency Checker for Published Research Results
Version 0.2.0

License

MIT License - see LICENSE file for details

Contributing

Contributions welcome! Please open an issue or pull request at https://github.com/giladfeldman/escicheck.

Related Tools

statcheck - Checks p-values in APA-style text
effectsize - R package for effect size computation

Support

For issues, questions, or feature requests, please open an issue on the project repository.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
R		R
build		build
inst		inst
man		man
tests		tests
vignettes		vignettes
DESCRIPTION		DESCRIPTION
LICENSE		LICENSE
MD5		MD5
NAMESPACE		NAMESPACE
NEWS.md		NEWS.md
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

effectcheck

Reference

Features

Statistical Coverage

Key Capabilities

Installation

From Source

Optional Dependencies

Quick Start

Basic Usage

statcheck-Compatible API

Working with Results

Working with Variants (New in v2.0)

Shiny App

Understanding Results

Status Levels

Uncertainty Levels

Output Columns

Examples

Example 1: Independent t-test

Example 2: ANOVA

Example 3: Correlation

Example 4: Multiple Statistics

Advanced Usage

Custom Tolerances

Paired t-test r-grid

CI Level Detection

File Format Support

PDF Options

Philosophy

Limitations

API Reference

Core Functions

statcheck-Compatible Functions

S3 Methods

Filter Functions

Export Functions

Citation

License

Contributing

Related Tools

Support

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages