falsifyr is Chaos Monkey for statistical claims. Give it one fitted result
and one term; it searches for the smallest plausible perturbation that changes
the conclusion.
What would it take to kill this result?
library(falsifyr)
fit <- lm(score ~ treatment + age + baseline_score, data = fragile_trial)
result <- attack(fit, term = "treatment")
result
plot(result)
smallest_kill(result)The headline is the smallest kill, not a generic model-diagnostics dashboard. A killed claim is fragile under the named attack; it is not proof that the claim is false or that the perturbed analysis is preferable.
The development release is available from GitHub:
pak::pak("msaule/falsifyr")The 1.0 engine searches across:
- influence-ranked, greedy, beam, and grouped row deletion;
- bootstrap uncertainty and optional HC0-HC3 robust standard errors;
- drop-one covariate dependence and bounded specification changes;
- mean, median, factor/mode, and missingness-indicator alternatives;
- outcome/predictor noise and binary label flips;
- label permutation, fake predictors, and supplied placebo outcomes;
- random and stratified split stability.
intensity = "fast", "normal", "deep", or "insane" controls search
breadth. parallel = TRUE runs independent attack families on at most two
workers. All stochastic attacks are deterministic for a fixed seed.
Grouped deletion is enabled when observations belong to meaningful units:
attack(
fit,
term = "treatment",
data = trial_data,
cluster = "site",
attacks = "row_deletion"
)| Model object | Claim extraction | Perturbation refits |
|---|---|---|
lm, glm, aov |
yes | yes |
lme4::lmer, lme4::glmer |
yes | yes, when lme4 is installed |
survival::coxph |
yes | yes, when survival is installed |
t.test, wilcox.test, cor.test (htest) |
yes | limited: source data are not retained |
anova tables |
yes | limited: fitted model is not retained |
For lmerMod objects, coefficient p-values use an explicitly labeled Wald
normal approximation when the fitted model does not provide p-values.
When broom is installed, broom::tidy(result) returns the attack leaderboard
and broom::glance(result) returns a one-row claim and verdict summary.
attack_leaderboard(result)
report(result, file = tempfile(fileext = ".html"))The standalone HTML report contains the claim card, score, smallest kill, ranked attacks, family details, fragility curves, limitations, and a reproducibility appendix. In RStudio, Addins -> Attack This Claim provides model, term, intensity, alpha, and attack-family controls and opens the report in the Viewer.
The CRAN package includes fast deterministic unit tests and vignettes. A larger
GitHub-only corpus in validation/ checks known fragile/resilient claims,
grouped deletion, factor terms, GLMs, mixed models, Cox models, score ordering,
and interpretation language.
The survival score is a heuristic communication device, not a probability that the claim is true. Read every verdict together with the attacks that were run and the report's limitations.
Existing tools help define, diagnose, or run robustness analyses. falsifyr
automatically attacks a fitted claim and searches for the smallest plausible
perturbation that changes the conclusion.