Skip to content

[Phase 1.5] Validation suite: marginal vs joint_gcomp on ARTnet 2017-18 #65

@smjenness

Description

@smjenness

Context

Before releasing the joint g-computation method in production, we need a rigorous comparison to the current marginal approach to:

  1. Quantify the bias in current marginal-method netstats.
  2. Validate that joint g-comp reproduces marginal-method behavior when the target distribution matches ARTnet (internal consistency check).
  3. Identify which ERGM terms are most affected by the correction.
  4. Generate the comparison figures/tables that will anchor the methods paper.

Scope

Run both methods on ARTnet 2017-18 data with:

  • Scenario 1 — ARTnet-self target (sanity): target population = ARTnet 2017-18 respondents themselves. Joint and marginal should give similar results (any difference is pure marginal-bias artifact).
  • Scenario 2 — Current ARTnet default target (NCHS age + national race): what build_netstats() currently does. Joint method may differ if interactions matter.
  • Scenario 3 — Shifted target (e.g., older population or different race mix): explicitly stress-test the correction. This is where joint-vs-marginal should diverge most.

For each scenario, report:

  • All netstats values side-by-side (marginal vs joint).
  • % difference per target stat.
  • Which interactions drive the differences (via ablation).

Tasks

  • Write validation script in tests/testthat/ or inst/validation/ directory.
  • Produce a side-by-side comparison table for main / casl / inst × all target stats × marginal / joint.
  • Identify stats where joint-marginal difference is >5% — flag as "materially affected by method choice."
  • Write up validation results (will feed methods paper + vignette).
  • Sanity check: on Scenario 1 (ARTnet-self target), all differences should be negligible (<2%).

Acceptance criteria

  • Reproducible validation script.
  • Comparison table for all ERGM target stats.
  • Clear interpretation of which stats shift how much under which scenarios.

Related

Stretch

Extend validation to a second dataset if available (e.g., a city-specific subset). Internal consistency check: the bias magnitude should correlate with how much the target distribution differs from ARTnet.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions