Feedback welcome: checking partner dependence in dyadic models

[Pascal-Kueng]

PR #18 introduces an experimental diagnostic for dyadic mixed models in dyadMLM:

#18

The diagnostic asks whether a fitted model reproduces important features of the relationship between partners. For example:

• Does the model reproduce how strongly partners' responses are related?
• Does it reproduce how much responses vary within each role?
• For dyads without meaningful roles, does it reproduce variation between dyads and variation between the two partners within a dyad?

It does this by keeping the fitted model and the observed study setup fixed, generating many new possible datasets from that model, and comparing them with the observed data. This is a predictive check: it asks whether data generated by the model resemble the data we actually observed.

Scope of this check

This is deliberately a descriptive plug-in check. Here, “plug-in” means that every simulation uses the single set of estimates from the fitted model as if those estimates were known. The model is not fitted again for each simulated dataset, and uncertainty about the estimates is not included.

The plots show where an observed summary falls among the values produced by the model. The observed positions and middle 95% ranges are descriptive guides only. They are not p-values, calibrated hypothesis tests, or pass/fail rules. A close match does not prove that the model is correct, and a mismatch does not by itself identify which part of the model caused it.

The underlying calculations have been checked against the published Woody-Sadler formulas and an independent implementation. We also ran a simulation study with 7,000 fitted models. The diagnostic identified large omitted partner relationships and incorrect equal-variation assumptions in the expected direction.

The current implementation is deliberately limited to cross-sectional Gaussian glmmTMB models—that is, models for approximately continuous, normally distributed outcomes measured once per dyad or observation period.

Questions where feedback would be especially helpful

1. Is this the right kind of simulation?

The diagnostic uses the estimates from the fitted model and generates new random differences between dyads, partners, and observations. It does not refit the model for every simulated dataset.

Is this a useful comparison for detecting questionable assumptions about partner relationships and variation? Are there situations where a different kind of simulation would be preferable?

2. How should reassuring results be interpreted?

A flexible model may reproduce partner dependence simply because it was allowed to estimate that dependence from the same data. A reassuring result therefore does not necessarily provide independent evidence that the model is correct.

Is this limitation explained clearly enough? How would you recommend communicating what users can—and cannot—conclude?

3. Are the selected summaries useful?

For distinguishable dyads, such as partners with different roles, the diagnostic compares the variability within each role and the relationship between partners.

For exchangeable dyads, where partner labels are arbitrary, it instead describes variation between dyads and variation between partners within the same dyad.

Are these the most useful quantities to show? Are there important patterns that these summaries could overlook?

4. Is the distinction between role-aware and exchangeable analyses clear?

The two presentations describe related information in different ways. We would appreciate feedback on when each view is most helpful and whether showing both might confuse users.

5. What should be studied next?

The current simulation study covers clear, relatively large model problems. It does not yet establish how sensitive the diagnostic is to smaller problems, unusual data, an incorrectly specified mean model, or more complicated random-effects structures.

Which additional situations would be most important before treating this as a more established diagnostic?

Code review is also welcome

Comments on the implementation are welcome directly on PR #18, especially from people familiar with glmmTMB, dyadic models, or simulation-based model diagnostics.

Feedback on any single question would already be very helpful—reviewing the entire method or pull request is not expected.