Example 20: Power of Rejecting Misspecified Models with Varying Sample Size (Version 0.2)
This example will show the power in rejecting misspecified model when the sample size in simulation ranges from 25 to 500. The null (hypothesized) model is a confirmatory factor analysis model where Variables 1-5 are loaded on Factor 1 and Variables 6-8 are loaded on Factor 2. However, the alternative model is a confirmatory factor analysis model where Variables 1-4 are loaded on Factor 1 and Variables 5-8 are loaded on Factor 2. The datasets created by the alternative model should be rejected when fitting them to the null model. We will check for the power of rejecting the wrong model given each value of sample size.
For both models, the factor loadings are 0.7, the factor correlations are 0.5, and the residual variance are the values making the total indicator variances equal to 1. The trivial misspecification added to the datasets created by both models are the random error correlation of normal distribution with the mean of 0 and standard deviation of 0.1. We will see the proportion of datasets created from the alternative model rejected when fitting the null model given each sample size.
All relevant distribution objects are specified:
n1 <- simNorm(0, 0.1)
The null (hypothesized) population model is specified:
loading.null <- matrix(0, 8, 2)
loading.null[1:5, 1] <- NA
loading.null[6:8, 2] <- NA
LX.NULL <- simMatrix(loading.null, 0.7)
latent.cor.null <- matrix(NA, 2, 2)
diag(latent.cor.null) <- 1
RPH <- symMatrix(latent.cor.null, 0.5)
RTD <- symMatrix(diag(8))
CFA.Model.NULL <- simSetCFA(LY = LX.NULL, RPS = RPH, RTE = RTD)
The alternative population model is specified:
loading.alt <- matrix(0, 8, 2)
loading.alt[1:4, 1] <- NA
loading.alt[5:8, 2] <- NA
LX.ALT <- simMatrix(loading.alt, 0.7)
CFA.Model.ALT <- simSetCFA(LY = LX.ALT, RPS = RPH, RTE = RTD)
The misspecification for both models is specified:
error.cor.mis <- matrix(NA, 8, 8)
diag(error.cor.mis) <- 1
RTD.Mis <- symMatrix(error.cor.mis, "n1")
CFA.Model.Mis <- simMisspecCFA(RTE = RTD.Mis)
The data objects with trivial misspecification are specified:
SimData.NULL <- simData(CFA.Model.NULL, 500, misspec=CFA.Model.Mis)
SimData.ALT <- simData(CFA.Model.ALT, 500, misspec=CFA.Model.Mis)
The model object used to fit the datasets created from both data objects is specified:
SimModel <- simModel(CFA.Model.NULL)
The result object with varying sample size is specified:
Output.NULL <- simResult(NULL, SimData.NULL, SimModel, n=25:500)
Output.ALT <- simResult(NULL, SimData.ALT, SimModel, n=25:500)
From here, we can find cutoffs or plot cutoffs of the correct population model:
cutoff <- getCutoff(Output.NULL, alpha=0.05, nVal=250)
plotCutoff(Output.NULL, alpha=0.05)
The figure below shows the graph provided by the plotCutoff function:
We can find the proportion of samples from the alternative model that was rejected in a given value of sample size by the getPowerFit function as
getPowerFit(Output.ALT, nullObject=Output.NULL, alpha=0.05, nVal=250)
The first argument is the result object based on the alternative model or the model we wish to reject. The nullObject argument is the result object based on the null model or the correct model. The alpha argument is the alpha level. The nVal argument is the value of sample size that we wish to find the power. This basic idea behind this function is to find the fit indices cutoffs for all values of sample size, create a variable to indicate whether each replication is significant (based on a fit index cutoff for each sample size), and use logistic regression to predict the proportion of significance (power) given a specific sample size.
The figure below shows the graph provided by the getPowerFit function:
If the fit indices cutoffs are known, we can find the power by
getPowerFit(Output.ALT, cutoff=cutoff, nVal=250, condCutoff=TRUE)
The cutoff argument is the vector of fit indices cutoffs. The condCutoff means whether the given cutoffs are sample size specific, which is TRUE by default. The given cutoffs are calculated above based on a specific sample size so condCutoff=TRUE. When condCutoff=TRUE, the power is estimated by the quantile regression that finds the quantile rank in a given sample size that provides a given cutoff.
The figure below shows the graph provided by the getPowerFit function when the cutoffs are specified:
The power can be plotted against the sample size, which is the only varying parameter:
plotPowerFit(Output.ALT, Output.NULL, alpha=0.05)
The resulting plot is a logistic curve predicting the significance result by the varying parameter, sample size, given the alpha level is .05.
The figure below shows the graph provided by the plotPower function:
Users may plot overlapping scatterplots to investigate the actual distributions of fit indices in both models given sample size values by specifying logistic argument as FALSE in the plotPowerFit function.
plotPowerFit(Output.ALT, Output.NULL, alpha=0.05, logistic=FALSE)
The figure below shows the graph provided by the plotPower function when logistic=FALSE:
We may set a priori cutoffs, such as RMSEA < .05, CFI > .95, TLI > .95, and SRMR < .06, and use these cutoffs to find the power:
cutoff2 <- c(RMSEA = 0.05, CFI = 0.95, TLI = 0.95, SRMR = 0.06)
getPowerFit(Output.ALT, cutoff=cutoff2, nVal=250, condCutoff=FALSE)
The figure below shows the graph provided by the getPowerFit function when the a priori cutoffs are specified:
The power using a priori cutoffs can be plotted:
plotPowerFit(Output.ALT, cutoff=cutoff2)
plotPowerFit(Output.ALT, cutoff=cutoff2, logistic=FALSE)
The figure below shows the logistic plot provided by the plotPower function with cutoff2:
The figure below shows the overlapping scatterplots provided by the plotPower function with cutoff2:
Here is the summary of the whole script in this example.