02-smart-choices.qmd

# Smart Choices in CFA

This paper presents a comprehensive approach to conducting a standard CFA within the applied social sciences, following the guidelines outlined by @rogers2024. According to @rogers2024, a typical CFA study seeks to fit a reflective common factor model with a predefined multifactor structure, established psychometric properties, and a maximum of five Likert-type response options. This scenario frequently occurs in research endeavors where the measurement model facilitates the examination of hypotheses derived from the structural model.

The initial phase in such research involves data preprocessing. Specifically, for categorical data, @rogers2024 advises employing multiple imputation to handle missing data, taking into consideration the limitations posed by available software and methodologies [@rogers2024]. When a measurement model allows for the treatment of items as continuous variables, addressing this challenge can be deferred to the estimation process stage through the selection of an appropriate estimator [@robitzsch2022].

This paper reinterprets the insights from @rogers2024 for CFAs that accommodate continuous item treatment. Thus, a strategic choice involves opting for measurement models that permit this approach, thereby circumventing methodological hurdles [@robitzsch2022; @robitzsch2020] associated with binary and/or ordinal response items with up to four or five gradations. Such a decision influences various aspects of the research process, including the choice of software, power analysis, estimation techniques, criteria for model adjustment, and model comparisons. These choices, in turn, affect requirements concerning sample size, computational resources, and the researcher's expertise [@robitzsch2020].

Subsequent sections delve into themes previously summarized by @rogers2024, specifically concerning CFAs with ordinal items. These themes are explored in terms of recommended practices [@flake2017; @nye2022; @rogers2024], pitfalls to avoid [@crede2019; @rogers2024], and reporting guidelines [@flake2020; @jackson2009; @rogers2024], all within the context of selecting measurement models that accommodate continuous data interpretation.

Assuming that readers possess a foundational understanding of the topic, this paper omits certain technical details, directing readers to authoritative texts [@brown2015; @kline2023] and scholarly articles that provide an introduction to Covariance-Based Structural Equation Modeling (CB-SEM) [@davvetas2020; @shek2014]. The discussion is framed within the CB-SEM paradigm [@brown2015; @jackson2009; @kline2023; @nye2022], with a focus on CFA. The paper explicitly excludes discussions on measurement model modifications in Variance-Based SEM (VB-SEM), which are predominantly addressed in the literature on Partial Least Squares SEM (PLS-SEM) [@hair2022; @hair2017; @henseler2021].

## Measurement Model Selection

Selecting an appropriate measurement model is a critical initial step in the research process. For robust analysis, it is advisable to prioritize models that provide five or more ordinal response options. Research has shown that a higher number of response gradations enhances the ability to detect inaccurately defined models [@green1997; @maydeu-olivares2017a], even when using estimators designed for ordinal items [@xia2018]. This strategy also mitigates some of the methodological challenges associated with the analysis of ordinal data in CFA [@rhemtulla2012; @robitzsch2022; @robitzsch2020].

When choosing a measurement scale, it is crucial to select ones that have been validated in the language of application and with the study's target audience [@flake2017]. Avoid scales that are proprietary or specific to certain professions. An examination of your country's Psychological Test Assessment System can be an effective starting point. If the desired scale is not found within these resources, consider looking into scales developed with the support of public institutions, non-governmental organizations, research centers, or universities, as these entities often invest significant resources in validating measurement models for broader public policy purposes.

An extensive literature review is essential for selecting a suitable measurement model. This should include consulting specialized journals, books, technical reports, and academic dissertations or theses. @schumacker2021 provide a detailed guide for initiating this search. Consideration should also be given to systematic reviews or meta-analyses focusing on measurement models related to your topic of interest. It is important to review both the original articles on the scales and subsequent applications. @kline2016 offers a useful checklist for assessing various measurement methods.

Incorporate control questions, such as requiring respondents to select "strongly agree" on specific items, and monitor survey response times to gauge participant engagement [@collier2020].

Avoid adopting measurement models designed for narrow purposes or those lacking rigorous psychometric validation [@flake2020; @kline2016]. The mere existence of a scale does not ensure its validity [@flake2017]. Also, steer clear of seldom-used or outdated scales, as they may have compromised psychometric properties. Translating a scale from another language for immediate use without thorough translation and retranslation processes is inadvisable. Be cautious of overlooking alternative factorial structures (e.g., higher-order or bifactor models) that could potentially salvage the research if considered thoroughly [@crede2019].

When selecting a scale, justify its choice by highlighting its strong psychometric properties, including previous empirical evidence of its application within the target population and its reliability and validity metrics [@flake2020; @jackson2009; @kline2016]. If the scale has multiple potential factorial structures, provide a rationale for the chosen model to prevent the misuse of CFA for exploratory purposes [@jackson2009].

Clearly specify the selected model and rationalize your choice by detailing its advantages over other theoretical models. Illustrating the models under consideration can further clarify your research approach [@jackson2009]. Finally, identify and explain any potential cross-loadings based on prior empirical evidence [@brown2023; @nye2022], ensuring a comprehensive and well-justified methodological foundation for your study.

## Power Analysis

When addressing Power Analysis (PA) in CFA and SEM, it's essential to move beyond general rules of thumb for determining sample sizes. Commonly cited guidelines suggesting minimum sizes or specific ratios of observations to parameters (e.g., 50, 100, 200, 300, 400, 500, 1000 for sample sizes or 20/1, 10/1, 5/1 for observation/parameter ratios) [@kline2023; @kyriazos2018] are based on controlled conditions that may not directly transfer to your study's context.

Reliance on lower-bound sample sizes as a substitute for thorough PA risks inadequate power for detecting meaningful effects in your model [@westland2010; @wang2023]. Tools like Soper's calculator (<https://www.danielsoper.com/statcalc/>), while popular and frequently cited (as of 02/20/2024, with almost four years of existence, it had collected more than 1,000 citations on Google Scholar), should not replace a tailored PA approach. Such calculators, despite their utility, may not fully accommodate the complexities and specific requirements of your research design [@kyriazos2018; @feng2023; @moshagen2023].

A modern perspective on sample size determination emphasizes customizing power calculations to fit the unique aspects of each study, incorporating specific research settings and questions [@feng2023; @moshagen2023]. This approach underscores that there is no universal sample size or minimum that applies across all research scenarios [@kline2023].

Planning for PA should ideally precede data collection, enhancing the researcher's understanding of the study and facilitating informed decisions regarding the measurement model based on existing literature and known population characteristics [@feng2023; @leite2023]. A priori PA not only ensures adequate sample size for detecting the intended effects, minimizing Type II errors, but also aids in budgeting for data collection and enhancing overall research design [@feng2023].

PA in SEM can be approached analytically, using asymptotic theory, or through simulation methods. Analytical methods require specifying the effect size in relation to the non-centrality parameter, while simulated PA leverages a population model to empirically estimate power [@moshagen2023; @feng2023]. These approaches are applicable to assessing both global model fit and specific model parameters.

For CFA, evaluating the power related to the global fit of the measurement model is recommended [@nye2022]. Although analytical solutions have their limitations, they can serve as preliminary steps, complemented by simulation techniques for a more comprehensive PA [@feng2023; @moshagen2023].

Several resources offer analytical solutions for global fit PA, including ShinyApps by @jak2021, @moshagen2023, @wang2021, and @zhang2018, with the last application providing a comprehensive suite for Monte Carlo Simulation (SMC) that accommodates missing data, non-normal distributions, and facilitates model testing without extensive coding [@wang2021]. For an overview of these solutions and a discussion of analytical approaches, see @feng2023, @jak2021, @nye2022, and @wang2023.

However, it is a smart decision to run an SMC for the PA of your CFA model using solutions that are consistent with the results' reproducible and replicability. In this way, even analytical solutions that the researcher may use as a starting point are recommended in the R environment via the semTools packages [@jak2021] and semPower 2 [@jobst2023; @moshagen2023]. The first option is compatible with the lavaan syntax and looks to be enough. The second, albeit including SMC in some cases, has a more difficult syntax.

For detailed and tailored PA, especially in complex models or unique study designs, the simsem package offers a robust solution, allowing for the relaxation of traditional assumptions and supporting the use of robust estimators. This package, which utilizes the familiar lavaan syntax, simplifies the learning curve for researchers already accustomed to SEM analyses, providing a user-friendly interface for conducting SMC [@pornprasertmanit2022].

Publishing the sampling design and methodology enhances the reproducibility and replicability of research, contributing to the scientific community's collective understanding and validation of measurement models [@flake2017; @flake2022; @flake2020; @leite2023]. In the context of CFA, acknowledging the power limitations of your study can signal potential concerns for the broader inferences drawn from your research, emphasizing the importance of external validity and the relevance of the outcomes over mere precision [@leite2023].

## Pre-processing

Upon gathering and tabulating original data, ideally in non-binary formats such as CSV, TXT, or JSON, the first step in data preprocessing should be to eliminate responses from participants who have abandoned the study. This identification often occurs at the end of preprocessing, where these incomplete responses can offer insights into handling missing data, outliers, and multicollinearity.

Incorporating control questions and measuring response time allows researchers to further refine their dataset by excluding participants who fail control items or complete the survey unusually quickly [@collier2020]. Calculating individual response variability (standard deviation) can identify respondents who may not have engaged meaningfully with the survey, indicated by minimal variation in their responses.

These preliminary data cleaning steps are fundamental yet frequently overlooked in empirical research. They can significantly enhance data quality before engaging in more complex statistical analyses. Visual and descriptive examination of measurement model items is implicitly beneficial for any statistical investigation and should be considered standard practice.

While data transformation methods like linearization or normalization are available, they are generally not necessary given the robust estimation processes that can handle non-normal data [@brown2015]. Parceling items is also discouraged due to its potential to obscure underlying multidimensional structures [@brown2015; @crede2019].

Addressing missing data, outliers, and multicollinearity is critical. Single imputation methods should be avoided as they underestimate error variance and can lead to identification problems in your model [@enders2023]. For missing data under 5%, the impact may be minimal, but for higher rates, Full Information ML (FIML) or Multiple Imputation (MI) should be utilized, with FIML often being the most straightforward and effective choice for CFA [@brown2015; @kline2023].

FIML and MI are preferred for handling missing data due to their ability to produce consistent and efficient parameter estimates under conditions similar to MI [@enders2023; @kline2023]. FIML it can be adapted for non-normal data using robust estimators [@brown2015].

Calculating the Variance Inflation Factor (VIF) helps identify items with problematic multicollinearity (VIF \> 10), which should be addressed to prevent model convergence issues and misinterpretations [@kline2016; @whittaker2022]. Reflective constructs in CFA require some level of item correlation but not to the extent that it causes statistical or validity concerns.

Consider multivariate outliers rather than univariate ones, identifying and assessing their exclusion based on sample characteristics. Reporting all data cleaning processes, including any loss of items and strategies for assessing respondent engagement, is crucial for transparency. Additionally, documenting signs of multicollinearity and the software or packages used (with versions) enhances the reproducibility and credibility of the research [@flake2020; @jackson2009].

Finally, making raw data public adheres to the principles of open science, promoting transparency and allowing for independent validation of research findings [@crede2019; @flake2022; @flake2020]. This practice not only contributes to the scientific community's collective knowledge base but also reinforces the integrity and reliability of the research conducted.

## Estimation Process

In CFA with ordinal items, such as those involving Likert-type scales with up to five points, @rogers2024 advocates for the use of estimators from the Ordinary Least Squares (OLS) family. Specifically, for smaller samples, the recommendation is to utilize the Unweighted Least Squares (ULS) in its robust form (RULS), and for larger samples, the Diagonally Weighted Least Squares (DWLS) in its robust version (RDWLS), citing substantial supporting research.

Despite this, empirical evidence [@rhemtulla2012; @robitzsch2022] and theoretical considerations [@robitzsch2020] suggest that treating ordinal data as continuous can yield acceptable outcomes when the response options number five or more. Particularly with 6-7 categories, comparisons between methods under various conditions reveal little difference, and it is recommended to use a greater number of response alternatives (≥5) to enhance the power for detecting model misspecifications [@maydeu-olivares2017a].

The ML estimator, noted for its robustness to minor deviations from normality [@brown2015], is further improved by using robust versions like MLR (employing Huber-White standard errors and Yuan-Bentler scaled $\chi^2$. This adjustment allows for generating robust standard errors and adjusted test statistics, with MLR offering extensive applicability including in scenarios of missing data (RFIML) or where data breaches the independence of observations assumption [@brown2015; @rosseel2012]. Comparative empirical studies have supported the effectiveness of MLR against alternative estimators [@bandalos2014; @holgado-tello2016; @li2016cfa; @nalbantoglu-yilmaz2019; @yang2013; @yang-wallentin2010].

Researchers are advised to carefully describe and justify the chosen estimation method based on the data characteristics and the specific model being evaluated [@crede2019]. It is also critical to report any estimation challenges encountered, such as algorithm non-convergence or model misidentification [@nye2022]. In case of estimation difficulties, alternative approaches like MLM estimators (employing robust standard errors and Satorra-Bentler scaled $\chi^2$) or the default ML with non-parametric bootstrapping, as proposed by Bollen-Stine, can be considered. This latter approach is also capable of accommodating missing data [@brown2015; @kline2023].

Additionally, it is important to clarify whether the variance of a marker variable was fixed (=1) to scale the latent variables [@jackson2009], and to provide both standardized and unstandardized parameter estimates [@nye2022]. These steps are crucial for ensuring transparency, reproducibility, and the ability to critically assess the validity of the CFA results.

## Model Fit

In conducting CFA with ordinal items, such as Likert-type scales, it's crucial to approach model evaluation with nuance and avoid reliance on rigid cutoff values for fit indices. Adhering strictly to traditional cutoffs – whether more conservative (e.g., SRMR ≤ .06, RMSEA ≤ .06, CFI ≥ .95) or less conservative (e.g., RMSEA ≤ .08, CFI ≥ .90, SRMR ≤ .08) – should not be the sole criterion for model acceptance [@xia2019]. The origins of these thresholds are in simulation studies with specific configurations (up to three factors, fifteen items, factor loadings between 0.7 and 0.8) [@west2023], and may not universally apply due to the variance in the number of items, factors, model degrees of freedom, misfit types, and presence of missing data [@groskurth2023; @niemand2018; @west2023].

Evaluation of global fit indices (SRMR, RMSEA, CFI) should be done in a collective manner, rather than fixating on any single index. A deviation from traditional cutoffs warrants further investigation into whether the discrepancy is attributable to data characteristics or limitations of the index, rather than indicating a fundamental model misspecification [@nye2022]. Interpreting fit indices as effect sizes can offer a more meaningful assessment of model fit, aligning with their original conceptualization [@mcneishwolf2023cfa; @mcneish2023geral].

The SRMR is noted for its robustness across various conditions, including non-normality and different measurement levels of items. Pairing SRMR with CFI can help balance Type I and Type II errors, but reliance on alternative indices may increase the risk of Type I error [@mai2021; @niemand2018].

Emerging methods like the Dynamic Fit Index (DFI) and Flexible Cutoffs (FCO) offer tailored approaches to evaluating global fit. DFI, based on simulation, provides model-specific cutoff points, adjusting simulations to match the empirical model's characteristics [@mcneish2023likert; @mcneishwolf2023dddf; @mcneishwolf2023cfa]. FCO, while not requiring identification of a misspecified model like DFI, conservatively defines misfit, shifting focus from approximate to accurate fit [@mcneishwolf2023dddf].

For those hesitant to delve into simulation-based methods, Equivalence Testing (EQT) presents an alternative. EQT aligns with the analytical mindset of PA and incorporates DFI principles, challenging the conventional hypothesis testing framework by considering model specification and misspecification size control [@yuan2016].

When addressing reliability, Cronbach's Alpha should not be the default measure due to its limitations. Instead, consider McDonald's Omega or the Greatest Lower Bound (GLB) for a more accurate reliability assessment within the CFA context [@bell2023; @cho2022; @dunn2014; @flora2020; @goodboy2020; @green2015; @hayes2020; @kalkbrenner2023; @mcneish2018; @trizano-hermosilla2016].

Before modifying the model, first check for Heywood instances, which are standardized factor loadings greater than one or negative variances [@nye2022] and document the chosen cutoffs for evaluation. Tools and resources like ShinyApp for DFI and the FCO package in R can facilitate the application of these advanced methodologies [@mcneishwolf2023cfa; @mai2021; @niemand2018]. Always report corrected chi-square and degrees of freedom, alongside a minimum of three global fit indices (RMSEA, CFI, SRMR) and local fit measures to provide a comprehensive view of model fit and adjustment decisions [@crede2019; @flake2020].

## Model Comparisons and Modifications

Researchers embarking on CFA should avoid prematurely committing to a specific factor structure without thoroughly evaluating and comparing alternate configurations. It's advisable to consider various potential structures early in the study design, ensuring the selected model is based on its merits relative to competing theories [@jackson2009]. Since models are inherently approximations of reality, adopting the most effective "working hypothesis" is a dynamic process, contingent on ongoing assessments against emerging alternatives [@preacher2023].

Good models are characterized not only by their interpretability, simplicity, and generalizability but notably by their capacity to surpass competing models in critical aspects. This competitive advantage frames the selected theory as the prevailing hypothesis until a more compelling alternative is identified [@preacher2023].

The evaluation of model fit should extend beyond isolated assessments using fit indices. A comprehensive approach involves comparing multiple models, each grounded in substantiated theories, to discern the most accurate representation of the underlying structure. This comparative analysis is preferred over singular model evaluations, fostering a more holistic understanding of the phenomena under study [@preacher2023].

Uniform application of models across the same dataset, utilizing identical software and sample size, ensures consistency in the researcher's analytical freedom, mitigating the risk of results manipulation. This standardized approach underpins a more rigorous and transparent investigative process [@preacher2023].

Model selection is instrumental in pinpointing the most effective explanatory framework for the observed phenomena, enabling the dismissal of less performance models while retaining promising ones for further exploration. This methodological flexibility enhances the depth of analysis, contributing to the advancement of knowledge within the social sciences [@preacher2023].

Adjustments to a model, particularly in response to unsatisfactory fit indices, should be theoretically grounded and reflective of findings from prior research. Blind adherence to a pre-established model or making hasty modifications can adversely affect the structural model's integrity. Thoughtful adjustments, potentially revisiting exploratory factor analysis (EFA) or considering Exploratory SEM (ESEM) for cross-loadings representation, are preferable to drastic changes that might shift the study from confirmatory to exploratory research [@brown2023; @flake2017; @jackson2009; @crede2019].

All modifications to the measurement model, especially those enhancing model fit, must be meticulously documented to maintain transparency and support reproducibility [@flake2020]. Openly reporting these adjustments, including item exclusions and inter-item correlations, is vital for the scientific integrity of the research [@nye2022; @flake2022].

Regarding model comparison and selection, traditional fit indices (SRMR, RMSEA, CFI) have limitations for direct model comparisons. Adjusted chi-square tests and information criteria like AIC and BIC are more suitable for this purpose, balancing model fit and parsimony. These criteria, however, should be applied with an understanding of their constraints and complemented by theoretical judgment to inform model selection decisions [@preacher2023; @brown2015; @huang2017; @lai2020nonnested; @lai2021fit].

Ultimately, model selection in SEM is a nuanced process, blending empirical evidence with theoretical insights. Researchers are encouraged to leverage a range of models based on theoretical foundations, ensuring that the eventual model selection is not solely determined by statistical criteria but is also informed by substantive theory and expertise [@preacher2023]. This balanced approach underscores the importance of theory-driven research in the social sciences, guiding the interpretation and application of findings derived from chosen models.