Testing suite

[grantdadams]

@Cole-Monnahan-NOAA think through any additional tests to add to testing suite.

We currently have tests to:

File Location	Test Name	Brief Overview
test-env-linkages.R	Test environmental linkage with mean rec	Verifies that a single environmental index correctly scales recruitment.
	Test multiple recruitment linkages...	Validates recruitment calculations when multiple environmental indices are used.
	Test multiple M linkages	Checks natural mortality (M) calculations influenced by multiple environmental indices.
	Test single M, multiple M/sex linkages...	Tests complex M configurations, including sex-specific and time-varying environmental effects.
test-fit-sanity-model.R	Fit sanity model: match baseline	Compares current model outputs (R0, SSB, biomass) against a saved .rds baseline to detect regressions.
	Index, biomass, and catch = 0 match	Verifies population scaling and zero-catch logic in a simplified scenario.
	Dynamics match CEATTLE classic (SS)	Validates single-species results against legacy ADMB/TMB CEATTLE outputs.
	Dynamics match CEATTLE classic (MS)	Validates multi-species results against legacy ADMB/TMB CEATTLE outputs.
test-growth-sizesel.R	size-based logistic selectivity	Tests length-based selectivity and its conversion to age-based selectivity via the growth matrix.
test-index-calculations.R	Basic index and index likelihood	Validates survey index abundance/biomass calculations and the resulting log-likelihood.
test-input-validation.R	Input validation for fit_mod	Checks error handling for missing data or incorrectly formatted input structures.
test-logistic-selectivity.R	logistic selectivity (various)	A suite of 5 tests checking logistic selectivity normalization (max, age-specific, or range) and time-varying versions.
test-m1-input-output.R	Test update M1 from data	Ensures that M1_at_age is correctly updated and read from the user-provided M1_base data if updateM1 = TRUE .
test-multi-spp-model.R	Rceattle vs. multi-species model	Validates the full suite of multi-species population dynamics against an independent R-based simulator.
	Equilibrium MSVPA dynamics	Specifically validates the convergence and accuracy of empirical MSVPA suitability and predation mortality (M2).
	Diet proportion tests (various)	A suite of 8 tests verifying that different diet data aggregations (by age, sex, or year) are correctly predicted.
	Test joint single-species models	Verifies that the multi-species engine correctly reduces to single-species dynamics when interactions are disabled.
test-nonparametric-selectivity.R	non-parametric selectivity	Validates Ianelli/AMAK style non-parametric selectivity and associated curvature/descending penalties.
test-parameter-recovery.R	Parameter recovery	A basic check to ensure the optimizer moves parameters from their initial values and produces finite results.
test-recruitment.R	mean recruitment and devs	Verifies the basic recruitment equation: $R = \bar{R} \cdot e^{\text{dev}}$.
	ssb under mean recruitment	Validates equilibrium SSB calculation when recruitment is held at its mean.
	ssb and beverton/ricker	Validates equilibrium SSB and R0 derivation for specific stock-recruit relationships.
test-SBF.R	Test SB40 under mean recruitment	Verifies that the NPFMC Tier 3 Harvest Control Rule (HCR) correctly targets 40% of $SB_0$.
	Test SPR0 calculation	Validates the algebraic derivation of Spawning Potential Ratio at $F=0$.
test-simulated-model-TVM.R	Test IID/AR1 time-varying M	Validates the implementation and likelihood of IID and AR1 random effects applied to natural mortality.
test-tmb-makeadfun_smoke.R	MakeADFun smoke test	A fast test to ensure the TMB object constructs properly and returns finite objective and gradient values.

[Cole-Monnahan-NOAA]

I would say some testing of the plotting and output functions

[Bai-Li-NOAA]

I found the discussion here, so I’m going to provide the testing review here instead of opening an issue. This way, you can decide which suggestions should be turned into issues and how to prioritize them. Feel free to let me know if you have any questions! All suggestions below are based on a review of commit d3618d2 from the dev branch.

High priority tasks before early summer

• Expand GitHub Actions (GHA) triggers to the dev branch

Currently, R CMD Check GHAs are only running on master, so tests on dev are not being automatically validated yet. Since development work happens on the dev branch, adding CI checks for pushes and pull requests to dev would help catch issues earlier in the development cycle before they reach master.

At the same time, it may be worth considering a transition to reusable workflows from {ghactions4r} (e.g., ghactions4r::use_r_cmd_check()), since these workflows reduce long-term maintenance on your side. They also automatically handle uploading artifacts and diagnostic logs when workflows fail, which makes troubleshooting easier.

The vignettes are also acting as lightweight integration/system tests. Because of that, reusable workflows like ghactions4r::use_update_pkgdown() and ghactions4r::use_build_pkgdown() could be used to handle pkgdown-related CI steps in a more consistent way and reduce workflow maintenance overall.

• Add targeted unit tests for important functions not already covered by integration tests

The project already has a strong set of integration tests, especially in folders like tests/testthat/tests-Mortality and tests/testthat/tests-Dynamics, and some individual functions (e.g., fit_mod()) are already well covered. However, because {covr} is not currently functioning correctly due to some skipped tests, it is difficult for me to identify missing coverage. In the short term (before early summer), maybe go through the R/ directory, find important functions that aren’t already covered by integration tests from memory, and add focused unit tests for those.

Copilot or similar coding assistants could also help generating initial tests. Mocking feature from {testthat} could be useful if you want to test outputs without adding too many full integration tests before the deadline.

• Expand output testing using snapshot-style tests

I agree with Cole here, one potential test would be a small set of end-to-end system tests that 1) fit a model using a known input, 2) save outputs in csv, 3) compare outputs using a snapshot tests (see FIMS expect_snapshot() example here). This would help catch unintended changes in model outputs, reporting functions, tables, and plotting behavior.

Long-term tasks after early summer

• Implement automated code coverage reporting

Automated coverage reporting would make it much easier to see gaps in testing over time. ghactions4r::use_calc_cov_summaries() and ghactions4r::use_create_cov_badge() can be used to automatically generate coverage summaries for pull requests and pushes to master.

Right now, {covr} skips some tests because of testthat::skip_on_cran() and possibly because some tests live in subfolders. Longer term, it would be worth digging into how to make sure the full test suite is included in coverage calculations. Once that’s sorted, you could use x <- covr::package_coverage(); covr::report(x) to quickly identify missing coverage.

• Standardize future tests

The integration tests are already well organized and easy to follow. As more tests get added, it would help to keep a consistent structure.
You could also reuse the FIMS testthat template (e.g., usethis::use_standalone("NOAA-FIMS/FIMS", file = "use-testthat-template", ref = "dev-add-standalone-file")) to bring in a standard pattern. Within each file, tests could be grouped into 1) input/output correctness tests, 2) edge-case tests, 3) error-message tests. This testing template works well with Copilot for adding new tests, since it can pick up consistent patterns more easily.

• Rebalance the test suite to improve runtime and developer workflow

The test suite currently takes a bit of time to run. Longer term, it might be worth rethinking the balance between integration and unit tests. Some slower integration tests could potentially be replaced with smaller, faster unit tests while still keeping good overall coverage. This is something being explored in the FIMS test suite as well, and I can share more concrete ideas later if helpful.

Feel free to assign me any tasks if you need help with this. I’m a bit torn about when to fix the {covr} setup. If it would be useful to get it done sooner rather than later, we could also set aside a couple of hours to work on it together.