testthat issue with latest R devel on windows?

[strengejacke]

I'm not quite sure about the source of this error, which I got as feedback from the CRAN team after submitting an update.

What is weird is that the first line of the test that fails is skip_on_cran(), so I wonder why this test is run at all? Could this be a testthat issue, or a temporary technical issue?

* using log directory 'd:/RCompile/CRANincoming/R-devel/insight.Rcheck'
* using R Under development (unstable) (2025-11-11 r89006 ucrt)
* using platform: x86_64-w64-mingw32
* R was compiled by
    gcc.exe (GCC) 14.3.0
    GNU Fortran (GCC) 14.3.0
* running under: Windows Server 2022 x64 (build 20348)
* using session charset: UTF-8
* checking for file 'insight/DESCRIPTION' ... OK
* checking extension type ... Package
* this is package 'insight' version '1.4.3'
* package encoding: UTF-8
* checking CRAN incoming feasibility ... [14s] OK
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking whether package 'insight' can be installed ... OK
* checking installed package size ... OK
* checking package directory ... OK
* checking for future file timestamps ... OK
* checking 'build' directory ... OK
* checking DESCRIPTION meta-information ... OK
* checking top-level files ... OK
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking code files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* checking whether the package can be loaded ... OK
* checking whether the package can be loaded with stated dependencies ... OK
* checking whether the package can be unloaded cleanly ... OK
* checking whether the namespace can be loaded with stated dependencies ... OK
* checking whether the namespace can be unloaded cleanly ... OK
* checking loading without being on the library search path ... OK
* checking use of S3 registration ... OK
* checking dependencies in R code ... OK
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... [51s] OK
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd line widths ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking contents of 'data' directory ... OK
* checking data for non-ASCII characters ... OK
* checking data for ASCII and uncompressed saves ... OK
* checking installed files from 'inst/doc' ... OK
* checking files in 'vignettes' ... OK
* checking examples ... [45s] OK
* checking for unstated dependencies in 'tests' ... OK
* checking tests ... [45s] ERROR
  Running 'testthat.R' [45s]
Running the tests in 'tests/testthat.R' failed.
Complete output:
  > library(testthat)
  > library(insight)
  > test_check("insight")
  Starting 2 test processes.
  > test-find_transformation.R: boundary (singular) fit: see help('isSingular')
  > test-gamlss.R: GAMLSS-RS iteration 1: Global Deviance = 365.2328 
  > test-gamlss.R: GAMLSS-RS iteration 2: Global Deviance = 365.1292 
  > test-gamlss.R: GAMLSS-RS iteration 3: Global Deviance = 365.1269 
  > test-gamlss.R: GAMLSS-RS iteration 4: Global Deviance = 365.1268 
  > test-gamlss.R: GAMLSS-RS iteration 1: Global Deviance = 5779.746 
  > test-gamlss.R: GAMLSS-RS iteration 2: Global Deviance = 5779.746 
  > test-gamlss.R: GAMLSS-RS iteration 1: Global Deviance = 703.1164 
  > test-gamlss.R: GAMLSS-RS iteration 2: Global Deviance = 703.1164 
  > test-get_model.R: Loading required namespace: GPArotation
  > test-get_random.R: boundary (singular) fit: see help('isSingular')
  > test-glmmPQL.R: iteration 1
  > test-lcmm.R: 
  
  Error:
  ! testthat subprocess exited in file 'test-lcmm.R'.
  Caused by error:
  ! R session crashed with exit code -1073741819
  Backtrace:
       ▆
    1. └─testthat::test_check("insight")
    2.   └─testthat::test_dir(...)
    3.     └─testthat:::test_files(...)
    4.       └─testthat:::test_files_parallel(...)
    5.         ├─withr::with_dir(...)
    6.         │ └─base::force(code)
    7.         ├─testthat::with_reporter(...)
    8.         │ └─base::tryCatch(...)
    9.         │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
   10.         │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
   11.         │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
   12.         └─testthat:::parallel_event_loop_chunky(queue, reporters, ".")
   13.           └─queue$poll(Inf)
   14.             └─base::lapply(...)
   15.               └─testthat (local) FUN(X[[i]], ...)
   16.                 └─private$handle_error(msg, i)
   17.                   └─cli::cli_abort(...)
   18.                     └─rlang::abort(...)
  Execution halted
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes ... OK
* checking re-building of vignette outputs ... OK
* checking PDF version of manual ... [16s] OK
* checking HTML version of manual ... [19s] OK
* DONE
Status: 1 ERROR

[hadley]

Hard to know more without seeing the test.

[strengejacke]

Here's the related testfile:
https://github.com/easystats/insight/blob/main/tests/testthat/test-lcmm.R

I thought it was temporary, because CRAN wasn't suppose to check that file. However, after resubmitting yesterday, I now get:

https://win-builder.r-project.org/incoming_pretest/insight_1.4.3_20251120_121429/Debian/00check.log

* using log directory ‘/srv/hornik/tmp/CRAN_pretest/insight.Rcheck’
* using R Under development (unstable) (2025-11-19 r89040)
* using platform: x86_64-pc-linux-gnu
* R was compiled by
    Debian clang version 21.1.5 (1)
    Debian flang version 21.1.5 (1)
* running under: Debian GNU/Linux forky/sid
* using session charset: UTF-8

...

  > test-rms.R: fatal Fortran runtime error(/tmp/RtmpLOLCZO/R.INSTALLaccff22fea350/rms/src/lrmll.f90:57): Internal error: RUNTIME_CHECK(maskType && maskType->first == TypeCategory::Logical) failed at flang/runtime/transformational.cpp(639)
  
  Error:
  ! testthat subprocess exited in file 'test-rms.R'.
  Caused by error:
  ! R session crashed with exit code -6
  Backtrace:
       ▆
    1. └─testthat::test_check("insight")
    2.   └─testthat::test_dir(...)
    3.     └─testthat:::test_files(...)
    4.       └─testthat:::test_files_parallel(...)
    5.         ├─withr::with_dir(...)
    6.         │ └─base::force(code)
    7.         ├─testthat::with_reporter(...)
    8.         │ └─base::tryCatch(...)
    9.         │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
   10.         │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
   11.         │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
   12.         └─testthat:::parallel_event_loop_chunky(queue, reporters, ".")
   13.           └─queue$poll(Inf)
   14.             └─base::lapply(...)
   15.               └─testthat (local) FUN(X[[i]], ...)
   16.                 └─private$handle_error(msg, i)
   17.                   └─cli::cli_abort(...)
   18.                     └─rlang::abort(...)
  Execution halted
* checking for unstated dependencies in vignettes ... OK
* checking package vignettes ... OK
* checking re-building of vignette outputs ... [7s/7s] OK
* checking PDF version of manual ... [7s/7s] OK
* checking HTML version of manual ... [7s/7s] OK
* checking for non-standard things in the check directory ... OK
* checking for detritus in the temp directory ... OK
* DONE
Status: 1 ERROR

The testfile is here:
https://github.com/easystats/insight/blob/main/tests/testthat/test-rms.R

However, we haven't changed this test for a longer time, so I wonder why we now get this error. Maybe it's totally unrelated to testthat, but especially the first example was strange, since the test shouldn't be run on CRAN, and I wonder why we got that error.

[hadley]

Is it just randomly failing on some test file? The two failures crash on different test files.

[strengejacke]

Yes, seems to be random. Especially the rms-test did not fail on CRAN in the past, and we haven't changed anything to it (except maybe styling with Air).

The first issue no longer occurs, so it seemed to be temporary.

[JanMarvin]

Just to provide a second opinion: I too see random testthat crashes on windows. The one below on CRAN and similar crashes on GitHub CI, but usually with R 4.1 on Windows. This started after testthat 3.3.0, but obviously it could something in my code.

https://www.r-project.org/nosvn/R.check/r-release-windows-x86_64/openxlsx2-00check.html

I don’t keep the CI logs, but there was no additional indication what is going on. Since the CI failures were on R 4.1 I assumed that it was related to something else and did not give it much thought.

[strengejacke]

I now skipped the test (which was related to the rms package) for my package submission, and everything is fine now with CRAN.

However, today I submitted another package, and again I got a failure for a test related to the rms package. Again a failing test that caused no problems in former package submissions:

https://win-builder.r-project.org/incoming_pretest/parameters_0.28.3_20251124_155308/Debian/00check.log

Complete output:
  > library(parameters)
  > library(testthat)
  > 
  > test_check("parameters")
  Starting 2 test processes.
  > test-Hmisc.R: 
  > test-Hmisc.R: fatal Fortran runtime error(/tmp/RtmpLOLCZO/R.INSTALLaccff22fea350/rms/src/ormll.f90:97): Internal error: RUNTIME_CHECK(maskType && maskType->first == TypeCategory::Logical) failed at flang/runtime/transformational.cpp(639)
  
  Error:
  ! testthat subprocess exited in file 'test-Hmisc.R'.
  Caused by error:
  ! R session crashed with exit code -6
  Backtrace:
       ▆
    1. └─testthat::test_check("parameters")
    2.   └─testthat::test_dir(...)
    3.     └─testthat:::test_files(...)
    4.       └─testthat:::test_files_parallel(...)
    5.         ├─withr::with_dir(...)
    6.         │ └─base::force(code)
    7.         ├─testthat::with_reporter(...)
    8.         │ └─base::tryCatch(...)
    9.         │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
   10.         │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
   11.         │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
   12.         └─testthat:::parallel_event_loop_chunky(queue, reporters, ".")
   13.           └─queue$poll(Inf)
   14.             └─base::lapply(...)
   15.               └─testthat (local) FUN(X[[i]], ...)
   16.                 └─private$handle_error(msg, i)
   17.                   └─cli::cli_abort(...)
   18.                     └─rlang::abort(...)
  Execution halted

Here's the test file:
https://github.com/easystats/parameters/blob/main/tests/testthat/test-Hmisc.R

Not sure if there's some strange interaction between testthat and rms?

[strengejacke]

After skipping that test on CRAN, the submission passes CRAN checks.

[JanMarvin]

The last of your crashes looks like a fortran error on Winbuilder @strengejacke.

Another of the access violation crashes. This time the Windows CI. There is nothing out of the ordinary in test-options.R. I assume that testthat parallel teardown is somehow colliding. In openxlsx2 we create zip files via tempfile() and tempdir(). The testsuite is ~2k cases with many created files which is probably why I see these more often. Still not sure how to debug this. Might have to disable parallel testing for a while.

── Test failures ───────────────────────────────────────────────── testthat ────

> library(testthat)
> library(openxlsx2)
> 
> test_check("openxlsx2")
Starting 2 test processes.
> test-options.R: 

Error:
! testthat subprocess exited in file 'test-options.R'.
Caused by error:
! R session crashed with exit code -1073741819
Backtrace:
     ▆
  1. └─testthat::test_check("openxlsx2")
  2.   └─testthat::test_dir(...)
  3.     └─testthat:::test_files(...)
  4.       └─testthat:::test_files_parallel(...)
  5.         ├─withr::with_dir(...)
  6.         │ └─base::force(code)
  7.         ├─testthat::with_reporter(...)
  8.         │ └─base::tryCatch(...)
  9.         │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 10.         │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 11.         │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 12.         └─testthat:::parallel_event_loop_chunky(queue, reporters, ".")
 13.           └─queue$poll(Inf)
 14.             └─base::lapply(...)
 15.               └─testthat (local) FUN(X[[i]], ...)
 16.                 └─private$handle_error(msg, i)
 17.                   └─cli::cli_abort(...)
 18.                     └─rlang::abort(...)
Execution halted

Attached is the log if it's of any use: Windows-X64-rrelease-1-results.zip

And another one

-- Test failures ------------------------------------------ testthat (i386) ----

> library(testthat)
> library(openxlsx2)
> 
> test_check("openxlsx2")
Starting 2 test processes.
> test-class-worksheet.R: 

Error:
! testthat subprocess exited in file 'test-class-worksheet.R'.
Caused by error:
! R session crashed with exit code -1073741819
Backtrace:
     x
  1. \-testthat::test_check("openxlsx2")
  2.   \-testthat::test_dir(...)
  3.     \-testthat:::test_files(...)
  4.       \-testthat:::test_files_parallel(...)
  5.         +-withr::with_dir(...)
  6.         | \-base::force(code)
  7.         +-testthat::with_reporter(...)
  8.         | \-base::tryCatch(...)
  9.         |   \-base (local) tryCatchList(expr, classes, parentenv, handlers)
 10.         |     \-base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 11.         |       \-base (local) doTryCatch(return(expr), name, parentenv, handler)
 12.         \-testthat:::parallel_event_loop_chunky(queue, reporters, ".")
 13.           \-queue$poll(Inf)
 14.             \-base::lapply(...)
 15.               \-testthat (local) FUN(X[[i]], ...)
 16.                 \-private$handle_error(msg, i)
 17.                   \-cli::cli_abort(...)
 18.                     \-rlang::abort(...)
Execution halted

[strengejacke]

On winbuilder, everything passes fine. It's the CRAN submission that fails

[JanMarvin]

Still not sure what is going on, but after several days trying to avoid what I assumed to be a race condition in openxlsx2 there are still these spurious access violation errors. And now I'm rather sure that it is not my code, but testthat that is causing these crashes.

Initially I've added Config/testthat/parallel: true to the DESCRIPTION of xml2, ran a couple of testthat runs (now in parallel) and the access violation crash appears. (I picked xml2 because I was guessing that the crash might have been caused by XPtr.)

Once I confirmed that the access violation was probably not my fault, I searched testthat's own CI logs and the access violation crash appears at least in three CI runs here and here, the third was somewhere in between the two. Therefore I conclude that whatever is causing the access violation is something in testthat.

What I tried narrowing it down and did not help:

• downgrading testthat to 3.2.3
• downgrading pkgload to pre 1.4.0

The access violation crash appear sporadic roughly every dozen runs and only on WIndows. Since I don't use Windows for development it might have been there forever or caused by a recent change either in R or in testthat and its dependencies. But the testthat sources and those of its dependencies are a little to large and widespread for my appetite to further dig into it. I'll disable parallel testing on CRAN for now.

==> devtools::test()

ℹ Testing xml2
Starting 2 test processes.
✔ | F W  S  OK | Context
⠋ [ FAIL 0 | WARN 0 | SKIP 0 | PASS 0 ] Starting up...                                                                                                  
> test-as_xml_document.R: 

Error:
! testthat subprocess exited in file test-as_xml_document.R.
Caused by error:
! R session crashed with exit code -1073741819
Backtrace:
     ▆
  1. └─devtools::test()
  2.   └─testthat::test_local(...)
  3.     └─testthat::test_dir(...)
  4.       └─testthat:::test_files(...)
  5.         └─testthat:::test_files_parallel(...)
  6.           ├─withr::with_dir(...)
  7.           │ └─base::force(code)
  8.           ├─testthat::with_reporter(...)
  9.           │ └─base::tryCatch(...)
 10.           │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 11.           │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 12.           │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 13.           └─testthat:::parallel_event_loop_smooth(queue, reporters, ".")
 14.             └─queue$poll(poll_time)
 15.               └─base::lapply(...)
 16.                 └─testthat (local) FUN(X[[i]], ...)
 17.                   └─private$handle_error(msg, i)
 18.                     └─cli::cli_abort(...)
 19.                       └─rlang::abort(...)
Ausführung angehalten

Exited with status 1.

> sessionInfo()
R version 4.5.1 (2025-06-13 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26200)

Matrix products: default
  LAPACK version 3.12.1

locale:
[1] LC_COLLATE=German_Germany.utf8  LC_CTYPE=German_Germany.utf8    LC_MONETARY=German_Germany.utf8 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.utf8    

time zone: Europe/Berlin
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] xml2_1.5.1

loaded via a namespace (and not attached):
[1] compiler_4.5.1    cli_3.6.5         tools_4.5.1       rstudioapi_0.17.1 rlang_1.1.6