Segfault in cmdstan_model(..., compile_model_methods = TRUE) on Ubuntu GitHub Actions

[MansMeg]

Describe the bug
I’m seeing a segfault in cmdstanr when model methods are compiled. The crash happens before sampling starts, during model construction, and it reproduces in a fresh R process without loading rstan.

I originally saw this intermittently in CI, so I created a small diagnostic workflow. In the attached cmdstanr_only artifact, the very first run segfaults in a fresh process. I also tested a mixed mode where an rstan fit runs first; that fit succeeds, and then the same cmdstanr step segfaults in the same place.

To Reproduce
Environment:

OS: GitHub Actions ubuntu-latest
R: release via r-lib/actions/setup-r@v2

Minimal reproduction

Using the attached Stan file model8k5.stan, in a fresh R session:

library(cmdstanr)

cat("cmdstanr:", as.character(packageVersion("cmdstanr")), "\n")
cat("CmdStan:", as.character(cmdstan_version()), "\n")
sessionInfo()

mod <- cmdstan_model(
  stan_file = "model8k5.stan",
  compile = TRUE,
  compile_model_methods = TRUE
)

Expected behavior
cmdstan_model() should return a CmdStanModel object.

Actual behavior
R aborts with a segfault during model-method compilation/loading. The relevant stack looks like:

cause 'invalid permissions'
dyn.load("/tmp/.../sourceCpp_2.so")
Rcpp::sourceCpp(...)
expose_model_methods(...)
CmdStanModel$new(... compile_model_methods = TRUE)

In the failing trace, the crash occurs before sampling, while cmdstanr is compiling/loading the generated model-method shared object.

Operating system
Environment
OS: GitHub Actions ubuntu-latest
R: release via r-lib/actions/setup-r@v2
cmdstanr: 0.9.0
CmdStan: 2.38.0
CmdStan installed via cmdstanr::install_cmdstan(cores = 2, overwrite = FALSE)

CmdStanR version number
0.9.0

Additional context
The same model works if I do not request model methods.

The cmdstanr_only reproduction is the strongest signal that this is not primarily an rstan/cmdstanr interaction.

In mixed mode, an rstan fit and log_prob() call succeed first, and then the cmdstanr step crashes in the same place.

If helpful, I also provide the full CI run artifacts and the small diagnostic script/workflow I used:

cmdstanr-model-methods-diagnostic-mixed.zip
cmdstanr-model-methods-diagnostic-cmdstanr_only.zip
model8k5.stan.zip

[jgabry]

Interesting, we have tests on ubuntu for compiling the model methods that pass CI and I've never seen them fail even intermittently.

Your model compiles fine with compile_model_methods=TRUE when I try it, but that's just locally on my Mac.

Are you able to try using the latest development version of cmdstanr from GitHub (on the master branch) and see if you get the same error?

[avehtari]

The code works for me with

R version 4.5.3 (2026-03-11)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.5 LTS

[1] cmdstanr_0.9.0.9000

and CmdStan 2.38.0 (also tested with CmdStan development version)

I tested also with cmdstanr: 0.9.0 and model methods compile fine. So in my computer matching CmdStan and CmdStanR versions, all fine.

[VisruthSK]

I found this old discourse post which provides a workaround, which might be helpful? https://discourse.mc-stan.org/t/segfault-when-using-brms-cmdstanr-compile-model-methods-true/33771

[avehtari]

I don't have STAN_THREADS=true and the above example works fine (with matching CmdStan and CmdStanR versions)

[avehtari]

@MansMeg your code example above doesn't have data, but you mention sampling and one of the log files show data. Can you clarify whether the segfault did come from cmdstan_model() call or later?

[MansMeg]

Sorry. Im still on easter holiday, so moving slowly. I have now checked, and indeed the running version of cmdstanr on CI is on 0.9.0.9000. Just like you, I don't have any problems running locally with 0.9 (on Mac), so I just commented out the test for now and filed this bug report. No rush from my side or need for workarounds. I attached the bug log.

Below is the actual unit test that segfaults. It's specific to the library, so it may not be clear whats going on.

test_that("model8k5 gives the same parameter space and log_prob with rstan and cmdstanr", {
  skip_if_no_stan_tests()
  skip_if_no_cmdstanr_tests()
#  skip_if_no_cmdstanr()
  assert_rstan_available()
#  skip("Temporarily skipped due to intermittent cmdstanr model-method segfault in CI.")

  case <- make_model8_mixed_smoke_case(npolls = 12)
  cfg <- list(
    sigma_kappa_hyper = 0.03,
    use_industry_bias = 1L,
    use_house_bias = 0L,
    use_design_effects = 0L,
    use_multivariate_version = 2L,
    use_softmax = 1L
  )

  fit_args <- list(
    y = case$parties,
    model = "model8k5",
    polls_data = case$polls_data,
    time_scale = case$time_scale,
    time_scale_overrides = case$time_scale_overrides,
    known_state = case$known_state,
    hyper_parameters = cfg,
    chains = 1,
    refresh = 0,
    seed = 4711,
    cache_dir = NULL
  )

  expect_silent(
    capture.output(
      suppressWarnings(
        suppressMessages(
          pop_rstan <- do.call(
            poll_of_polls,
            c(fit_args, list(
              backend = "rstan",
              iter = 10,
              warmup = 5
            ))
          )
        )
      )
    )
  )

  expect_silent(
    capture.output(
      suppressWarnings(
        suppressMessages(
          pop_cmdstanr <- do.call(
            poll_of_polls,
            c(fit_args, list(
              backend = "cmdstanr",
              iter_sampling = 5,
              iter_warmup = 5,
              compile_args = list(compile_model_methods = TRUE)
            ))
          )
        )
      )
    )
  )

  expect_identical(pop_rstan$time_scale_overrides, pop_cmdstanr$time_scale_overrides)
  expect_identical(pop_rstan$time_line$time_line, pop_cmdstanr$time_line$time_line)
  expect_equal(
    pop_rstan$stan_data$stan_data$step_scale_t,
    pop_cmdstanr$stan_data$stan_data$step_scale_t,
    tolerance = 0
  )

  expect_identical(parameter_names(pop_rstan), parameter_names(pop_cmdstanr))
  expect_identical(
    parameter_names(pop_rstan, rm_idx = TRUE),
    parameter_names(pop_cmdstanr, rm_idx = TRUE)
  )
  expect_identical(get_num_pars(pop_rstan), get_num_pars(pop_cmdstanr))

  nu_rstan <- get_num_upars(pop_rstan)
  nu_cmdstanr <- get_num_upars(pop_cmdstanr)
  probe <- seq(from = -0.15, to = 0.15, length.out = nu_rstan)

  expect_identical(nu_rstan, nu_cmdstanr)
  expect_equal(
    log_prob(pop_rstan, rep(0, nu_rstan)),
    log_prob(pop_cmdstanr, rep(0, nu_cmdstanr)),
    tolerance = 1e-8
  )
  expect_equal(
    log_prob(pop_rstan, probe),
    log_prob(pop_cmdstanr, probe),
    tolerance = 1e-8
  )

  sigma_x_rstan <- extract(pop_rstan, pars = "sigma_x")$sigma_x
  sigma_x_cmdstanr <- extract(pop_cmdstanr, pars = "sigma_x")$sigma_x
  x_pred_rstan <- extract(pop_rstan, pars = "x_pred")$x_pred
  x_pred_cmdstanr <- extract(pop_cmdstanr, pars = "x_pred")$x_pred
  ls_rstan <- latent_state(pop_rstan)
  ls_cmdstanr <- latent_state(pop_cmdstanr)
  md_rstan <- get_model_diagnostics(pop_rstan)
  md_cmdstanr <- get_model_diagnostics(pop_cmdstanr)

  expect_identical(dim(sigma_x_rstan), dim(sigma_x_cmdstanr))
  expect_identical(dim(x_pred_rstan), dim(x_pred_cmdstanr))
  expect_identical(dim(ls_rstan$latent_state), dim(ls_cmdstanr$latent_state))
  expect_identical(dimnames(ls_rstan$latent_state), dimnames(ls_cmdstanr$latent_state))
  expect_identical(names(md_rstan), names(md_cmdstanr))
  expect_true(all(is.finite(unlist(md_rstan))))
  expect_true(all(is.finite(unlist(md_cmdstanr))))
})

bug-log.txt

This is in a private repo now, but Im happy to invite you if you want to take a look. I will make the repo public later this spring.

This is the trace. So it seems to be when I call do.call(cmdstanr::cmdstan_model, c(list(stan_file = stan_file, compile = TRUE), compile_arguments))

*** caught segfault ***
address 0x7f1882b81008, cause 'invalid permissions'

Traceback:
 1: dyn.load("/tmp/RtmprxI6c1/working_dir/Rtmpdar1V0/sourceCpp-x86_64-pc-linux-gnu-1.1.1/sourcecpp_1d5e4f163a27/sourceCpp_2.so")
 2: eval(ei, envir)
 3: eval(ei, envir)
 4: withVisible(eval(ei, envir))
 5: source(scriptPath, local = env)
 6: Rcpp::sourceCpp(code = code, env = env, verbose = verbose, ...)
 7: force(code)
 8: force(code)
 9: with_envvar(c(R_MAKEVARS_USER = makevars_file), {    set_makevars(new, path, makevars_file, assignment = assignment)    force(code)})
10: withr::with_makevars(c(USE_CXX14 = 1, PKG_CPPFLAGS = cppflags,     PKG_CXXFLAGS = paste0(cxxflags, cmdstanr_includes, collapse = " "),     PKG_LIBS = libs), Rcpp::sourceCpp(code = code, env = env,     verbose = verbose, ...))
11: force(code)
12: withr::with_path(repair_path(file.path(cmdstan_path(), "stan/lib/stan_math/lib/tbb")),     withr::with_makevars(c(USE_CXX14 = 1, PKG_CPPFLAGS = cppflags,         PKG_CXXFLAGS = paste0(cxxflags, cmdstanr_includes, collapse = " "),         PKG_LIBS = libs), Rcpp::sourceCpp(code = code, env = env,         verbose = verbose, ...)))
13: rcpp_source_stan(code, env, verbose)
14: expose_model_methods(env = private$model_methods_env_, verbose = !quiet)
15: self$compile(...)
16: initialize(...)
17: CmdStanModel$new(stan_file = stan_file, exe_file = exe_file,     compile = compile, ...)
Error: Error: R CMD check found ERRORs
Execution halted
18: (function (stan_file = NULL, exe_file = NULL, compile = TRUE,     ...) {    if (is.null(exe_file) && is.null(stan_file)) {        stop("Unable to create a `CmdStanModel` object. ", "Both 'stan_file' and 'exe_file' are undefined.",             call. = FALSE)    }    CmdStanModel$new(stan_file = stan_file, exe_file = exe_file,         compile = compile, ...)})(stan_file = "/home/runner/work/stanpop/stanpop/check/stanpop.Rcheck/stanpop/stan_models/model8k5.stan",     compile = TRUE, compile_model_methods = TRUE)
19: do.call(cmdstanr::cmdstan_model, c(list(stan_file = stan_file,     compile = TRUE), compile_arguments))
20: backend_sample_cmdstanr(sample_arguments = sample_arguments,     stan_file = stan_file, compile_arguments = compile_arguments)
21: backend_sample(backend = backend, sample_arguments = backend_arguments,     stan_file = smfp, model_name = model, compile_arguments = compile_args)
22: (function (y, model, polls_data, time_scale, time_scale_overrides = NULL,     known_state = NULL, model_time_range = NULL, latent_time_ranges = NULL,     hyper_parameters = NULL, slow_scales = NULL, backend = "rstan",     compile_args = NULL, ..., cache_dir = file.path(tempdir(),         "pop_cache")) {    checkmate::assert_subset(x = y, choices = names(y(polls_data)))    if (checkmate::test_file_exists(model, extension = "stan")) {        smfp <- model        model <- remove_file_extension(basename(model))    }    else {        smfp <- get_pop_stan_model_file_path(model)    }    checkmate::assert_choice(model, choices = supported_pop_models())    assert_pop_backend(backend)    checkmate::assert_list(compile_args, null.ok = TRUE)    assert_polls_data(polls_data, min.rows = 1, min.cols = 1)    assert_known_state(known_state)    if (!is.null(known_state)) {        checkmate::assert_names(x = names(known_state), must.include = c("date",             y))    }    checkmate::assert_choice(time_scale, choices = supported_time_scales())    if (is.null(model_time_range)) {        mtr <- time_range(polls_data)    }    else {        mtr <- time_range(model_time_range)    }    assert_poll_data_in_time_line_using_time_range(polls_data,         time_scale, mtr)    assert_time_scale_overrides(x = time_scale_overrides, dates = tibble::tibble(date = seq(from = mtr["from"],         to = mtr["to"], by = 1)))    if (!is.null(time_scale_overrides) && nrow(time_scale_overrides) >         0 && !model_supports_time_scale_overrides(model)) {        stop("'time_scale_overrides' requires a model that uses 'step_scale_t'. ",             "Model '", model, "' does not support time scale overrides in fitting.",             call. = FALSE)    }    assert_latent_time_range_list(latent_time_ranges)    ltr <- setup_latent_time_ranges(x = latent_time_ranges, y,         mtr)    assert_poll_data_and_latent_time_range_list_agree(polls_data,         ltr)    checkmate::assert_list(hyper_parameters, null.ok = TRUE)    sd <- stan_polls_data(x = polls_data, time_scale = time_scale,         time_scale_overrides = time_scale_overrides, y_name = y,         model = model, known_state = known_state, model_time_range = mtr,         latent_time_ranges = ltr, hyper_parameters = hyper_parameters,         slow_scales = slow_scales)    if (!is.null(cache_dir)) {        if (!checkmate::test_directory(cache_dir))             dir.create(cache_dir)        checkmate::assert_directory(cache_dir)    }    fun_args <- names(formals(poll_of_polls))[-which(names(formals(poll_of_polls)) %in%         c("...", "cache_dir"))]    sha_fun_args <- list(y = y, model = readLines(smfp), polls_data = polls_data,         backend = backend, compile_args = compile_args, time_scale = time_scale,         time_scale_overrides = time_scale_overrides, known_state = known_state,         model_time_range = mtr, latent_time_ranges = ltr, hyper_parameters = hyper_parameters,         slow_scales = slow_scales)    checkmate::assert_set_equal(fun_args, names(sha_fun_args))    sha_stan_args <- list(data = sd$stan_data, ...)    sha <- digest::digest(c(sha_fun_args, sha_stan_args), algo = "sha1")    if (!is.null(cache_dir)) {        cache_fp <- cache_file_path(sha, cache_dir)        if (file.exists(cache_fp)) {            pop <- readRDS(file = cache_fp)            message("Cached results used.")            return(pop)        }        else {            dir.create(cache_dir, recursive = TRUE, showWarnings = FALSE)        }    }    backend_arguments <- list(...)    stan_arguments <- list(...)    if (!is.null(backend_arguments$data))         warning("The 'data' argument has been overwritten")    backend_arguments$data <- sd$stan_data    if (backend == "cmdstanr") {        cmdstanr_rstan_only_args <- c("file", "model_name", "control",             "iter", "warmup", "cores", "algorithm", "init_r")        found_rstan_only_args <- intersect(names(backend_arguments),             cmdstanr_rstan_only_args)        if (length(found_rstan_only_args) > 0) {            stop("With backend = 'cmdstanr', supply CmdStanR sample arguments directly in '...'. ",                 "Unsupported RStan-style arguments: ", paste0(found_rstan_only_args,                   collapse = ", "), ". Use e.g. 'iter_warmup', 'iter_sampling', 'parallel_chains', and 'compile_args'.",                 call. = FALSE)        }    }    if (backend == "rstan" && !is.null(compile_args) && length(compile_args) >         0) {        warning("'compile_args' is ignored when backend = 'rstan'.",             call. = FALSE)    }    stan_fit <- backend_sample(backend = backend, sample_arguments = backend_arguments,         stan_file = smfp, model_name = model, compile_arguments = compile_args)    pop <- list(y = y, model = model, backend = backend, compile_arguments = compile_args,         polls_data = polls_data, time_scale = time_scale, time_scale_overrides = time_scale_overrides,         known_state = known_state, model_time_range = mtr, latent_time_range = ltr,         stan_arguments = stan_arguments, sha = sha, input_args = list(y = y,             model = model, backend = backend, compile_args = compile_args,             time_scale = time_scale, time_scale_overrides = time_scale_overrides,             model_time_range = model_time_range, latent_time_ranges = latent_time_ranges,             hyper_parameters = hyper_parameters, slow_scales = slow_scales,             stan_arguments = stan_arguments, cache_dir = cache_dir),         git_sha = get_git_sha(), cache_dir = cache_dir, time_line = sd$time_line,         stan_fit = stan_fit, diagnostics = compute_diagnostics(stan_fit,             backend = backend), model_arguments = hyper_parameters,         stan_data = sd)    class(pop) <- c(paste0("pop_", model), "poll_of_polls")    assert_pop(pop)    if (!is.null(cache_dir))         saveRDS(pop, file = cache_fp)    pop})(y = c("x3", "x4"), model = "model8k5", polls_data = list(poll_info = list(    .poll_id = c("1", "2", "3", "4", "5", "6", "7", "8", "9",     "10", "11", "12"), .house = c(1L, 1L, 1L, 1L, 1L, 1L, 1L,     1L, 1L, 1L, 1L, 1L), .publish_date = c(15294, 15243, 15243,     15181, 15126, 15090, 15081, 14972, 14950, 14751, 14707, 14663    ), .start_date = c(15261, 15235, 15215, 15171, 15114, 15072,     15053, 14962, 14941, 14718, 14704, 14649), .end_date = c(15288,     15242, 15242, 15179, 15121, 15088, 15079, 14968, 14948, 14745,     14706, 14659), .n = c(1428L, 1808L, 1337L, 1250L, 1611L,     2699L, 2000L, 2000L, 1010L, 1414L, 1500L, 1946L)), y = list(    .poll_id = c("1", "2", "3", "4", "5", "6", "7", "8", "9",     "10", "11", "12"), x3 = c(0.239495798319328, 0.247787610619469,     0.25056095736724, 0.2552, 0.224705152079454, 0.220452019266395,     0.232, 0.235, 0.242574257425743, 0.264497878359264, 0.214,     0.26464542651593), x4 = c(0.142857142857143, 0.146017699115044,     0.136125654450262, 0.1792, 0.128491620111732, 0.161541311596888,     0.168, 0.217, 0.201980198019802, 0.27015558698727, 0.238,     0.246659815005139)), time_range = c(from = 14610, to = 15305)), time_scale = "week", time_scale_overrides = list(from = 14734,     to = 14739, time_scale = "day"), known_state = list(date = c(14911, 15107), x3 = c(0.240215178693441, 0.223311674991608), x4 = c(0.222196764834674, 0.142137024462568)), hyper_parameters = list(sigma_kappa_hyper = 0.03,     use_industry_bias = 1L, use_house_bias = 0L, use_design_effects = 0L,     use_multivariate_version = 2L, use_softmax = 1L), chains = 1,     refresh = 0, seed = 4711, cache_dir = NULL, backend = "cmdstanr",     iter_sampling = 5, iter_warmup = 5, compile_args = list(compile_model_methods = TRUE))
23: do.call(poll_of_polls, c(fit_args, list(backend = "cmdstanr",     iter_sampling = 5, iter_warmup = 5, compile_args = list(compile_model_methods = TRUE))))

[MansMeg]

Note! It doesn't seem to be intermittent. It segfaults deterministically.

[jgabry]

Maybe the workaround in the link in @VisruthSK shared would help? Seems like it should be irrelevant but it was suggested that it could fix the issue. Or I guess work around the issue, not fix it.

[MansMeg]

Sure. I'm testing the following in CI now:

      - name: Install CmdStan
        run: |
          cat("cmdstanr:", as.character(utils::packageVersion("cmdstanr")), "\n")
          cmdstan_dir <- cmdstanr::install_cmdstan(cores = 2, overwrite = FALSE)
          cmdstanr::set_cmdstan_path(cmdstan_dir)
          cmdstanr::cmdstan_make_local(cpp_options = list(STAN_THREADS = TRUE))
          cmdstanr::rebuild_cmdstan()
          cat("CmdStan path:", cmdstanr::cmdstan_path(), "\n")
          cat("CmdStan version:", cmdstanr::cmdstan_version(), "\n")
          cat("make/local:\n")
          cat(cmdstanr::cmdstan_make_local(), sep = "\n")
        shell: Rscript {0}

We'll know soon.

[MansMeg]

It did indeed work. Now it's passing/working. So we know that the bug from 2024 is still lurking.

[jgabry]

Ok well I'm glad it's working now! But I'm not glad that I don't understand why this was happening ;)

[MansMeg]

Yes. I now have a way to isolate the segfault at least. So we should be able to test for it.