CmdStan checks for file suffix .csv but should be checking for suffix .R

[mitzimorris]

Summary:

The introduction of single-process multi-chain parallel threads introduced a significant bug into CmdStan because instead of checking for input files with suffix .R the code checks for input files with suffix .csv.

cmdstan/src/cmdstan/command.hpp

Line 129 in 1667128

} else if (file_ending == ".csv") {

We don't have input routines that can parse CSV files into a var_context object, only parsers for rdump and json.

Description:

reported on Discourse by Rob Goedman: https://discourse.mc-stan.org/t/file-ending-of-r-is-not-supported-by-cmdstan/26134

Reproducible Steps:

Run CmdStan sample method with num_chains greater than 1 and specify init file in Rdump format.

Additional Information:

• either change .csv to .R or omit check and just throw file into the rdump reader.
• add a unit test

Current Version:

v2.29.0

[mitzimorris]

Rdump files can have any suffix - FWIW

[WardBrian]

Introduced here: https://github.com/stan-dev/cmdstan/blame/2de737b4b5e64753cc1104b72334836c810ddca3/src/cmdstan/command.hpp

Before that, didn't check any suffix if it was not json

https://github.com/stan-dev/cmdstan/blame/59baf8bc8099f93ab87e866f42304564be00ff65/src/cmdstan/command.hpp

[mitzimorris]

reverting to old way will definitely work. checking to see if there's a way to catch the rdump reader error.

[mitzimorris]

so-called "Rdump" format really is an R file which consists of variable definitions for Stan program variables.

it's not clear whether or not the rdump reader would do if it found other kinds of statements.
the rdump reader is used for data inputs as well as parameter and metric inits. getting rid of this input format will require a lot of planning and buy-in from the R user base.

[WardBrian]

It appears we went nearly 10 months with the incorrect check in place before someone noticed and reported it, so I’m not sure if it’s being used all that much. I don’t think cmdstanr uses it over JSON, so it would most likely be people manually preparing files who would use it?

[rok-cesnovar]

I don’t think cmdstanr uses it over JSON

That is correct. Parsing rdump in CmdStan is a few times slower than JSON, so we never use rdump. Its definitely a niche thing at this point and the only users that faced this issue are those that used CmdStan directly with num_chains > 1. But as long as we havent deprecated the format, we have to support it.

[WardBrian]

@rok-cesnovar does rdump support complex numbers?

[mitzimorris]

definitely not. it's an ad-hoc parser for a very small subset of R statements.
https://github.com/stan-dev/stan/blob/develop/src/stan/io/dump.hpp

[WardBrian]

It seems like the answer is actually yes. It works the same as JSON where a complex value is just like a 2-long array:

data {
  complex z;
}

z <- c(1,2)

I'm sure this has all sorts of edge cases, though

[mitzimorris]

cool! and of course, given that encoding, it should work.