[BUG] Torsten installation issue

[zhrandell]

Hello,

I'm keen to utilize the event schedule functionality of Torsten, and I've encountered an installation issue.

my information:
Windows 10;
R version 4.0.4;
rstan version 2.21.2;
cmdstan version 2.27.0;

cmdstan_path() 
[1] "D:/OneDrive/Documents/.cmdstanr/cmdstan-2.27.0"

check_cmdstan_toolchain()
The C++ toolchain required for CmdStan is setup properly!

I am able to compile models via cmdstan using code from this page.

I next cloned the Torsten repo, and set the cmdstan path to the version of cmdstan within the cloned Torsten repo (per the directions on your installation page):

cmdstanr::set_cmdstan_path("C:/Programs/Git/Torsten/cmdstan/")
CmdStan path set to: C:/Programs/Git/Torsten/cmdstan/

Next I tried to compile example models from the Torsten repo (via code from the Installation page):

library(cmdstanr)
library(posterior)
library(bayesplot)

file.dir <- file.path("C:/Programs/Git/Torsten", "example-models", "pk2cpt")
file  <- file.path(file.dir, "pk2cpt.stan")
model <- cmdstan_model(file)`` 

which produced the following error:

Compiling Stan program...
process_begin: CreateProcess(NULL, https://github.com/metrumresearchgroup/stanc3/releases/download/torsten_v0.89rc2/windows-stanc -o bin/stanc.exe --retry 5 --retry-delay 10, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make.exe: *** [make/stanc:70: bin/stanc.exe] Error 2
Error: An error occured during compilation! See the message above for more information.

After examining the error and after reading posts online, I realized Torsten's version of cmdstan did not contain a stanc.exe file . . . specifically, there was no stanc.exe file in Torsten/cmdstan/bin.

So, in a fit of improvisation, I copied the stanc.exe file from the "standard" cmdstan into Torsten's cmdstan
(which may be highly inappropriate -- I apologize -- I'm at the very edge of my competence here).

Running the above model then returned:

Compiling Stan program...
Semantic error in 'C:/Users/rande/AppData/Local/Temp/RtmpMhm7gw/model-7a101c7d73a3.stan', line 50, column 6 to column 71:
   -------------------------------------------------
    48:    theta[5] = ka;
    49:  
    50:    x = pmx_solve_twocpt(time, amt, rate, ii, evid, cmt, addl, ss, theta);
               ^
    51:  
    52:    cHat = x[2, :] ./ V1; // we're interested in the amount in the second compartment
   -------------------------------------------------
A returning function was expected but an undeclared identifier 'pmx_solve_twocpt' was supplied.mingw32-make.exe: *** [make/program:47: C:\Users\rande\AppData\Local\Temp\RtmpMhm7gw\model-7a101c7d73a3.hpp] Error 1
Error: An error occured during compilation! See the message above for more information.

I'm assuming my copying of the stanc.exe file somewhat patched one issue, but created another? Or is there an underlying issue with my Make? I'm assuming since cmdstan works properly, my g++ and Make toolchain are in place?

I very much appreciate any assistance here!!

[yizhang-yiz]

Can you go to Torsten/cmdstan and do

make build

and let me know the output? Thanks.

[zhrandell]

Here's what I entered into the cmd terminal (and the results). And note that I've removed the stanc.exe file that I'd previously pasted in.

C:\Programs\Git\Torsten\cmdstan> make build
'make' is not recognized as an internal or external command,
operable program or batch file.

[yizhang-yiz]

I've just pushed a fix, please git pull or just re-download and see if works now.

C:\Programs\Git\Torsten\cmdstan> make build
'make' is not recognized as an internal or external command,
operable program or batch file.

Perhaps you can try again using "mingw32-makeinstead ofmake`. For future reference the easiest way to debug this issue is through cmdstan + RTools. Check out https://mc-stan.org/docs/2_27/cmdstan-guide/cmdstan-installation.html#windows-make .

[zhrandell]

Thank you. I re-cloned your develop branch and tried again, but I encountered the same error.

It does seem there's something off with my make. I do have Rtools40 installed, which I verified with:

library(devtools)
> find_rtools()
[1] TRUE

I had previously also put both g++ and make on the PATH, per the cmdstan-installation guide you linked above. Here's a screenshot (see Rtools1 and Rtools2):

Furthermore, in R, I checked the PATH, which also confirms the presence of Rtools40 (I truncated the output here):

> Sys.getenv('PATH')
[1] "C:/Programs/Git/Torsten/cmdstan/stan/lib/stan_math/lib/tbb;C:\\Programs\\rtools40\\usr\\bin;C:\\Programs\\rtools40\\mingw64\\bin;C:\\Programs\\rtools40\\usr\\bin;C:\\Programs\\R-4.0.4\\bin\\x64; . . . 

mingw32-make in R:

> system('mingw32-make -v')
GNU Make 4.2.1
Built for x86_64-w64-mingw32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[1] 0

and g++ in R

> system('g++ -v')
Using built-in specs.
COLLECT_GCC=C:\Programs\rtools40\mingw64\bin\g++.exe
COLLECT_LTO_WRAPPER=C:/Programs/rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-8.3.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran --disable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Built by Jeroen for the R-project' --with-bugurl=https://github.com/r-windows --with-gnu-as --with-gnu-ld
Thread model: posix
gcc version 8.3.0 (Built by Jeroen for the R-project) 
[1] 0

And note that cmdstan by itself (i.e. not the Torsten version) compiles models properly, so I'm assuming that to a certain degree I have a functional make . . . but something is clearly still off and I'm uncertain how to proceed . . .

[yizhang-yiz]

what's the output of
mingw32-make path/to/model
using Torsten/cmdstan?

[zhrandell]

Okay, I'm not sure this is exactly what you're looking for, but I tried both make --version and mingw32-make --version in the cmd terminal (i.e. not in R) in both ~\Torsten\cmdstan, ~\Torsten\example-models\pck2cpt, and in ~\rtools40\usr\bin, where I know it should be.

C:\Programs\Git\Torsten\cmdstan>mingw32-make --version
'mingw32-make' is not recognized as an internal or external command,
operable program or batch file.

C:\Programs\Git\Torsten\example-models\pk2cpt>mingw32-make --version
'mingw32-make' is not recognized as an internal or external command,
operable program or batch file.

C:\Programs\rtools40\usr\bin>mingw32-make --version
'mingw32-make' is not recognized as an internal or external command,
operable program or batch file.

C:\Programs\rtools40\usr\bin>make --version
GNU Make 4.2.1
Built for x86_64-pc-msys
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[yizhang-yiz]

Sorry for being terse. What's the output of

cd C:\Programs\Git\Torsten\cmdstan
mingw32-make.exe ..\example-models\pk2cpt\pk2cpt.exe

?

[zhrandell]

ah okay, got it, thank you for the clarification!

once I cd in:

C:\Programs\Git\Torsten\cmdstan>mingw32-make.exe ..\example-models\pk2cpt\pk2cpt.exe
'mingw32-make.exe' is not recognized as an internal or external command,
operable program or batch file.

[yizhang-yiz]

This helps, thanks. Now please try

pacman -Sy mingw-w64-x86_64-make --noconfirm

to install mingw32-make. You'll need to ensure the newly installed mingw32-make is in your PATH.

[zhrandell]

hmm here's what happened in the Windows 10 cmd prompt:

C:\Programs\Git\Torsten\cmdstan>pacman -Sy mingw-w64-x86_64-make --noconfirm
'pacman' is not recognized as an internal or external command,
operable program or batch file.

I also tried in Cygwin64, but it too didn't like pacman.

Would mingw64 from this site work? If so, in addition to making sure it is in my PATH, should mingw64 be placed anywhere in particular (i.e., does it need to be within ..\Torsten\cmdstan, or is it enough that it's in my PATH?

Thank you for all your help!!

[yizhang-yiz]

Would mingw64 from this site work?

I don't know. But you can check the previous discussion here on how to setup mingw32-make: https://discourse.mc-stan.org/t/cmdstanr-with-rtools-4-0-on-windows-10-mingw32-make-exe-missing/14791

[zhrandell]

Thanks for that link, @yizhang-yiz.

I followed the directions, opened the rtools bash, and ran the following:

$ pacman -Syu mingw-w64-x86_64-make
:: Synchronizing package databases...
 mingw32 is up to date
 mingw64 is up to date
warning: mingw-w64-x86_64-make-4.2.1-2 is up to date -- reinstalling
:: Starting core system upgrade...
 there is nothing to do
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (1) mingw-w64-x86_64-make-4.2.1-2

Total Installed Size:  1.33 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                     [#####################] 100%
(1/1) checking package integrity                   [#####################] 100%
(1/1) loading package files                        [#####################] 100%
(1/1) checking for file conflicts                  [#####################] 100%
(1/1) checking available disk space                [#####################] 100%
:: Processing package changes...
(1/1) reinstalling mingw-w64-x86_64-make           [#####################] 100%

and again in the rtools bash, I ran:

$ make -v
GNU Make 4.2.1
Built for x86_64-pc-msys
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

and per the attached photo on my second post on this thread, I have the following on the PATH:

C:\Programs\rtools40\usr\bin
C:\Programs\rtools40\mingw64\bin

as stated in the instructions here and here.

the only difference I can see between my PATH and the instructions is that, for whatever reason, I don't have an Rtools folder, and instead only a rtools40 folder. But Rtools seems to work fine?

I apologize for continuing to put my difficulty here. I know we seem to have wandered from the Torsten install. I simply don't understand how I can successfully use cmdstan to compile models (outside of Torsten), and then upon attempting to compile Torsten models, my system acts as if it can't find make.

[yizhang-yiz]

Then you should have mingw32-make.exe sitting in C:\Programs\rtools40\mingw64\bin. Try

where mingw32-make.exe

and let me know the output. If you do find C:\Programs\rtools40\mingw64\bin\mingw32-make.exe, try to compile a Torsten model by

cd C:\Programs\Git\Torsten\cmdstan
C:\Programs\rtools40\mingw64\bin\mingw32-make.exe ..\example-models\pk2cpt\pk2cpt.exe

[zhrandell]

yup, as best as I can tell, mingw32-make.exe is sitting right in C:\Programs\rtools40\mingw64\bin. Screen shot of it below:

However (and once again in the rtools bash):

$ where mingw32-make.exe
INFO: Could not find files for the given pattern(s).

and

rande@DESKTOP-HT7JUT3 MSYS /c/Programs/Git/Torsten/cmdstan
$ C:/Programs/rtools40/mingw64/bin/mingw32-make.exe ../example-models/pk2cpt/pk2cpt.exe/usr/bin/sh: g++: command not found
INFO: Could not find files for the given pattern(s).
https://github.com/metrumresearchgroup/stanc3/releases/download/torsten_v0.89rc2/windows-stanc -o bin/stanc.exe --retry 5 --retry-delay 10
process_begin: CreateProcess(NULL, https://github.com/metrumresearchgroup/stanc3/releases/download/torsten_v0.89rc2/windows-stanc -o bin/stanc.exe --retry 5 --retry-delay 10, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make: *** [make/stanc:70: bin/stanc.exe] Error 2

BUT, once I navigated into the exact file where I can see that mingw32-make.exe is:

rande@DESKTOP-HT7JUT3 MSYS /mingw64/bin
$ where mingw32-make.exe
C:\Programs\rtools40\mingw64\bin\mingw32-make.exe

[yizhang-yiz]

rande@DESKTOP-HT7JUT3 MSYS /c/Programs/Git/Torsten/cmdstan
$ C:/Programs/rtools40/mingw64/bin/mingw32-make.exe ../example-models/pk2cpt/pk2cpt.exe/usr/bin/sh: g++: command not found
INFO: Could not find files for the given pattern(s).
https://github.com/metrumresearchgroup/stanc3/releases/download/torsten_v0.89rc2/windows-stanc -o bin/stanc.exe --retry 5 --retry-delay 10
process_begin: CreateProcess(NULL, https://github.com/metrumresearchgroup/stanc3/releases/download/torsten_v0.89rc2/windows-stanc -o bin/stanc.exe --retry 5 --retry-delay 10, ...) failed.
make (e=2): The system cannot find the file specified.
mingw32-make: *** [make/stanc:70: bin/stanc.exe] Error 2

Could you git pull at master branch? This link issue should've been resolved.

[zhrandell]

Yesterday I deleted the Torsten repo and re-cloned from the develop branch (as I saw that that where was you'd recently pushed).

I'm using GitHub Desktop (sorry, ecologist here trying to use developer tools . . . ), and on the Torsten Master branch just now I indeed was able to pull an update.

Re-running (again, in rtools bash) an attempt at compiling a Torsten model produced a slightly new flavor of error:

rande@DESKTOP-HT7JUT3 MSYS /c/Programs/Git/Torsten/cmdstan
$ C:/Programs/rtools40/mingw64/bin/mingw32-make.exe ../example-models/pk2cpt/pk2cpt.exe
/usr/bin/sh: g++: command not found
INFO: Could not find files for the given pattern(s).
curl -L https://github.com/metrumresearchgroup/stanc3/releases/download/torsten_v0.89rc2/windows-stanc -o bin/stanc.exe --retry 5 --retry-delay 10
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   621  100   621    0     0   2317      0 --:--:-- --:--:-- --:--:--  2317
100 20.5M  100 20.5M    0     0  17.1M      0  0:00:01  0:00:01 --:--:-- 33.7M
g++ -pipe   -m64 -D_REENTRANT  -O3 -I stan/lib/stan_math/lib/sundials_5.7.0/include -DNO_FPRINTF_OUTPUT   -D_USE_MATH_DEFINES  -O3  -c -x c -include stan/lib/stan_math/lib/sundials_5.7.0/include/stan_sundials_printf_override.hpp stan/lib/stan_math/lib/sundials_5.7.0/src/nvector/serial/nvector_serial.c -o stan/lib/stan_math/lib/sundials_5.7.0/src/nvector/serial/nvector_serial.o
g++: error: CreateProcess: No such file or directory
mingw32-make: *** [stan/lib/stan_math/make/libraries:64: stan/lib/stan_math/lib/sundials_5.7.0/src/nvector/serial/nvector_serial.o] Error 1

[yizhang-yiz]

This is odd. Could you do

C:/Programs/rtools40/mingw64/bin/mingw32-make.exe build

?

[zhrandell]

sure thing. it makes me feel a little better that this is an odd issue, and not something blatantly obvious!

first, from a new rtools bash:

rande@DESKTOP-HT7JUT3 MSYS ~
$ C:/Programs/rtools40/mingw64/bin/mingw32-make.exe build
mingw32-make: *** No rule to make target 'build'.  Stop.

and then once I cd into the exact file where mingw32-make.exe is:

rande@DESKTOP-HT7JUT3 MSYS /mingw64/bin
$ mingw32-make.exe build
-bash: mingw32-make.exe: command not found

[yizhang-yiz]

You need to do it at Torsten/cmdstan:

cd C:\Programs\Git\Torsten\cmdstan
C:\Programs\rtools40\mingw64\bin\mingw32-make.exe build

[zhrandell]

Ah okay, thank you. And since I updated Torsten, I re-checked the cmdstan_path() in R, and once more ran:

cmdstanr::set_cmdstan_path("C:/Programs/Git/Torsten/cmdstan/")
CmdStan path set to: C:/Programs/Git/Torsten/cmdstan/

and now per your most recent post:

rande@DESKTOP-HT7JUT3 MSYS /c/Programs/Git/Torsten/cmdstan
$ C:/Programs/rtools40/mingw64/bin/mingw32-make.exe build
/usr/bin/sh: g++: command not found
INFO: Could not find files for the given pattern(s).
g++ -std=c++1y -m64 -D_REENTRANT -Wall -Wno-unused-function -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-variable -Wno-sign-compare -Wno-unused-local-typedefs -Wno-int-in-bool-context -Wno-attributes -Wno-ignored-attributes      -I stan/lib/stan_math/lib/tbb_2020.3/include   -O3 -I src -I stan/src -I lib/rapidjson_1.1.0/ -I lib/CLI11-1.9.1/ -I stan/lib/stan_math/ -I stan/lib/stan_math/lib/eigen_3.3.9 -I stan/lib/stan_math/lib/boost_1.75.0 -I stan/lib/stan_math/lib/sundials_5.7.0/include -DBOOST_MPL_CFG_NO_PREPROCESSED_HEADERS -DBOOST_MPL_LIMIT_LIST_SIZE=30  -D_USE_MATH_DEFINES  -DBOOST_DISABLE_ASSERTS         -c -fvisibility=hidden -o bin/cmdstan/stansummary.o src/cmdstan/stansummary.cpp
g++: error: CreateProcess: No such file or directory
mingw32-make: *** [make/command:5: bin/cmdstan/stansummary.o] Error 1

[yizhang-yiz]

That means you don't have g++ in your PATH and why I find it odd as clearly your PATH contains C:/Programs/rtools40/mingw64/bin. You can try either restart windows (roll eyes) or add a new file C:\Programs\Git\Torsten\cmdstan\make\local that has a line:

CXX=C:/Programs/rtools40/mingw64/bin/g++.exe
CC=C:/Programs/rtools40/mingw64/bin/g++.exe

But first, please double check C:/Programs/rtools40/mingw64/bin/g++.exe exists.

[zhrandell]

Yes, I do believe g++. exe is right where it is supposed to be:

and just to confirm re: a new file (definitely preferable over restarting Windows!). I don't have a \local folder in C:\Programs\Git\Torsten\cmdstan\make\local (though I have local.example . . . see screenshot below), so should I make a local folder, or will local be the new file? I can add those two lines, and then what extension should I save the file as?

Thank you again for all of your help!!

[yizhang-yiz]

Do

cd C:\Programs\Git\Torsten\cmdstan\make
touch local

then use wordpad to open local and add the lines mentioned earlier.

[zhrandell]

ahhh okay. thank you! done.

I reran:

cd C:\Programs\Git\Torsten\cmdstan
C:\Programs\rtools40\mingw64\bin\mingw32-make.exe build

and also tried to compile a Torsten model, and got the same error from both:

> model <- cmdstan_model(file)
Compiling Stan program...
make/local:1: *** missing separator.  Stop.
Error: An error occured during compilation! See the message above for more information.

I'm not familiar with missing separators, but I'm starting to read online. I tried both:

CXX=C:/Programs/rtools40/mingw64/bin/g++.exe
     CC=C:/Programs/rtools40/mingw64/bin/g++.exe

and

CXX=C:/Programs/rtools40/mingw64/bin/g++.exe
<tab>CC=C:/Programs/rtools40/mingw64/bin/g++.exe

which both produced the same error as above (and I'll confess I don't actually know what I'm doing here).

[yizhang-yiz]

Just want to log the diagnostics done with @zhrandell offline before closing.

The issue of downloading of the latest stanc.exe has been fixed here.

The issue of building binaries was caused by Windows Path var missing rtools40/mingw64/bin as well as rtools40/usr/bin. Adding those path resolves the issue.