install_cmdstan fails

[Auerilas]

Summary:

Hi, I just downloaded cmdstanpy and am trying to install cmdstan but it fails.

Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cmdstanpy
>>> cmdstanpy.install_cmdstan()
Installing CmdStan version: 2.25.0
Install directory: /home/lemoinelab2/.cmdstan
Downloading CmdStan version 2.25.0
Download successful, file: /tmp/tmprdacja_9
Unpacked download as cmdstan-2.25.0
Building version cmdstan-2.25.0
Command "make build" failed
stan/lib/stan_math/make/libraries:115: *** "Need to set TBB_CXX_TYPE for non-standard compiler other than gcc or clang.".  Stop.
deleting tmpfiles dir: /tmp/tmpcnvpipxt
done

WARNING:cmdstanpy:CmdStan installation failed.
False

I don't know what this error means. I have two similar machines with the same set up, and cmdstan installed on one but not the other. I'm not sure where to start with this.

[mitzimorris]

@rok-cesnovar or @wds15 or @SteveBronder - CmdStan's make build fails on linux with GCC 7.5.0 - should it?

[Auerilas]

I'm not sure if this is a gcc issue? It happens if I try to make the build on my own:

(base) nate@red-leader:~$ which gcc
/usr/bin/gcc
(base) nate@red-leader:~$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(base) nate@red-leader:~$ cd .cmdstan/cmdstan-2.25.0/
(base) nate@red-leader:~/.cmdstan/cmdstan-2.25.0$ make build
/bin/sh: 1: /home/nate/miniconda3/bin/x86_64-conda_cos6-linux-gnu-c++: not found
stan/lib/stan_math/make/libraries:115: *** "Need to set TBB_CXX_TYPE for non-standard compiler other than gcc or clang.".  Stop.

I found a workaround here: https://discourse.mc-stan.org/t/error-installing-cmdstan/19216/4

But I'm still confused how this differs among systems. I've now tried it on three near-identical systems: two running Linux Mint 20 and one running Linux Mint 19. On all systems I followed the same work flow:

conda remove pystan
conda clean -tp
sudo apt install g++
pip install cmdstanpy
install_cmdstan

This worked on one system (one of the LM20), but not the other two. And it happens if I try to build manually as well. The only solution is to set make/local
CXX=g++
TBB_CXX_TYPE=gcc

Note that on both the LM20 systems, I get

gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.

but

Python 3.7.6 | packaged by conda-forge | (default, Jun  1 2020, 18:57:50) 
[GCC 7.5.0] on linux

Like conda is using a different GCC version than the system one. But I'm not sure that's the issue since it worked on one machine and not the other.

[SteveBronder]

Can you try building the version on develop? We had a patch for this for after the 2.25 release candidate. Maybe that didn't get into the 2.25.0 release somehow?

[Auerilas]

Well here's the weird part. cmdstan installed fine on my LM20 laptop, and gave me problems on the LM20 desktop and LM19 server.

I cleared pystan, updated conda, and updated all my packages on the LM20 desktop, gave a reboot, and it worked fine! No idea what changed.

I tried the same on the LM19 server, and still got the error. Error was resolved by
make/local
CXX=g++
TBB_CXX_TYPE=gcc

Since that appears to have worked, I'm going with this. No idea what little difference exists among the systems to cause this error.

[Auerilas]

I'm having another issue and I don't know if this is the appropriate place to ask, or if I should start a new topic. Once cmdstan is installed, everything works fine locally. However, I do all my work on remote shared folders (I have an NAS in my lab that hosts my data and scripts). I mount the directories locally on my machines, but I'm having issues:

If I mount the directory manually:
sudo mount -t nfs <host address> <local folder>
everything works just fine. However, on several of my machines (my office desktop and the lab computing server), the files are mounted automatically via modifying /etc/fstab:
<host address> <local folder> nfs defaults,user 0 0
when I do this, cmdstan cannot sample. I can compile models, and the model files show up in the remote server. But then when I try to sample them (via cmdstanpy), it crashes out and gives and error:
RuntimeError: Error during sampling. RunSet: chains=4 cmd: ['/home/nate/Documents/research-share/2020_herbiv_tolerance/scripts/A_gs_model', 'id=1', 'random', 'seed=56477', 'data', 'file=/tmp/tmpopgvot33/szdt_cu_.json', 'output', 'file=/tmp/tmpopgvot33/A_gs_model-202011111324-1-x8o7u1yx.csv', 'method=sample', 'num_samples=10000', 'num_warmup=10000', 'algorithm=hmc', 'adapt', 'engaged=1'] retcodes=[-1, -1, -1, -1] csv_files: /tmp/tmpopgvot33/A_gs_model-202011111324-1-x8o7u1yx.csv /tmp/tmpopgvot33/A_gs_model-202011111324-2-wnfxwafo.csv /tmp/tmpopgvot33/A_gs_model-202011111324-3-zhgg2vny.csv /tmp/tmpopgvot33/A_gs_model-202011111324-4-uds7vas7.csv
I was a little confused, but trying to optimize via MLE (not doing full MCMC, mod.optimize) gives me a little more clarity:

PermissionError                           Traceback (most recent call last)                                                                                                  
<ipython-input-3-9cd6bf9c0e1d> in <module>                                                                                                                                   
      1 for s in ['APLE4', 'SYFO', 'ANGE', 'BOCU']:                                                                                                                          
      2     for r in ['A', 'gs']:                                                                                                                                            
----> 3         analyze(merged, s, mod, r, response_dict, spp_dict)          
.....
~/miniconda3/lib/python3.7/subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
    798                                 c2pread, c2pwrite,
    799                                 errread, errwrite,
--> 800                                 restore_signals, start_new_session)
    801         except:
    802             # Cleanup if the child failed starting.

~/miniconda3/lib/python3.7/subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, start_new_session)
   1549                         if errno_num == errno.ENOENT:
   1550                             err_msg += ': ' + repr(err_filename)
-> 1551                     raise child_exception_type(errno_num, err_msg, err_filename)
   1552                 raise child_exception_type(err_msg)
   1553 

PermissionError: [Errno 13] Permission denied: '/home/nate/Documents/research-share/2020_herbiv_tolerance/scripts/A_gs_model'

I don't know why using sudo mount is different permissions than /etc/fstab, but I suspect that's the problem.

[mitzimorris]

I'm having another issue and I don't know if this is the appropriate place to ask, or if I should start a new topic. Once cmdstan is installed, everything works fine locally. However, I do all my work on remote shared folders (I have an NAS in my lab that hosts my data and scripts). I mount the directories locally on my machines, but I'm having issues

this is a different issue, but it's not clear whether it's a CmdStan or CmdStanPy issue.

have you tried to running CmdStan directly? assuming you've got the input data somewhere, the call to the sampler would be something like:

> /home/nate/Documents/research-share/2020_herbiv_tolerance/scripts/A_gs_model id=1 random seed=56477 \
data file=szdt_cu_.json output file=A_gs_model-1.csv method=sample num_samples=100 num_warmup=100

(note, yes, I cut down iters from 10000 to 100)

[Auerilas]

It's not a cmdstan issue, it's an fstab issue. I had to add the option 'exec' to all my fstab mounts to allow the executions of binaries. Sorry for this, I'm switching from pystan, and cmdstan[py] operate sufficiently differently to cause a bunch of issues to pop up. Should be all set now.

…

On Wed, Nov 11, 2020 at 1:53 PM Mitzi Morris ***@***.***> wrote: I'm having another issue and I don't know if this is the appropriate place to ask, or if I should start a new topic. Once cmdstan is installed, everything works fine locally. However, I do all my work on remote shared folders (I have an NAS in my lab that hosts my data and scripts). I mount the directories locally on my machines, but I'm having issues this is a different issue, but it's not clear whether it's a CmdStan or CmdStanPy issue. have you tried to running CmdStan directly? assuming you've got the input data somewhere, the call to the sampler would be something like: > /home/nate/Documents/research-share/2020_herbiv_tolerance/scripts/A_gs_model id=1 random seed=56477 \ data file=szdt_cu_.json output file=A_gs_model-1.csv method=sample num_samples=10000 num_warmup=10000 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#947 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHQESQ3GO4TGU2AKLHL4MTSPLTUTANCNFSM4TSHDHGA> .

[rok-cesnovar]

Great! Feel free to close this then!