Cygwin: pip install fbprophet fails with Error 11 - Resource temporarily unavailable

[tommybutler]

I have pystan and all prerequisites installed on cygwin, windows 7 64 bit. When I attempt to install fbprophet via pip, I get the following error with no further details...

$ pip install fbprophet
Collecting fbprophet
  Downloading fbprophet-0.1.post1.tar.gz
Requirement already satisfied: pandas in /usr/lib/python3.4/site-packages (from fbprophet)
Requirement already satisfied: pystan in /usr/lib/python3.4/site-packages (from fbprophet)
Requirement already satisfied: pytz>=2011k in /usr/lib/python3.4/site-packages (from pandas->fbprophet)
Requirement already satisfied: python-dateutil>=2 in /usr/lib/python3.4/site-packages (from pandas->fbprophet)
Requirement already satisfied: numpy>=1.7.0 in /usr/lib/python3.4/site-packages (from pandas->fbprophet)
Requirement already satisfied: Cython!=0.25.1,>=0.22 in /usr/lib/python3.4/site-packages (from pystan->fbprophet)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.4/site-packages (from python-dateutil>=2->pandas->fbprophet)
Building wheels for collected packages: fbprophet
  Running setup.py bdist_wheel for fbprophet ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-6r49tg2p/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpr_88z480pip-wheel- --python-tag cp34:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/fbprophet
  creating build/lib/fbprophet/stan_models
  INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_35bf14a7f93814266f16b4cf48b40a5a NOW.
  INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_53588bb481d3f3b19450fc8e3b147683 NOW.
  error: [Errno 11] Resource temporarily unavailable

  ----------------------------------------
  Failed building wheel for fbprophet
  Running setup.py clean for fbprophet
Failed to build fbprophet
Installing collected packages: fbprophet
  Running setup.py install for fbprophet ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-6r49tg2p/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-7c5dep4c-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib
    creating build/lib/fbprophet
    creating build/lib/fbprophet/stan_models
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_35bf14a7f93814266f16b4cf48b40a5a NOW.
    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_53588bb481d3f3b19450fc8e3b147683 NOW.
    error: [Errno 11] Resource temporarily unavailable

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-6r49tg2p/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-7c5dep4c-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-6r49tg2p/fbprophet/

Is this because python3 can't fork properly on windows? I've already done a rebaseall and verified cygwin core components as part of the exercise leading up to this while getting pystan installed (no small feat). Am I doing something wrong?

[tommybutler]

Update - python3 doesn't seem to have any issues forking for me... I don't know what could be causing the error.

"forks child processes until you type 'q'"
import os
def child():
    print('Hello from child', os.getpid())
    os._exit(0) # else goes back to parent loop

def parent():
    while True:
        newpid = os.fork()
        if newpid == 0:
            child()
        else:
            print('Hello from parent', os.getpid(), newpid)
        if input() == 'q': break

parent()

...

$ python3 forktest.py
Hello from child 7032
Hello from parent 1980 7032
q

$ echo $?
0

[tommybutler]

Results are the same if I just try build from source git clone...

$ git clone 'https://github.com/facebookincubator/prophet.git'
$ cd prophet/python
$ python3 setup.py install
running install
running bdist_egg
running egg_info
creating fbprophet.egg-info
writing requirements to fbprophet.egg-info/requires.txt
writing top-level names to fbprophet.egg-info/top_level.txt
writing dependency_links to fbprophet.egg-info/dependency_links.txt
writing fbprophet.egg-info/PKG-INFO
writing manifest file 'fbprophet.egg-info/SOURCES.txt'
reading manifest file 'fbprophet.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'fbprophet.egg-info/SOURCES.txt'
installing library code to build/bdist.cygwin-2.7.0-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/fbprophet
creating build/lib/fbprophet/stan_models
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_7a788197ac3493030b72020b9ffdbe7d NOW.
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_5d2797effa1e1f26479fe813b143dee3 NOW.
error: [Errno 11] Resource temporarily unavailable

[tommybutler]

I ran the pip install command through strace. From what I can tell, it looks like pip tries and fails to delete its temporary pip build directory after doing a lot of compilation work, possibly because its open/locked elsewhere (given the error). I could be reading this wrong, but at least the details are here below for someone who may know better. Hope it helps. I sure would love to try out fbprophet!

$ strace /usr/bin/python3.4m.exe /usr/bin/pip install fbprophet
...
...
...
   40 -902346600 [main] python3.4m 8172 unlink_nt: Trying to delete \??\C:\cygwin64\tmp\pip-build-78yyieeg, isdir = 1
   88 -902346512 [main] python3.4m 8172 unlink_nt: Sharing violation when opening \??\C:\cygwin64\tmp\pip-build-78yyieeg
 6514 -902339998 [main] python3.4m 8172 try_to_bin: \??\C:\cygwin64\tmp\pip-build-78yyieeg, return bin_status 2
   48 -902339950 [main] python3.4m 8172 unlink_nt: \??\C:\cygwin64\tmp\pip-build-78yyieeg, return status = 0x0
   30 -902339920 [main] python3.4m 8172 rmdir: 0 = rmdir(/tmp/pip-build-78yyieeg)
  236 -902339684 [main] python3.4m 8172 close: close(7)
   25 -902339659 [main] python3.4m 8172 fhandler_base::close: closing '/tmp/pip-build-78yyieeg' handle 0x140
  156 -902339503 [main] python3.4m 8172 close: 0 = close(7)
 1099 -902338404 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
  184 -902338220 [main] python3.4m 8172 close: close(4)
  216 -902338004 [main] python3.4m 8172 fhandler_socket::close: 0 = fhandler_socket::close()
   37 -902337967 [main] python3.4m 8172 close: 0 = close(4)
  263 -902337704 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
  119 -902337585 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   39 -902337546 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   35 -902337511 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   60 -902337451 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   35 -902337416 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   60 -902337356 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   58 -902337298 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   59 -902337239 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
  250 -902336989 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
 9210 -902327779 [main] python3.4m 8172 close: close(5)
  239 -902327540 [main] python3.4m 8172 fhandler_socket::close: 0 = fhandler_socket::close()
   36 -902327504 [main] python3.4m 8172 close: 0 = close(5)
  829 -902326675 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
  104 -902326571 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   37 -902326534 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   36 -902326498 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   59 -902326439 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   34 -902326405 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   59 -902326346 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   59 -902326287 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
   58 -902326229 [main] python3.4m 8172 __set_errno: int semaphore::_trywait():3610 setting errno 11
 1365 -902324864 [main] python3.4m 8172 getpid: 8172 = getpid()
  299 -902324565 [main] python3.4m 8172 isatty: 1 = isatty(2)
  466 -902324099 [main] python3.4m 8172 fhandler_pty_slave::write: pty0, write(0x600117500, 427)
   27 -902324072 [main] python3.4m 8172 fhandler_pty_common::process_opost_output: (1902): pty output_mutex (0x144): waiting -1 ms
   25 -902324047 [main] python3.4m 8172 fhandler_pty_common::process_opost_output: (1902): pty output_mutex: acquired
   29 -902324018 [main] python3.4m 8172 fhandler_pty_common::process_opost_output: (1941): pty output_mutex(0x144) released
Command "/usr/bin/python3.4m -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-78yyieeg/fbprophet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" inst   27 -902323991 [main] python3.4m 8172 fhandler_pty_common::process_opost_output: (1902): pty output_mutex (0x144): waiting -1 ms
   25 -902323966 [main] python3.4m 8172 fhandler_pty_common::process_opost_output: (1902): pty output_mutex: acquired
all --record /tmp/pip-ucmu_zqn-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-78yyieeg/fbprophet/

[bletham]

I think this is the first report of Prophet in Windows via Cygwin.

In Windows Python not-via-cygwin, there is an ongoing issue that is preventing successful installation (#2) which seems to be an upstream issue in PyStan not supporting matrix multiplication in Windows (stan-dev/pystan#308). You do have a different error however, so it would be great if you could check that PyStan is working with matrix multiplication.

This code is a simple test that PyStan is working at all (sounds like it is):

import pystan
model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
model = pystan.StanModel(model_code=model_code)  # this will take a minute
y = model.sampling(n_jobs=1).extract()['y']
y.mean()  # should be close to 0

This code will test matrix multiplication, which is currently failing in Windows not-cygwin:

import pystan
model_code = """
data {
  int n;                                
  int m;                                
  matrix[n, m] A;
  vector[n] y;
}

parameters {
  vector[m] delta;
}

model {y ~ normal(A * delta,1);}
"""
model = pystan.StanModel(model_code=model_code)

[tommybutler]

@bletham thanks for responding! Your suspicions are confirmed. There is no issue with pystan when running your test code:

$ python3
Python 3.4.5 (default, Oct 10 2016, 14:41:48)
[GCC 5.4.0] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pystan
>>> model_code = 'parameters {real y;} model {y ~ normal(0,1);}'
>>> model = pystan.StanModel(model_code=model_code)  # this will take a minute
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_5944b02c79788fa0db5b3a93728ca2bf NOW.
>>> y = model.sampling(n_jobs=1).extract()['y']
Iteration:    1 / 2000 [  0%]  (Warmup) (Chain 0)
Iteration:  200 / 2000 [ 10%]  (Warmup) (Chain 0)
Iteration:  400 / 2000 [ 20%]  (Warmup) (Chain 0)
Iteration:  600 / 2000 [ 30%]  (Warmup) (Chain 0)
Iteration:  800 / 2000 [ 40%]  (Warmup) (Chain 0)
Iteration: 1000 / 2000 [ 50%]  (Warmup) (Chain 0)
Iteration: 1001 / 2000 [ 50%]  (Sampling) (Chain 0)
Iteration: 1200 / 2000 [ 60%]  (Sampling) (Chain 0)
Iteration: 1400 / 2000 [ 70%]  (Sampling) (Chain 0)
Iteration: 1600 / 2000 [ 80%]  (Sampling) (Chain 0)
Iteration: 1800 / 2000 [ 90%]  (Sampling) (Chain 0)
Iteration: 2000 / 2000 [100%]  (Sampling) (Chain 0)
#
#  Elapsed Time: 0.016 seconds (Warm-up)
#                0.031 seconds (Sampling)
#                0.047 seconds (Total)
#
Iteration:    1 / 2000 [  0%]  (Warmup) (Chain 1)
Iteration:  200 / 2000 [ 10%]  (Warmup) (Chain 1)
Iteration:  400 / 2000 [ 20%]  (Warmup) (Chain 1)
Iteration:  600 / 2000 [ 30%]  (Warmup) (Chain 1)
Iteration:  800 / 2000 [ 40%]  (Warmup) (Chain 1)
Iteration: 1000 / 2000 [ 50%]  (Warmup) (Chain 1)
Iteration: 1001 / 2000 [ 50%]  (Sampling) (Chain 1)
Iteration: 1200 / 2000 [ 60%]  (Sampling) (Chain 1)
Iteration: 1400 / 2000 [ 70%]  (Sampling) (Chain 1)
Iteration: 1600 / 2000 [ 80%]  (Sampling) (Chain 1)
Iteration: 1800 / 2000 [ 90%]  (Sampling) (Chain 1)
Iteration: 2000 / 2000 [100%]  (Sampling) (Chain 1)
#
#  Elapsed Time: 0.016 seconds (Warm-up)
#                0.015 seconds (Sampling)
#                0.031 seconds (Total)
#
Iteration:    1 / 2000 [  0%]  (Warmup) (Chain 2)
Iteration:  200 / 2000 [ 10%]  (Warmup) (Chain 2)
Iteration:  400 / 2000 [ 20%]  (Warmup) (Chain 2)
Iteration:  600 / 2000 [ 30%]  (Warmup) (Chain 2)
Iteration:  800 / 2000 [ 40%]  (Warmup) (Chain 2)
Iteration: 1000 / 2000 [ 50%]  (Warmup) (Chain 2)
Iteration: 1001 / 2000 [ 50%]  (Sampling) (Chain 2)
Iteration: 1200 / 2000 [ 60%]  (Sampling) (Chain 2)
Iteration: 1400 / 2000 [ 70%]  (Sampling) (Chain 2)
Iteration: 1600 / 2000 [ 80%]  (Sampling) (Chain 2)
Iteration: 1800 / 2000 [ 90%]  (Sampling) (Chain 2)
Iteration: 2000 / 2000 [100%]  (Sampling) (Chain 2)
#
#  Elapsed Time: 0.016 seconds (Warm-up)
#                0.015 seconds (Sampling)
#                0.031 seconds (Total)
#
Iteration:    1 / 2000 [  0%]  (Warmup) (Chain 3)
Iteration:  200 / 2000 [ 10%]  (Warmup) (Chain 3)
Iteration:  400 / 2000 [ 20%]  (Warmup) (Chain 3)
Iteration:  600 / 2000 [ 30%]  (Warmup) (Chain 3)
Iteration:  800 / 2000 [ 40%]  (Warmup) (Chain 3)
Iteration: 1000 / 2000 [ 50%]  (Warmup) (Chain 3)
Iteration: 1001 / 2000 [ 50%]  (Sampling) (Chain 3)
Iteration: 1200 / 2000 [ 60%]  (Sampling) (Chain 3)
Iteration: 1400 / 2000 [ 70%]  (Sampling) (Chain 3)
Iteration: 1600 / 2000 [ 80%]  (Sampling) (Chain 3)
Iteration: 1800 / 2000 [ 90%]  (Sampling) (Chain 3)
Iteration: 2000 / 2000 [100%]  (Sampling) (Chain 3)
#
#  Elapsed Time: 0.032 seconds (Warm-up)
#                0.015 seconds (Sampling)
#                0.047 seconds (Total)
#
>>> y.mean()  # should be close to 0
-0.018427703937476635
>>>

[bletham]

How about the second block with matrix multiplication?

[tommybutler]

@bletham I get no errors... was there supposed to be more to the code snippet?

$ python3
Python 3.4.5 (default, Oct 10 2016, 14:41:48)
[GCC 5.4.0] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pystan
>>> model_code = """
... data {
...   int n;
...   int m;
...   matrix[n, m] A;
...   vector[n] y;
... }
...
... parameters {
...   vector[m] delta;
... }
...
... model {y ~ normal(A * delta,1);}
... """
>>> model = pystan.StanModel(model_code=model_code)
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_3b30fd09063e3f75d25ac608ea03998b NOW.
>>>

[tommybutler]

If these code snippets work, what do I need to do to get fbprophet installed?

I'm about to take a look at the installation scripts, see if I can find out where pip is trying to delete it's build directory, and modify that code. However I doubt I'll have the same success as the developers who wrote the actual installer suite.

Anybody out there? I have rstan working. I just need to get prophet to install.

[bletham]

PyStan is definitely working, and the issue here is not the same issue that prevented installation from working in Windows without Cygwin. My best guess is that there is something specific to Cygwin that is causing it to not work. Can you try installing directly from source, to take pip out of the equation?

[tommybutler]

I did, @bletham. I also dug into the source of the build scripts and into the source of pystan and found and fixed/worked around two separate issues in order to get prophet installed and working. I will follow up with an explanation as soon as I get the chance to document what I did. Both problems come down to pystan and it's dynamic pickling/unpickling its models both during install and post install. A more elegant solution probably needs to be implemented.

[tommybutler]

Alright. During build, setup.py uses a loop to go through two model types by name ("linear", "logistic"), and compile them with pystan. The first iteration of the loop works as expected, but at some point during the second iteration there is a failure. Some part of the code refers back to the model that pystan compiled in the first iteration. Therein lies the problem -- when calling on pystan to compile a model, it subsequently deletes the temporary directory it used to run the compilation commands, and the files that were produced go away. (see /usr/lib/python{whatever}/site-packages/pystan/model.py). At the time of writing this comment, the relevant code lies on line 319 and calls shutil.rmtree on the temporary directory where the model was compiled: https://github.com/stan-dev/pystan/blob/develop/pystan/model.py#L319

I got around this first problem by modifying the loop in setup.py so that it only built the first model. Then I ran python3 setup.py install and it worked. I subsequently modified setup.py so that it would only build the second model, and ran python3 setup.py install again. This ugly workaround got prophet to actually install with both models (yay!)

The second problem happened when I tried running prophet. Again, the culprit is pystan. Just a few lines above the rmtree() call, it writes out self.module_bytes to a temp file and then calls load_module() on the temp file. Whaaaaat? When it does this under cygwin, the module name ends up being a .dll file and you get a fatal exception when load_module() is called because the permissions of the temp file aren't right for Windows. That .dll file needs to be executable!

I solved the problem by modifying model.py to include an os.chmod() on the temp library file, (I set the perms equivalent to a chmod a+x in bash). This prevented the error from happening anymore, but there may be security implications to this, so I hesitate to advise that a+x be used if u+x is sufficient. Bottom line here is that if this dynamic creation and deletion of files does need to keep happening, it needs to be fixed so that the files have the necessary permissions to work right. I really don't want to be critical of anyone, but it feels like the dynamic loading of these compiled stan models could be handled better by some alternate means. I don't think it can be considered thread-safe as long as this remains the case.

[bletham]

Thanks for the thorough breakdown of the issue.

I'm a bit unsure of why it would have a problem compiling both models. They should be built entirely independently, I don't see where in the build script it would be referring to the linear model while building logistic (https://github.com/facebookincubator/prophet/blob/master/python/setup.py#L35-L40).

Regardless it sounds like in Cygwin PyStan is unable to load compiled models from a pickle. This is the recommended way to persist a PyStan model, so I'm not sure what workarounds there may be until there is an upstream fix.

[tommybutler]

I wouldn't say pystan is unstable on cygwin. It's working fine for me now that it's installed and proper file permissions are being set on unpickled models via my changes to model.py. It's quite stable. It's just not portable... Not without an upstream fix. Please consider this issue closed with prophet and I'll report the problem to the pystan devs and send a pull request. You've been very helpful. Thank you!

[bletham]

I'm going to close this for now and if the upstream issues are fixed we can revisit this.

[cnlohr]

I'm from almost two years later and this whole msys-python [Errno 11] Resource temporarily unavailable thing is still an issue. I found this thread in a google search. Has anyone found a a solution for it?

[tommybutler]

@cnlohr see my solution above - #92 (comment)

[cnlohr]

Oh I just saw this after I did this: https://stackoverflow.com/a/12006549/2926815 << that worked, too... and is different. This seems like a much, much simpler solution.

Now I am scratching my head how the one in #92 would have worked for my case, but shrug?