CLI installation fails with "Python.h: no such file or directory"

[iceback]

I'm trying to install open-cravat for our team with a shared resource. I do not have sudo privileges. My command line, as follows, attempts to place the product into our space.
pip3 install
--verbose
--install-option="--prefix=/group3/tools"
open-cravat

....
Running command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-w_xiajc1/pyyaml/setup.py'; \
f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');\
f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-tfeikwwc-record/install-record.txt \
--single-version-externally-managed --compile --prefix=/group3/tools
....
copying lib3/yaml/scanner.py -> build/lib.linux-x86_64-3.6/yaml                                                                           
running build_ext                                                                                                                         
creating build/temp.linux-x86_64-3.6                                                                                                      
checking if libyaml is compilable                                                                                                         
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong $
checking if libyaml is linkable                                                                                                           
gcc -pthread build/temp.linux-x86_64-3.6/check_libyaml.o -L/usr/lib64 -lyaml -o build/temp.linux-x86_64-3.6/check_libyaml                 
building '_yaml' extension                                                                                                                
creating build/temp.linux-x86_64-3.6/ext                                                                                                  
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong $
ext/_yaml.c:4:20: fatal error: Python.h: No such file or directory                                                                        
 #include "Python.h"                                                                                                                      
                    ^                                                                                                                     
compilation terminated.                                                                                                                   
error: command 'gcc' failed with exit status 1                                                                                            
Running setup.py install for pyyaml: finished with status 'error'

Adding -v to the last action ("Running command") I see
import 'setuptools.msvc' # <_frozen_importlib_external.SourceFileLoader object at 0x2ae8d794b5c0>
import 'setuptools' # <_frozen_importlib_external.SourceFileLoader object at 0x2ae8cac41a20>
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib64/python3.6/tokenize.py", line 452, in open buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-ydzqg6mr/pyyaml/setup.py'

I was wondering if there was a connect between "msvc" and not find "Python.h" in a case insensitive manor?

Any help very much appreciated.

[rkimoakbioinformatics]

Hi @iceback , the installation failed while installing pyyaml, due to the lack of Python development package. Which OS are you using? Python 3.6 is used?

[iceback]

Yes. Python 3.6. Working with sys admin if larger system

…

On Apr 15, 2020, at 12:57 PM, Rick Kim ***@***.***> wrote:  Hi @iceback , the installation failed while installing pyyaml, due to the lack of Python development package. Which OS are you using? Python 3.6 is used? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[kmoad]

Unless this is a already a way your team commonly installs python packages, you may be better off using python virtual environments. As in,

python3 -m venv /group3/tools/pythonvenv
source /group3/tools/pythonvenv/bin/activate
python3 -m pip install open-cravat

This will ensure that the oc tool is available on the command line, as long as the venv is active.

From some older python docs, the --install-option="--prefix=/group3/tools" option is changing the install write directory, while running python from the system path. Unless you have the relevant directories under /group3/tools added to your PATH, the oc command line tool will not be available. The same issue would still apply if you used the newer --target option for pip as well.

[iceback]

I’ll take that under advisement. Getting the whole group into venv not straight forward

…

On Apr 15, 2020, at 1:45 PM, Kyle Moad ***@***.***> wrote:  Unless this is a already a way your team commonly installs python packages, you may be better off using python virtual environments. As in, python3 -m venv /group3/tools/pythonvenv source /group3/tools/pythonvenv/bin/activate python3 -m pip install open-cravat This will ensure that the oc tool is available on the command line, as long as the venv is active. From some older python docs, the --install-option="--prefix=/group3/tools" option is changing the install write directory, while running python from the system path. Unless you have the relevant directories under /group3/toolsadded to your PATH, theoccommand line tool will not be available. The same issue would still apply if you used the newer--target` option for pip as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[iceback]

If I'm following correctly you removed linux option 2), leaving only the single-user option 1.
The system we're on now has the python-dev pkg so "pip3 install open-cravat" succeeds, including the installation of the rather old Yaml (3.*!?). Yippee

However, I do not see any "oc" (nor "cravat") executable so "oc module install-base" is off the table

In interactive python3 session, "import cravat" get this little scree, twice

  >>>import cravat
/uufs/chpc.utah.edu/common/home/camp-group1/tools/lib/python3.6/site-packages/cravat/constants.py:35: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full d\
etails.
  conf = yaml.load(f)

Is that expected?

[kmoad]

That message is expected with pyyaml>=5.1. It will be gone in the next version of OC, and the warning can be disabled by setting an environment variable as described here. PyYaml made some non-backwards compatible changes and was unstable for a few versions around 5.1, so we temporarily pinned to a stable version.

The oc executable is a script that pip automatically creates in the user's path. The script is

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'open-cravat==1.7.2','console_scripts','oc'
__requires__ = 'open-cravat==1.7.2'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('open-cravat==1.7.2', 'console_scripts', 'oc')()
    )

Putting this script somewhere in path should make oc work.

[kmoad]

Also, depending on how your machine is configured, you may want to put oc's modules directory (where reference data gets installed) somewhere other than the default, which is under python's package directory. This can be done with oc config md [path], before oc module install-base.

[iceback]

Thank you, tons and tons

…

On 4/15/20 4:02 PM, Kyle Moad wrote: Also, depending on how your machine is configured, you may want to put oc's modules directory (where reference data gets installed) somewhere other than the default, which is under python's package directory. This can be done with |oc config md [path]|, before |oc module install-base|. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACY7PFB5W36J2GSUTKGAKTRMYVFTANCNFSM4MIDTNBQ>.

[kmoad]

@iceback We've just release OpenCRAVAT 1.8.0, which won't throw pyyaml warnings. If you update, be sure to change the version number in the oc script above.

[iceback]

I'll see if I can recreate the magic
Thanks for the heads-up

[kmoad]

Oh, and also, please run oc module update. There are some new modules with 1.8.0 that should speed things up, and take up less disk space.

[iceback]

Hm - this is tricky:
To hide the yaml warnings I had redirected stderr in a wrapper script. After install of 1.8.0, ocwrapper module update listed four things to update. Then nothing. For a really long time - until I clued in. Not sure I agree with the use of both stdout and stderr in the same breath?
Then a slew of fun of our own making - a nasty permissions thing. Redo the install/update.Then a trial -expect nothing - run of oc module update:
oc module update
Newer versions of (cravat-converter, oldcravat-converter, vcf-converter) are available, but would break dependencies. You may use --strategy=force to force installation.
No module updates are needed

I think we're good, but a little shakey. (I don't use oc, I'll pester she-who-does to try it asap)

[kmoad]

Appreciate the detail here. Looks like the "Continue" prompt for oc module update goes to stderr, and was being redirected. Oddly, it appears to get redirected by 1> /dev/null and by 2> /dev/null. Gonna dig a bit to figure out what python's up to there.