OpenShift PIP install broken with Python-2.7 cartridge

[dneyirp]

Hi

I followed instructions from https://developers.openshift.com/languages/python/flask.html
to install a tutorial Flask app. A few days ago all worked but as of yesterday (05 May 2016) it is not working.

When I add "Flask==0.10.1" to "requirements.txt" file, commit and push with git I get the following error:
remote: Collecting Flask==0.10.1 (from -r /var/lib/openshift/57550b1c0c1e666d9400012d/app-root/runtime/repo/requirements.txt (line 1)) remote: The repository located at mirror1.ops.rhcloud.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host mirror1.ops.rhcloud.com'. remote: Could not find a version that satisfies the requirement Flask==0.10.1 (from -r /var/lib/openshift/57550b1c0c1e666d9400012d/app-root/runtime/repo/requirements.txt (line 1)) (from versions: ) remote: No matching distribution found for Flask==0.10.1 (from -r /var/lib/openshift/57550b1c0c1e666d9400012d/app-root/runtime/repo/requirements.txt (line 1))

As mentioned, this worked flawlessly a few days ago.

When I ssh into my app account and run:
pip install Flask --no-cache-dir
there are no problems installing Flask.

If I add Flask==0.10.1 to install_requires in file setup.py. The installation works correctly with no problems.

Can someone help with this problem?

[dneyirp]

Hi

Here is an update.

I have added this line to the top of the requirements.txt file:
--index-url https://pypi.python.org/simple/
which takes me a bit further in that I actually get the Flask package downloaded but the installation fails with this error:
OSError: [Errno 13] Permission denied: '/var/lib/openshift/57550b1c0c1e666d9400012d/.cache'

When I ssh into the app account and check with pip list Flask is still not installed.
I also do not see a .cache folder and neither can I create one.

So far only adding install dependencies to setup.py or ssh'ing and running pip without using the cache dir works.

[dneyirp]

Here is a fix I managed to find using OpenShift actions hooks.
The key web links that helped are listed in the code listing below.

Detailed instructions:

• In your local repo, change to the folder .openshift\action_hooks
• Add a file named build to your local repo folder called .openshift\action_hooks
• In the file build add these lines:

#!/bin/bash

# Written by Priyend Somaroo, 06 Jun 2016, Vardaan Enterpises, www.vardaan.com
#
# This will execute pip install but using no caching in order to fix broken 
# cache problems with python-2.7 cartridge as at 06 Jun 2016
#
#   Ref: http://stackoverflow.com/questions/29913677/openshift-app-with-flask-sqlalchemy-and-sqlite-problems-with-database-reverti
#   Ref: http://stackoverflow.com/questions/21691202/how-to-create-file-execute-mode-permissions-in-git-on-windows
#
# 
# *** Very important *** : this file must be marked executable using 'chmod +x'.
# In Windows this is a problem so we simply mark it in the git repo as executable as follows.
# - In Windows open command prompt and change to the folder with this build file
# - Then run 'git update-index --chmod=+x build'
# - Ten check the permissions aer 0755 using 'git ls-files --stage' .
#
# Normal commits and push's occur after that.

# This build hook gets executed at the end of the build cycle before delpoy

# Change to repo directory and run pip install with no caching
cd ${OPENSHIFT_REPO_DIR}
pip install -r requirements.txt --no-cache-dir

• Mark the build file executable for it to run correctly:
  • In Linux run: chmod +x build
  • For Windows you will need to run this instead: git update-index --chmod=+x build
• Finally complete your git commit and git push

You can confirm the correct packages got installed by ssh'ing into your app folder and running:
pip list

Hopefully this helps someone if they experience the same problems.

Best regards

[nypgand1]

@dneyirp You save my day! Thank you.

[tiwillia]

Interesting, I was not able to reproduce this issue. Adding Flask==0.10.1 to my requirements.txt file and pushing the changes to an application results in a successful deployment.

Is anyone still hitting this issue? If so, are you pushing to openshift.com or running origin-server locally?

[nypgand1]

@tiwillia I can deploy the application actually, but can not import flask at runtime.

[dneyirp]

Hi. Sorry for the late reply. Been super busy.

@nypgand1 Thanks, glad it helped someone else out.

@tiwillia My app is hosted on openshift.redhat.com. All items in the requirements.txt worked perfectly until suddenly on 5 May 2016 is just did not work as mentioned in my post.

Some more info on my experience:
What was noticed is that an existing app that I created prior to 5 May 2016 worked properly, I could add items to requirements.txt and it will pull them through and install correctly but cloning or creating a new project had issues.

I have not checked if it is still broken, I just needed a fix and wrote the post to help others.

[UncleVasya]

Is anyone still hitting this issue?

I do. I have an app on openshift.com with python-2.7 cartridge and dependencies written in requirements.txt.

When I do rhc app-deploy master -a hktools I can see this in output:

...
Collecting dj-database-url==0.3.0 (from -r /var/lib/openshift/576a515089f5cf2f0c000110/app-root/runtime/repo/requirements.txt (line 1))
  The repository located at mirror1.ops.rhcloud.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host mirror1.ops.rhcloud.com'.
  Could not find a version that satisfies the requirement dj-database-url==0.3.0 (from -r /var/lib/openshift/576a515089f5cf2f0c000110/app-root/runtime/repo/requirements.txt (line 1)) (from versions: )
No matching distribution found for dj-database-url==0.3.0 (from -r /var/lib/openshift/576a515089f5cf2f0c000110/app-root/runtime/repo/requirements.txt (line 1))
...

Deploy proceeds and shown as successful:

Preparing build for deployment
Deployment id is a94a9546
Activating deployment
Starting Python 2.7 cartridge (Apache+mod_wsgi)
Application directory "/" selected as DocumentRoot
$OPENSHIFT_PYTHON_WSGI_APPLICATION ENV VAR detected
Application "hackerrank_tools/wsgi.py" selected as default WSGI entry point
Success

But app doesn't work since dependencies weren't installed.

---

Workaround posted above works.

[gdomod]

I got the same problems with pip install. Easy_install works for me from ssh console.

[mikeazo]

I was having the same problem today. The workaround listed above fixed the issue.

[dneyirp]

Just an update.

Tried a new Python-2.7 app yesterday, 24 Oct 2016.
Bug still exists and workaround fixes the problem.

Hopefuly someone at Openshift will be able to get someone to look into it, it seems to be a simple folder permission issue to solve the problem.

[dneyirp]

Hi

From your log a system folder is trying to get removed which is odd. Normally a virtualenv is where installations should occur.

Can you ssh in and run 'pip install Werkzeug' and see if that works?

The work around I posted is to get packages installed using requirements.txt, I can still ssh in and run pip commands and they work.

I can also add packages to setup.py and that works too, maybe you can try it that way and see if it helps.

My work around is basically running the pip installation at the end of the build which then ensures it is run inside the virtual environment.

Something is definitely off with Openshift's system, there does not seem to be a consistency in the way the virtual system works. Their

build system seems to be running the pip installations using requirements.txt before it is inside a virtual environment which then

triggers all sorts of permission problems.

---

From: flowstack notifications@github.com
Sent: Wednesday, October 19, 2016 9:41 PM
To: openshift/origin-server
Cc: dneyirp; Mention
Subject: Re: [openshift/origin-server] OpenShift PIP install broken with Python-2.7 cartridge (#6390)

I am still having this issue. The workaround listed (--no-cache-dir) isn't working for me, regardless of whether the command is run via the build action hook, or in the terminal after SSHing in.

The initial failure was with Werkzeug, which is attempting to write to a folder it doesn't have permission to. I removed it but then another package hit the same issue.

This is causing a problem on a new application I started today, but two other ones created months ago are fine.

Any suggestions?

Exception:
Traceback (most recent call last):
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip/commands/install.py", line 308, in run
strip_file_prefix=options.strip_file_prefix,
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 726, in uninstall
paths_to_remove.remove(auto_confirm)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/pip/utils/init.py", line 314, in renames
shutil.move(old, new)
File "/opt/rh/python27/root/usr/lib64/python2.7/shutil.py", line 300, in move
rmtree(src)
File "/opt/rh/python27/root/usr/lib64/python2.7/shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/opt/rh/python27/root/usr/lib64/python2.7/shutil.py", line 250, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission denied: '/opt/rh/python27/root/usr/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg-info/dependency_links.txt'

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/6390#issuecomment-254949434, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AM078Fz_jZMc-YrOKss26Q6SBcMjbf1Xks5q1o6mgaJpZM4IuoHA.

[phuein]

I have the same issue. Manually ssh-ing gives the same error as detailed in git push-commit.

remote: OSError: [Errno 13] Permission denied: '/var/lib/openshift/580e8b367628e19fd1000012/.cache'

[tiwillia]

This issue seems to contain two actual issues. The first report is around the inability to use the default trusted mirror for pip (mirror1.ops.rhcloud.com). I can't reproduce this issue at all. It most likely came about during an outage or issue with the mirror. Is anyone still seeing this error when deploying changes to their python application?

remote: Collecting Flask==0.10.1 (from -r /var/lib/openshift/57550b1c0c1e666d9400012d/app-root/runtime/repo/requirements.txt (line 1)) remote: The repository located at mirror1.ops.rhcloud.com is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host mirror1.ops.rhcloud.com'. remote: Could not find a version that satisfies the requirement Flask==0.10.1 (from -r /var/lib/openshift/57550b1c0c1e666d9400012d/app-root/runtime/repo/requirements.txt (line 1)) (from versions: ) remote: No matching distribution found for Flask==0.10.1 (from -r /var/lib/openshift/57550b1c0c1e666d9400012d/app-root/runtime/repo/requirements.txt (line 1))

As for the other issue reported here, a permission denied issue on the $OPENSHIFT_HOMEDIR/.cache directory, this issue should be resolved by the following commit: 53619d5

Basically, the $OPENSHIFT_HOMEDIR/.cache directory should have been created by the system during the gear's initial creation. When created during the install phase, it should have the correct permissions to allow pip to write to it.

The fix must not have made its way into openshift.com's infrastructure yet, so I'll leave this issue open until it does. For what its worth, I can currently reproduce this issue in openshift.com.

[ludusrusso]

I got the same issue and try to solve it using hooks, but now a got an other permission issue when I try to install something

remote: OSError: [Errno 13] Permission denied: '/opt/rh/python27/root/usr/lib64/python2.7/site-packages/SQLAlchemy-0.7.9-py2.7.egg-info/dependency_links.txt'

[raditv]

I've got the same issue for installing dependencies from requirement.txt, until today I still have to install it via ssh or setup.py

[dneyirp]

@radtiv - Solution using hooks above will then solve your problem if you prefer to use the "requirements.txt" file instead

[vietvudanh]

Hi, I thought this was fixed in #6430?

I have double check my wsgi.py, it seems ok:

virtenv = os.environ['OPENSHIFT_PYTHON_DIR'] + '/virtenv/'
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
    execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
    pass

I still encounter the error today.

remote: Stopping Python 2.7 cartridge        
remote: Waiting for stop to finish        
remote: Waiting for stop to finish        
remote: Building git ref 'master', commit d1a6bcb        
remote: Activating virtenv        
remote: Checking for pip dependency listed in requirements.txt file..        
remote: The directory '/var/lib/openshift/58a9a44e2d5271ccd2000060/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.        
remote: You are using pip version 7.1.0, however version 9.0.1 is available.        
remote: You should consider upgrading via the 'pip install --upgrade pip' command.        
remote: The directory '/var/lib/openshift/58a9a44e2d5271ccd2000060/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.        
remote: Collecting Flask==0.10.1 (from -r /var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/repo/requirements.txt (line 1))        
remote:   Downloading https://mirror.openshift.com/mirror/python/web/packages/db/9c/149ba60c47d107f85fe52564133348458f093dd5e6b57a5b60ab9ac517bb/Flask-0.10.1.tar.gz (544kB)        
remote: Collecting requests (from -r /var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/repo/requirements.txt (line 2))        
remote:   Downloading https://mirror.openshift.com/mirror/python/web/packages/7e/ac/a80ed043485a3764053f59ca92f809cc8a18344692817152b0e8bd3ca891/requests-2.13.0-py2.py3-none-any.whl (584kB)        
remote: Collecting beautifulsoup4 (from -r /var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/repo/requirements.txt (line 3))        
remote:   Downloading https://mirror.openshift.com/mirror/python/web/packages/cf/9b/8a6891f8c349431ee94669ce592fc375a19d4e3f4273a01a47207c7990b2/beautifulsoup4-4.5.3-py2-none-any.whl (85kB)        
remote: Requirement already satisfied (use --upgrade to upgrade): Werkzeug>=0.7 in /opt/rh/python27/root/usr/lib/python2.7/site-packages (from Flask==0.10.1->-r /var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/repo/requirements.txt (line 1))        
remote: Requirement already satisfied (use --upgrade to upgrade): Jinja2>=2.4 in /opt/rh/python27/root/usr/lib/python2.7/site-packages (from Flask==0.10.1->-r /var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/repo/requirements.txt (line 1))        
remote: Collecting itsdangerous>=0.21 (from Flask==0.10.1->-r /var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/repo/requirements.txt (line 1))        
remote:   Downloading https://mirror.openshift.com/mirror/python/web/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)        
remote: Building wheels for collected packages: Flask, itsdangerous        
remote: Exception:        
remote: Traceback (most recent call last):        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main        
remote:     status = self.run(options, args)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/pip/commands/install.py", line 293, in run        
remote:     wb.build(autobuilding=True)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/pip/wheel.py", line 756, in build        
remote:     ensure_dir(output_dir)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/pip/utils/__init__.py", line 70, in ensure_dir        
remote:     os.makedirs(path)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/python/virtenv/lib64/python2.7/os.py", line 150, in makedirs        
remote:     makedirs(head, mode)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/python/virtenv/lib64/python2.7/os.py", line 150, in makedirs        
remote:     makedirs(head, mode)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/python/virtenv/lib64/python2.7/os.py", line 150, in makedirs        
remote:     makedirs(head, mode)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/python/virtenv/lib64/python2.7/os.py", line 150, in makedirs        
remote:     makedirs(head, mode)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/python/virtenv/lib64/python2.7/os.py", line 150, in makedirs        
remote:     makedirs(head, mode)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/python/virtenv/lib64/python2.7/os.py", line 150, in makedirs        
remote:     makedirs(head, mode)        
remote:   File "/var/lib/openshift/58a9a44e2d5271ccd2000060/python/virtenv/lib64/python2.7/os.py", line 157, in makedirs        
remote:     mkdir(name, mode)        
remote: OSError: [Errno 13] Permission denied: '/var/lib/openshift/58a9a44e2d5271ccd2000060/.cache'

[NickAltmann]

Still seeing this problem as of today. Fixed using the instructions from @dneyirp, thanks very much.

[NickAltmann]

Actually I still have the problem and can't find a way round it. I get the permission problem regardless of whether I use the suggested hook workaround, use setup.py or run pip directly in a terminal session. I've wasted enough time on it and will probably just give up on Openshift for what I'm trying to do.

[dneyirp]

Hi Nick It is a pity the problem still exists. I did notice that sometimes deleting a project and starting a new one works with the work around. Maybe give that a shot. I also remember reading on Open shifts website that the old system use cartridges is problematic and they have abandoned It in their new product offering which is Openshift 3 (I think). I haven’t tried it but maybe it will work for you as it uses Docker and I have had better luck with Docker based apps. Best regards Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: Nick Altmann<mailto:notifications@github.com> Sent: Friday, 12 May 2017 9:23 PM To: openshift/origin-server<mailto:origin-server@noreply.github.com> Cc: dneyirp<mailto:priyend@hotmail.com>; Mention<mailto:mention@noreply.github.com> Subject: Re: [openshift/origin-server] OpenShift PIP install broken with Python-2.7 cartridge (#6390) Actually I still have the problem and can't find a way round it. I get the permission problem regardless of whether I use the suggested hook workaround, use setup.py or run pip directly in a terminal session. I've wasted enough time on it and will probably just give up on Openshift for what I'm trying to do. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#6390 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AM078KLGnuhqCI4wr0iYLU6XCzT9boqxks5r5LGxgaJpZM4IuoHA>.

[vietvudanh]

Sadly, that’s also the solution for me. I have tried other service since I still got this problem after all workarounds and solutions mentioned here. --- Viet Vu vietvudanh@gmail.com <mailto:vietvudanh@gmail.com> (+84) 90 6058 062

…

On May 13, 2017, at 2:23 AM, Nick Altmann ***@***.***> wrote: Actually I still have the problem and can't find a way round it. I get the permission problem regardless of whether I use the suggested hook workaround, use setup.py or run pip directly in a terminal session. I've wasted enough time on it and will probably just give up on Openshift for what I'm trying to do. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6390 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD027Jxjf1xxkDahLmIAznRj5J6ZWA1Mks5r5LGxgaJpZM4IuoHA>.

[NickAltmann]

Hi dneyirp, vietvudanh. I actually tried moving over to Openshift 3, but I then got an entirely different problem installing pandas that looks like this one.

I'm now hosting with another service.

[dneyirp]

Hi Guys I have a feeling that in creating the various projects some script is not creating the accounts correctly because permissions are not right. I have not looked at the Openshift code in detail enough to know what is causing it. For me the very first app I tested with only worked and all apps thereafter needed my workaround. It is a pity that these issues exist, I hope Redhat is looking at them! Good luck Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: Nick Altmann<mailto:notifications@github.com> Sent: Tuesday, 16 May 2017 9:30 PM To: openshift/origin-server<mailto:origin-server@noreply.github.com> Cc: dneyirp<mailto:priyend@hotmail.com>; Mention<mailto:mention@noreply.github.com> Subject: Re: [openshift/origin-server] OpenShift PIP install broken with Python-2.7 cartridge (#6390) Hi dneyirp, vietvudanh. I actually tried moving over to Openshift 3, but I then got an entirely different problem installing pandas that looks like this one<http://stackoverflow.com/questions/30814931/cant-get-pandas-to-install-with-openshift>. I'm now hosting with another service. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#6390 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AM078OFXRqRqQ8N2orcbiRuyZ4Qvz3P9ks5r6fkwgaJpZM4IuoHA>.

[dobbymoodge]

@dneyirp Can you provide a bit more detail? Are you talking about the bug this issue is about, or some different behavior?

[dneyirp]

Hi John For me it is still related to the same bug but the bug is bigger than just the Python-2.7 cartridge. When I originally signed up, only my very first project ever had the correct permissions assigned and I did not have issues with the project. Even when I logged in via ssh into my project account I could run various shell commands without permission errors. After that not a single project I have created was without directory permission issues. I eventually came up with the work around that suited me but as of lately I noticed that it does not always work and this has been confirmed by at least one other user. What concerns me is that now the behaviour has changed and it seemss even more problematic than before. Permissions in folders seem to be set incorrectly before the complete infrastructure of the project is set up which results in broken projects. That seems to look like a race issue where multiple threads are spawned and each set up their section but on completion they might be setting permissions prematurely. This is pure speculation on my part I have not looked at the code nor confirmed this, but experience I have gained seems that this might be a possible explanation. I will try a new simple Flask project over the next few days and try to replicate my experience. Hopefully it is an easy to replicate issue to help make bug fixing easier. But we all know, it is never that easy! Best regards Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: John Lamb<mailto:notifications@github.com> Sent: Friday, 19 May 2017 9:35 PM To: openshift/origin-server<mailto:origin-server@noreply.github.com> Cc: dneyirp<mailto:priyend@hotmail.com>; Mention<mailto:mention@noreply.github.com> Subject: Re: [openshift/origin-server] OpenShift PIP install broken with Python-2.7 cartridge (#6390) @dneyirp<https://github.com/dneyirp> Can you provide a bit more detail? Are you talking about the bug this issue is about, or some different behavior? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#6390 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AM078JbTJO16sUQctZyvkor4cYyfms7rks5r7e7kgaJpZM4IuoHA>.