permissions problem can cause conda to delete itself and cripple Anaconda irrecoverably #1733

Closed
ijstokes opened this Issue Oct 23, 2015 · 17 comments

Comments

Projects
None yet
@ijstokes
Contributor

ijstokes commented Oct 23, 2015

In an attempt to do a conda update conda the permissions allowed a user to do the unlinking phase of the update, but then did not allow the same user to link in the updated conda. This resulted in conda disappearing, and an effectively crippled Anaconda deployment. This isn't the first time I've done this. This is tied in with the meta-issue of #1329 , though perhaps not entirely the same: my request (and really, expectation) is limited in scope to having conda be careful enough at least with itself that it won't self-annihilate.

[root@isr-anaconda-platform ~]# sudo -u wakari /opt/wakari/anaconda/bin/conda update conda
Fetching package metadata: ........
Solving package specifications: .
Package plan for installation in environment /opt/wakari/anaconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    openssl-1.0.2d             |                0         3.2 MB
    python-2.7.10              |                2        12.0 MB
    requests-2.8.1             |           py27_0         603 KB
    setuptools-18.4            |           py27_0         342 KB
    wheel-0.26.0               |           py27_1          76 KB
    conda-3.18.3               |           py27_0         175 KB
    ------------------------------------------------------------
                                           Total:        16.3 MB

The following packages will be UPDATED:

    conda:      3.17.0-py27_0 --> 3.18.3-py27_0
    openssl:    1.0.1k-1      --> 1.0.2d-0     
    python:     2.7.10-0      --> 2.7.10-2     
    requests:   2.7.0-py27_0  --> 2.8.1-py27_0 
    setuptools: 18.1-py27_0   --> 18.4-py27_0  
    wheel:      0.24.0-py27_0 --> 0.26.0-py27_1

Proceed ([y]/n)? y

Fetching packages ...
openssl-1.0.2d 100% |##############################################| Time: 0:00:00  15.21 MB/s
python-2.7.10- 100% |##############################################| Time: 0:00:00  16.47 MB/s
requests-2.8.1 100% |##############################################| Time: 0:00:00   7.16 MB/s
setuptools-18. 100% |##############################################| Time: 0:00:00   5.73 MB/s
wheel-0.26.0-p 100% |##############################################| Time: 0:00:00   3.11 MB/s
conda-3.18.3-p 100% |##############################################| Time: 0:00:00   4.66 MB/s
Extracting packages ...
[      COMPLETE      ]|#################################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|#################################################################| 100%
Linking packages ...
An unexpected error has occurred, please consider sending the                           |   0%
following traceback to the conda GitHub issue tracker at:

    https://github.com/conda/conda/issues

Include the output of the command 'conda info' in your report.


Traceback (most recent call last):
  File "/opt/wakari/anaconda/bin/conda", line 5, in <module>

  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/cli/main.py", line 194, in main
    args_func(args, p)
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/cli/main.py", line 201, in args_func
    args.func(args, p)
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/cli/main_update.py", line 38, in execute
    install.install(args, parser, 'update')
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/cli/install.py", line 417, in install
    plan.execute_actions(actions, index, verbose=not args.quiet)
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/plan.py", line 533, in execute_actions
    inst.execute_instructions(plan, index, verbose)
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/instructions.py", line 148, in execute_instructions
    cmd(state, arg)
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/instructions.py", line 91, in LINK_CMD
    link(state['prefix'], arg, index=state['index'])
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/instructions.py", line 87, in link
    install.link(pkgs_dir, prefix, dist, lt, index=index)
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/install.py", line 607, in link
    _link(src, dst, lt)
  File "/opt/wakari/anaconda/lib/python2.7/site-packages/conda/install.py", line 136, in _link
    shutil.copy2(src, dst)
  File "/opt/wakari/anaconda/lib/python2.7/shutil.py", line 130, in copy2
  File "/opt/wakari/anaconda/lib/python2.7/shutil.py", line 83, in copyfile
IOError: [Errno 13] Permission denied: '/opt/wakari/anaconda/include/openssl/opensslconf.h'
[root@isr-anaconda-platform ~]# chown -R wakari:wakari /opt/wakari/miniconda /opt/wakari/waka
ri-server /opt/wakari/anaconda/
[root@isr-anaconda-platform ~]# sudo -u wakari /opt/wakari/anaconda/bin/conda update conda
sudo: /opt/wakari/anaconda/bin/conda: command not found
@ilanschnell

This comment has been minimized.

Show comment
Hide comment
@ilanschnell

ilanschnell Oct 23, 2015

Member

Conda does not do anything special with user permissions. How would you suggest to solve this problem?

Member

ilanschnell commented Oct 23, 2015

Conda does not do anything special with user permissions. How would you suggest to solve this problem?

@ijstokes

This comment has been minimized.

Show comment
Hide comment
@ijstokes

ijstokes Oct 23, 2015

Contributor

I would suggest that we understand the kinds of file or directory permissions that allow unlinking of a package but that will block the subsequent linking that needs to happen right afterwards. And then we need to check for those conditions and not attempt a package update if those conditions occur. If this is really computationally burdensome, then special case conda and python (in fact, as I discovered, requests and a few others might also be required), and just check those. But destroying Anaconda is a really bad experience. With Anaconda being increasingly adopted by admins and installed as the root user on hosts then with some subsequent chown and chmod operations, other people will also setup the scenario (as we have with Wakari) where they can more than shoot themselves in the foot, but, in fact, shoot themselves in the head.

Contributor

ijstokes commented Oct 23, 2015

I would suggest that we understand the kinds of file or directory permissions that allow unlinking of a package but that will block the subsequent linking that needs to happen right afterwards. And then we need to check for those conditions and not attempt a package update if those conditions occur. If this is really computationally burdensome, then special case conda and python (in fact, as I discovered, requests and a few others might also be required), and just check those. But destroying Anaconda is a really bad experience. With Anaconda being increasingly adopted by admins and installed as the root user on hosts then with some subsequent chown and chmod operations, other people will also setup the scenario (as we have with Wakari) where they can more than shoot themselves in the foot, but, in fact, shoot themselves in the head.

@ijstokes

This comment has been minimized.

Show comment
Hide comment
@ijstokes

ijstokes Oct 23, 2015

Contributor

OK, in case this ever comes up for anyone else, here is how I seem to have come close to recovering the system:

pushd /opt/wakari/anaconda
mv anaconda anaconda_orig
wget Anaconda.sh
chmod a+x Anaconda.sh
./Anaconda.sh # install to /opt/wakari/anaconda
rsync -a anaconda_orig/ anaconda/
chmod -R a+rX anaconda
chown wakari:wakari anaconda
sudo -u wakari /opt/wakari/anaconda/bin/conda update conda
sudo -u wakari /opt/wakari/anaconda/bin/conda install -f conda-env
popd

(the last command is due to a now-resolved issue where the conda update may actually remove conda-env, so this will restore it)

And at some later point, you'd probably want to do:

rm -Rf /opt/wakari/anaconda_orig
Contributor

ijstokes commented Oct 23, 2015

OK, in case this ever comes up for anyone else, here is how I seem to have come close to recovering the system:

pushd /opt/wakari/anaconda
mv anaconda anaconda_orig
wget Anaconda.sh
chmod a+x Anaconda.sh
./Anaconda.sh # install to /opt/wakari/anaconda
rsync -a anaconda_orig/ anaconda/
chmod -R a+rX anaconda
chown wakari:wakari anaconda
sudo -u wakari /opt/wakari/anaconda/bin/conda update conda
sudo -u wakari /opt/wakari/anaconda/bin/conda install -f conda-env
popd

(the last command is due to a now-resolved issue where the conda update may actually remove conda-env, so this will restore it)

And at some later point, you'd probably want to do:

rm -Rf /opt/wakari/anaconda_orig
@damianavila

This comment has been minimized.

Show comment
Hide comment
@damianavila

damianavila Oct 26, 2015

I could recover from this somehow using the conda revision mechanism... I would try to set up a minimal reproducible case and test my fix to post it here (as time permits).

I could recover from this somehow using the conda revision mechanism... I would try to set up a minimal reproducible case and test my fix to post it here (as time permits).

@0u812

This comment has been minimized.

Show comment
Hide comment
@0u812

0u812 Feb 17, 2016

Just wanted to say that this caused a user of an Anaconda package I created to accidentally wipe out their conda installation. I completely agree with @ijstokes point that conda shouldn't attempt an update unless the update can be completed. Updates should be atomic anyway, i.e. there should either be a check that ensures that the update can complete or it should be possible to roll back the changes if the update fails. 👍 for fixing this, as it is currently blocking my team from deploying to conda due to the potential for harming our users' Anaconda installation.

0u812 commented Feb 17, 2016

Just wanted to say that this caused a user of an Anaconda package I created to accidentally wipe out their conda installation. I completely agree with @ijstokes point that conda shouldn't attempt an update unless the update can be completed. Updates should be atomic anyway, i.e. there should either be a check that ensures that the update can complete or it should be possible to roll back the changes if the update fails. 👍 for fixing this, as it is currently blocking my team from deploying to conda due to the potential for harming our users' Anaconda installation.

@ilanschnell

This comment has been minimized.

Show comment
Hide comment
@ilanschnell

ilanschnell Feb 17, 2016

Member

The latest Miniconda installers have now a working -f option, which allows to force reinstall Miniconda into an existing prefix. This way one can recover.

Member

ilanschnell commented Feb 17, 2016

The latest Miniconda installers have now a working -f option, which allows to force reinstall Miniconda into an existing prefix. This way one can recover.

@0u812

This comment has been minimized.

Show comment
Hide comment
@0u812

0u812 Feb 17, 2016

Thanks, will pass the info along.

0u812 commented Feb 17, 2016

Thanks, will pass the info along.

@cmiller8

This comment has been minimized.

Show comment
Hide comment
@cmiller8

cmiller8 May 16, 2016

This has happened to me 3 times in the past 12 months -- each time I can't figure out how to recover and just end up reinstalling anaconda

This has happened to me 3 times in the past 12 months -- each time I can't figure out how to recover and just end up reinstalling anaconda

@egentry egentry referenced this issue in conda-forge/conda-forge.github.io Jun 28, 2016

Closed

`conda` executable deleted by `conda update --all` #180

@kalefranz kalefranz added this to the 4.3 milestone Jun 28, 2016

@dorzel dorzel self-assigned this Aug 9, 2016

@dorzel dorzel referenced this issue Aug 16, 2016

Closed

Check link 2 #3301

@robmarkcole

This comment has been minimized.

Show comment
Hide comment
@robmarkcole

robmarkcole Oct 15, 2016

I just experienced this, attempting a fresh install

I just experienced this, attempting a fresh install

@kalefranz

This comment has been minimized.

Show comment
Hide comment
@kalefranz

kalefranz Oct 15, 2016

Member

Progress towards a solution: #3571

Member

kalefranz commented Oct 15, 2016

Progress towards a solution: #3571

@sachinruk

This comment has been minimized.

Show comment
Hide comment
@sachinruk

sachinruk Nov 3, 2016

Has this been resolved? I did sudo chown -R sachin anaconda attempting to fix a PermissionError and all of a sudden conda is no longer in anaconda/bin. I think in my case the damage is done and I'll have to do a fresh install of anaconda.

Has this been resolved? I did sudo chown -R sachin anaconda attempting to fix a PermissionError and all of a sudden conda is no longer in anaconda/bin. I think in my case the damage is done and I'll have to do a fresh install of anaconda.

@robmarkcole

This comment has been minimized.

Show comment
Hide comment
@robmarkcole

robmarkcole Nov 3, 2016

Yes I had to do that 😂

On Thursday, November 3, 2016, Sachin Abeywardana notifications@github.com
wrote:

Has this been resolved? I did sudo chown -R sachin anaconda attempting to
fix a PermissionError and all of a sudden conda is no longer in
anaconda/bin. I think in my case the damage is done and I'll have to do a
fresh install of anaconda.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1733 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALTl2sEdgkk13DPDHpEZXVmNAXujKRUZks5q6Yc0gaJpZM4GUpzJ
.


Dr Robin Cole
Royal Surrey County Hospital
Guildford
Surrey
GU2 7XX

Tel : 07951810847

Yes I had to do that 😂

On Thursday, November 3, 2016, Sachin Abeywardana notifications@github.com
wrote:

Has this been resolved? I did sudo chown -R sachin anaconda attempting to
fix a PermissionError and all of a sudden conda is no longer in
anaconda/bin. I think in my case the damage is done and I'll have to do a
fresh install of anaconda.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1733 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ALTl2sEdgkk13DPDHpEZXVmNAXujKRUZks5q6Yc0gaJpZM4GUpzJ
.


Dr Robin Cole
Royal Surrey County Hospital
Guildford
Surrey
GU2 7XX

Tel : 07951810847

@kalefranz

This comment has been minimized.

Show comment
Hide comment
@kalefranz

kalefranz Jan 26, 2017

Member

fixed with conda 4.3

Member

kalefranz commented Jan 26, 2017

fixed with conda 4.3

@kalefranz kalefranz closed this Jan 26, 2017

@ksarma

This comment has been minimized.

Show comment
Hide comment
@ksarma

ksarma Feb 10, 2017

It just happened to me, with conda 4.3...

ksarma commented Feb 10, 2017

It just happened to me, with conda 4.3...

@stoneyv

This comment has been minimized.

Show comment
Hide comment
@stoneyv

stoneyv Jun 18, 2017

This happened to me with conda 4.3.21-py36_1 on ubuntu 16.04 LTS with a home directory install (not a site wide install) I rolled back to a revision that did not include an update to conda. The change that I rolled back from did include a change to conda. All of the envs bin directories use a link to the ~/anaconda3/bin/conda file. How does conda list --revisions compute the revisions? Does it use the json files in conda-meta?

fulano@laptop:~/Desktop$ conda install --revision 1
Fetching package metadata .........

Package plan for installation in environment /home/stoney/anaconda3:

The following packages will be REMOVED:

    branca:          0.2.0-py36_0       conda-forge
    click-plugins:   1.0.3-py36_0       conda-forge
    cligj:           0.4.0-py36_0       conda-forge
    conda:           4.3.21-py36_1      conda-forge
    conda-env:       2.6.0-0            conda-forge
    descartes:       1.1.0-py36_0       conda-forge
    fiona:           1.7.6-np111py36_0  conda-forge
    folium:          0.3.0-py36_0       conda-forge
    freexl:          1.0.2-1            conda-forge
    gdal:            2.1.3-np111py36_0  conda-forge
    geopandas:       0.2.1-py36_4       conda-forge
    geopy:           1.11.0-py36_0      conda-forge
    geos:            3.5.1-1            conda-forge
    giflib:          5.1.4-0            conda-forge
    hdf4:            4.2.12-0           conda-forge
... continued

stoneyv commented Jun 18, 2017

This happened to me with conda 4.3.21-py36_1 on ubuntu 16.04 LTS with a home directory install (not a site wide install) I rolled back to a revision that did not include an update to conda. The change that I rolled back from did include a change to conda. All of the envs bin directories use a link to the ~/anaconda3/bin/conda file. How does conda list --revisions compute the revisions? Does it use the json files in conda-meta?

fulano@laptop:~/Desktop$ conda install --revision 1
Fetching package metadata .........

Package plan for installation in environment /home/stoney/anaconda3:

The following packages will be REMOVED:

    branca:          0.2.0-py36_0       conda-forge
    click-plugins:   1.0.3-py36_0       conda-forge
    cligj:           0.4.0-py36_0       conda-forge
    conda:           4.3.21-py36_1      conda-forge
    conda-env:       2.6.0-0            conda-forge
    descartes:       1.1.0-py36_0       conda-forge
    fiona:           1.7.6-np111py36_0  conda-forge
    folium:          0.3.0-py36_0       conda-forge
    freexl:          1.0.2-1            conda-forge
    gdal:            2.1.3-np111py36_0  conda-forge
    geopandas:       0.2.1-py36_4       conda-forge
    geopy:           1.11.0-py36_0      conda-forge
    geos:            3.5.1-1            conda-forge
    giflib:          5.1.4-0            conda-forge
    hdf4:            4.2.12-0           conda-forge
... continued
@OSHI7

This comment has been minimized.

Show comment
Hide comment
@OSHI7

OSHI7 Jul 2, 2017

Hello. I am still having this problem.
For me, originally it seemed related to spyder launching, then I found either \anaconda3\pythonw.exe was missing (i.e. file was deleted from normal installation position). Then after performing some updates (I can post about this if relevant); then spyder behaved nicely; and was stable.
Now I struggle with a reminiscent problem with the Jupyter Notebooks.
After reading that evidently pyzmq was responsible, I followed the prescription by removing pyzmq, then 'conda install jupyter", proceeding to launch jupyter notebook. Victory!
It seemed...
When I then closed the server, I found \anaconda3\python.exe was missing again. Then I used a file recovery tool which showed python.exe was still located in the directory, but was deleted. I then tried to copy an archived copy of python.exe into the \anaconda3 folder but it would not let me, even after providing Admin rights. Seemed the file was still in memory or something, even though it was invisible.
After rebooting; before launching anything anaconda-related, I could rename the file to python.exe and all seemed well. Except it's not working as it should be. I seem to be able to repeat this loop ad infinitum.
I can post whatever you guys would like, if it is of interest...and (sorry) if you advise me how to capture relevant logs... I frankly don't know how to dump logs, as I am new at anaconda & python.
I appreciate your help and attention to details.
rgds, josh
edit: oh yes; I have a question. How can I do the miniconda in-situ upgrade with -f option? I read this could repair my install (without the reboot and manually patching files), but when I launch -f, it launches the GUI for the installer and will not let me point to the same directory where Anaconda is installed.

pps. The files that are (variously) deleted are \anaconda3\python.exe; \anaconda3\scripts\conda.exe; \anaconda3\jupyter-notebook.exe. The install can be repaired by recopying archived copies to replace the missing files...seemingly.

OSHI7 commented Jul 2, 2017

Hello. I am still having this problem.
For me, originally it seemed related to spyder launching, then I found either \anaconda3\pythonw.exe was missing (i.e. file was deleted from normal installation position). Then after performing some updates (I can post about this if relevant); then spyder behaved nicely; and was stable.
Now I struggle with a reminiscent problem with the Jupyter Notebooks.
After reading that evidently pyzmq was responsible, I followed the prescription by removing pyzmq, then 'conda install jupyter", proceeding to launch jupyter notebook. Victory!
It seemed...
When I then closed the server, I found \anaconda3\python.exe was missing again. Then I used a file recovery tool which showed python.exe was still located in the directory, but was deleted. I then tried to copy an archived copy of python.exe into the \anaconda3 folder but it would not let me, even after providing Admin rights. Seemed the file was still in memory or something, even though it was invisible.
After rebooting; before launching anything anaconda-related, I could rename the file to python.exe and all seemed well. Except it's not working as it should be. I seem to be able to repeat this loop ad infinitum.
I can post whatever you guys would like, if it is of interest...and (sorry) if you advise me how to capture relevant logs... I frankly don't know how to dump logs, as I am new at anaconda & python.
I appreciate your help and attention to details.
rgds, josh
edit: oh yes; I have a question. How can I do the miniconda in-situ upgrade with -f option? I read this could repair my install (without the reboot and manually patching files), but when I launch -f, it launches the GUI for the installer and will not let me point to the same directory where Anaconda is installed.

pps. The files that are (variously) deleted are \anaconda3\python.exe; \anaconda3\scripts\conda.exe; \anaconda3\jupyter-notebook.exe. The install can be repaired by recopying archived copies to replace the missing files...seemingly.

@aluo-x

This comment has been minimized.

Show comment
Hide comment
@aluo-x

aluo-x Jan 21, 2018

Just occured with me, using Windows 10, Anaconda 5.0.1 (Python 3 version). Tried
conda update --all and conda was gone

Upon further investigation, my issues actually seems to be #6482 and #8087

aluo-x commented Jan 21, 2018

Just occured with me, using Windows 10, Anaconda 5.0.1 (Python 3 version). Tried
conda update --all and conda was gone

Upon further investigation, my issues actually seems to be #6482 and #8087

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment