Avoid the Windows file lock by moving files to a trash directory #1133

asmeurer · 2015-02-06T00:21:53Z

This lets you update Python, pycosat, and psutil in the root environment.

Conda moves all files that it can't delete to a trash directory. Conda attempts to clear the trash directory every time it links a package. This typically will not actually work until the next time conda is run.

Preliminary tests indicate this is working, but obviously it needs a lot more testing. If you do test, I highly recommend installing a new Miniconda, as the risk of borking your existing installation is still high.

Fixes #1001.

The trash mechanism should make this no longer necessary.

This avoids issues when the a file with the same name is already in the trash.

Also, don't raise exceptions in move_to_trash. Try the next envs dir, and if everything fails, we only log errors.

asmeurer · 2015-02-06T00:23:54Z

Known issue: a lot of "access is denied" errors are printed to the screen on installation.

asmeurer · 2015-02-06T00:34:09Z

TODO:

Test envs dirs stuff
Put the trash in the prefix, not the envs dir (negates the above)
See if rm_rf does partial deletions
Access is denied errors
Use tasklist /m somehow?
Test failures
Move this logic to rm_rf?

This reverts commit 287b4a0. subprocess.DEVNULL is only available in Python 3.

The output is printed to the log.

asmeurer · 2015-02-06T17:41:50Z

An error I got changing Python versions:


C:\Users\Aaron Meurer\Documents\GitHub\conda>"C:\Users\Aaron Meurer\Miniconda3
-test\Scripts\conda.exe" install python=3
Fetching package metadata: ......
Solving package specifications: .......
Package plan for installation in environment C:\Users\Aaron Meurer\Miniconda3-te
st:

The following packages will be UPDATED:

    conda:     3.8.4-py27_0 defaults --> 3.8.4-py34_0 defaults
    conda-env: 2.0.2-py27_0 defaults --> 2.0.2-py34_0 defaults
    pycosat:   0.6.1-py27_0 defaults --> 0.6.1-py34_0 defaults
    python:    2.7.9-1      defaults --> 3.4.2-1      defaults
    pyyaml:    3.11-py27_0  defaults --> 3.11-py34_0  defaults
    requests:  2.5.1-py27_0 defaults --> 2.5.1-py34_0 defaults

Unlinking packages ...
[      COMPLETE      ] |#################################################| 100%
Linking packages ...
An unexpected error has occurred, please consider sending the            |   0%
following traceback to the conda GitHub issue tracker at:

    https://github.com/conda/conda/issues

Include the output of the command 'conda info' in your report.


Traceback (most recent call last):
  File "C:\Users\Aaron Meurer\Miniconda3-test\Scripts\conda-script.py", line 4,
in <module>
  File "c:\users\aaron meurer\documents\github\conda\conda\cli\main.py", line 20
2, in main
    args_func(args, p)
  File "c:\users\aaron meurer\documents\github\conda\conda\cli\main.py", line 20
7, in args_func
    args.func(args, p)
  File "c:\users\aaron meurer\documents\github\conda\conda\cli\main_install.py",
 line 46, in execute
    install.install(args, parser, 'install')
  File "c:\users\aaron meurer\documents\github\conda\conda\cli\install.py", line
 412, in install
    plan.execute_actions(actions, index, verbose=not args.quiet)
  File "c:\users\aaron meurer\documents\github\conda\conda\plan.py", line 498, i
n execute_actions
    inst.execute_instructions(plan, index, verbose)
  File "c:\users\aaron meurer\documents\github\conda\conda\instructions.py", lin
e 134, in execute_instructions
    getLogger('progress.update').info((arg, state['i']))
  File "C:\Users\Aaron Meurer\Miniconda3-test\lib\logging\__init__.py", line 116
0, in info
  File "C:\Users\Aaron Meurer\Miniconda3-test\lib\logging\__init__.py", line 127
9, in _log
  File "C:\Users\Aaron Meurer\Miniconda3-test\lib\logging\__init__.py", line 128
9, in handle
  File "C:\Users\Aaron Meurer\Miniconda3-test\lib\logging\__init__.py", line 132
9, in callHandlers
  File "C:\Users\Aaron Meurer\Miniconda3-test\lib\logging\__init__.py", line 757
, in handle
  File "c:\users\aaron meurer\documents\github\conda\conda\console.py", line 47,
 in emit
    progress.update(n)
  File "c:\users\aaron meurer\documents\github\conda\conda\progressbar\__init__.
py", line 283, in update
    self.fd.write(self._format_line() + '\r')
LookupError: unknown encoding: cp437

I don't know if this is a bug in conda or if it's due to pulling out DLLs from under Python.

asmeurer · 2015-02-06T21:37:18Z

I have no idea how to fix these mock tests.

asmeurer · 2015-02-06T22:11:24Z

Annoyingly, the above error is only triggered if you don't use --debug.

asmeurer · 2015-02-06T22:16:32Z

I've only seen it happen when moving from Python 2 to Python 3. Unfortunately, this is something that people try to do.

asmeurer · 2015-02-06T22:35:03Z

Heh, so catching LookupError on that line fixes it. I'm still quite concerned, though.

ilanschnell · 2015-02-06T23:19:17Z

I just gave it a try on Windows XP, it updated from Python 2.7.7 to 2.7.9 without any problems. Great!

asmeurer · 2015-02-06T23:24:16Z

Anyway, don't merge at least until all of these are checked off.

ilanschnell · 2015-02-07T00:14:31Z

Yes, testing is critical here, because otherwise people are likely to hose their installations.

Maggie-M · 2015-04-16T16:08:28Z

I've managed to complete the testing from the first checkbox (test envs dirs stuff), and I didn't run into any issues. It all seems to be working correctly.

The two are related, but the pkgs dir is where we can actually expect to be able to write things.

With the new trash functionality, it is unnecessary.

It now uses Popen, and it isn't worth my time to figure out how to change these meaningless mock tests.

Otherwise the retries are on a delay timer, which would slow down the install.

Once we are sure there are no other issues with the trash stuff we can remove it again.

asmeurer · 2015-05-05T22:41:12Z

This is pretty much ready for merging. I have left the win_ignore_root in for now (so conda still won't update Python in the root environment), as I'd like to make sure there are no other issues with this before we potentially break people's installations. I'd also like to eventually move this logic to rm_rf and use that everywhere instead of unlink, but that's a future TODO.

ilanschnell · 2015-05-07T17:23:07Z

Looks great. Thanks for the effort Aaron.

Avoid the Windows file lock by moving files to a trash directory

almarklein · 2015-08-30T08:26:56Z

@asmeurer Does this apply to any package with binary modules, or only to Python, pycosat, and psutil? (Going to test this in a minute, but would like to know whether it's intended to work.)

ilanschnell · 2015-08-30T08:43:29Z

This applies to any packages, which contain files that are opened by a process and then not closed. So yes, any package with binary modules, but it could also be any other file which is open.

almarklein · 2015-08-31T09:07:04Z

Awesome :)

Sometimes updating Python in the root environment on Windows raises this error. See conda#1657 and conda#1133.

github-actions · 2021-10-04T06:02:10Z

Hi there, thank you for your contribution to Conda!

This pull request has been automatically locked since it has not had recent activity after it was closed.

Please open a new issue or pull request if needed.

asmeurer added 14 commits February 5, 2015 16:49

Add move_to_trash function, which moves a file to envs_dir/.trash

6fb83b7

Use move_to_trash on Windows when unlinking fails

c401002

Add delete_trash function to try deleting the trash directories

12f9aac

Try deleting the trash every time link() is called

02c4535

Only try deleting the trash on Windows

0af3dec

Use the trash in unlink()

769338f

TODO item

77e33cc

Clear the Windows ignored packages list

f7652fe

The trash mechanism should make this no longer necessary.

Fix typo

3edbbf9

Include the exception in the logged debug message

654c3af

Don't fail making trash directory if it already exists

afb72ae

Always move files to the trash in a temporary directory

99e48f0

This avoids issues when the a file with the same name is already in the trash.

Fix typo

564b50e

Make sure the trash directory exists

9a73ec0

Also, don't raise exceptions in move_to_trash. Try the next envs dir, and if everything fails, we only log errors.

asmeurer added the in progress label Feb 6, 2015

asmeurer added 3 commits February 6, 2015 11:23

Suppress output from rm_rf on Windows

287b4a0

Revert "Suppress output from rm_rf on Windows"

ed393dd

This reverts commit 287b4a0. subprocess.DEVNULL is only available in Python 3.

Suppress output from rm_rf on Windows

9df9169

The output is printed to the log.

asmeurer added the os::windows relevant to Windows label Feb 6, 2015

asmeurer mentioned this pull request Feb 13, 2015

CEP 1: Replacing root environment with default environment #1145

Closed

asmeurer mentioned this pull request Mar 20, 2015

Requests throwing InsecurePlatformWarning #1204

Closed

ilanschnell mentioned this pull request Apr 13, 2015

Fixes issue related to upgrades on Windows. #1273

Closed

asmeurer added 11 commits April 29, 2015 16:29

Put the trash in the pkgs dir, not the envs dir

8f7ec04

The two are related, but the pkgs dir is where we can actually expect to be able to write things.

Merge branch 'master' into windows

3d8e054

Fix redefined function name

30f5441

Remove ensure_write

8fea9b9

With the new trash functionality, it is unnecessary.

Remove some tests that test that rm_rf uses check_call

4e5e0b3

It now uses Popen, and it isn't worth my time to figure out how to change these meaningless mock tests.

Print in the log that the file is moving to the trash

0c4be06

Print a log message when deleting the trash dir

7581021

Don't try to move files to the trash if they don't exist

4960e18

rm_rf: correctly retry if rd /s fails

7041e21

Only retry once when deleting the trash directory

c419701

Otherwise the retries are on a delay timer, which would slow down the install.

Re-enable win_ignore_root for now

7fdc894

Once we are sure there are no other issues with the trash stuff we can remove it again.

asmeurer mentioned this pull request May 6, 2015

update in conda issue #1305

Closed

ilanschnell added a commit that referenced this pull request May 7, 2015

Merge pull request #1133 from asmeurer/windows

a4406fb

Avoid the Windows file lock by moving files to a trash directory

ilanschnell merged commit a4406fb into conda:master May 7, 2015

ilanschnell mentioned this pull request Jul 13, 2015

cffi update issue #1355

Closed

Korijn mentioned this pull request Sep 11, 2015

conda.install.rm_rf can't delete old environments in Windows + Python 2.7 #1594

Closed

asmeurer added a commit to asmeurer/conda that referenced this pull request Sep 29, 2015

Catch LooupError in conda.console

b145ac8

Sometimes updating Python in the root environment on Windows raises this error. See conda#1657 and conda#1133.

asmeurer mentioned this pull request Sep 29, 2015

Catch LooupError in conda.console #1661

Merged

msarahan mentioned this pull request Apr 4, 2016

conda update conda destroyed the conda installation on Windows #2325

Closed

github-actions bot added the locked [bot] locked due to inactivity label Oct 4, 2021

github-actions bot locked as resolved and limited conversation to collaborators Oct 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid the Windows file lock by moving files to a trash directory #1133

Avoid the Windows file lock by moving files to a trash directory #1133

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

ilanschnell commented Feb 6, 2015

asmeurer commented Feb 6, 2015

ilanschnell commented Feb 7, 2015

Maggie-M commented Apr 16, 2015

asmeurer commented May 5, 2015

ilanschnell commented May 7, 2015

almarklein commented Aug 30, 2015

ilanschnell commented Aug 30, 2015

almarklein commented Aug 31, 2015

github-actions bot commented Oct 4, 2021

Avoid the Windows file lock by moving files to a trash directory #1133

Avoid the Windows file lock by moving files to a trash directory #1133

Conversation

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

asmeurer commented Feb 6, 2015

ilanschnell commented Feb 6, 2015

asmeurer commented Feb 6, 2015

ilanschnell commented Feb 7, 2015

Maggie-M commented Apr 16, 2015

asmeurer commented May 5, 2015

ilanschnell commented May 7, 2015

almarklein commented Aug 30, 2015

ilanschnell commented Aug 30, 2015

almarklein commented Aug 31, 2015

github-actions bot commented Oct 4, 2021