Windows: package install sometimes fails on shutil.move of temporary directory

[mcspr]

Configuration

Operating system:
Windows 10 1909

PlatformIO Version (platformio --version):
PlatformIO, version 4.1.0 (?) / need confirmation

Description of problem

Original issue (latest comment with traceback): xoseperez/espurna#2006 (comment)

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp01_1m.html
PLATFORM: Espressif 8266 1.5.0 > Espressif Generic ESP8266 ESP-01 1M
HARDWARE: ESP8266 80MHz, 80KB RAM, 1MB Flash
PACKAGES: toolchain-xtensa 1.40802.0 (4.8.2), framework-arduinoespressif8266 1.20300.1 (2.3.0), tool-esptool 1.409.0 (4.9)
Converting espurna.ino
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Unicode decode error has occurred, please remove invalid (non-ASCII or non-UTF8) characters from C:\Users\internet\.platformio\packages\framework-arduinoespressif8266@1.20300.1\libraries\ESP8266HTTPUpdateServer\library.properties file
Unicode decode error has occurred, please remove invalid (non-ASCII or non-UTF8) characters from C:\Users\internet\.platformio\packages\framework-arduinoespressif8266@1.20300.1\libraries\ESP8266HTTPUpdateServer\library.properties file
Looking for Embedis library in registry
Found: https://platformio.org/lib/show/408/Embedis
LibraryManager: Installing id=408
Using cache: C:\Users\internet\.platformio\.cache\cc\338ee11d777d75e24f4008826cd592cc
Embedis @ 1.2.0 has been successfully installed!
LibraryManager: Installing espsoftwareserial
git version 2.24.0.windows.2
Cloning into 'D:\mcdev\vscode\espurna\code\.pio\libdeps\generic-esp01s-relay-40-inv\_tmp_installing-16pyir93-package'...
Note: switching to '5378868de76e1a38d34e0fc888d26e3612a5497d'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

PermissionError: [WinError 5] Zugriff verweigert: 'D:\\mcdev\\vscode\\espurna\\code\\.pio\\libdeps\\generic-esp01s-relay-40-inv\\_tmp_installing-16pyir93-package\\.git\\objects\\21\\452c9bf32fc95f03c8adb16ad3382d92f0994e':
  File "C:\users\internet\.platformio\penv\lib\site-packages\platformio\builder\main.py", line 151:
    env.SConscript("$BUILD_SCRIPT")
  File "C:\Users\internet\.platformio\packages\tool-scons\script\..\engine\SCons\Script\SConscript.py", line 605:
    return _SConscript(self.fs, *files, **subst_kw)
  File "C:\Users\internet\.platformio\packages\tool-scons\script\..\engine\SCons\Script\SConscript.py", line 286:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "C:\users\internet\.platformio\platforms\espressif8266@1.5.0\builder\main.py", line 375:
    target_elf = env.BuildProgram()
  File "C:\Users\internet\.platformio\packages\tool-scons\script\..\engine\SCons\Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "C:\users\internet\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 140:
    _build_project_deps(env)
  File "C:\users\internet\.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py", line 48:
    project_lib_builder = env.ConfigureProjectLibBuilder()
  File "C:\Users\internet\.platformio\packages\tool-scons\script\..\engine\SCons\Environment.py", line 224:
    return self.method(*nargs, **kwargs)
  File "C:\users\internet\.platformio\penv\lib\site-packages\platformio\builder\tools\piolib.py", line 1027:
    project.install_dependencies()
  File "C:\users\internet\.platformio\penv\lib\site-packages\platformio\builder\tools\piolib.py", line 859:
    lm.install(uri)
  File "c:\users\internet\.platformio\penv\lib\site-packages\platformio\managers\lib.py", line 295:
    pkg_dir = BasePkgManager.install(
  File "c:\users\internet\.platformio\penv\lib\site-packages\platformio\managers\package.py", line 696:
    pkg_dir = self._install_from_url(name, url, requirements, track=True)
  File "c:\users\internet\.platformio\penv\lib\site-packages\platformio\managers\package.py", line 497:
    return self._install_from_tmp_dir(_tmp_dir, requirements)
  File "c:\users\internet\.platformio\penv\lib\site-packages\platformio\managers\package.py", line 583:
    shutil.move(tmp_dir, pkg_dir)
  File "C:\Program Files (x86)\Python38-32\lib\shutil.py", line 793:
    rmtree(src)
  File "C:\Program Files (x86)\Python38-32\lib\shutil.py", line 731:
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files (x86)\Python38-32\lib\shutil.py", line 604:
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files (x86)\Python38-32\lib\shutil.py", line 604:
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files (x86)\Python38-32\lib\shutil.py", line 604:
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files (x86)\Python38-32\lib\shutil.py", line 609:
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Program Files (x86)\Python38-32\lib\shutil.py", line 607:
    os.unlink(fullname)
====================================================================================================== [FAILED] Took 5.44 seconds ======================================================================================================

If platformio.ini has lib_deps = ... with git urls, package installation will fail on shutil.move after it copies temporary directory into a package dir and tries to remove that temporary directory.

Steps to Reproduce

1. Add some git libraries into the lib_deps = .... ESPurna repo is a general example that has a couple of those.
2. Try to build any environment or pio lib install (see the referenced issue)
3. It fails right after the first git library installation. Although, package will be there on the next pio run call, because data was already copied before the previous directory caused an error and was not removed.

I cannot reproduce, so cc @davebuk @oscarsan1 @knopserl

Actual Results

See above.

Expected Results

shutil.move fails to rename directories for some reason, I would expect that step to succeed:
https://github.com/python/cpython/blob/fa919fdf2583bdfead1df00e842f24f30b2a34bf/Lib/shutil.py#L780

The specific issue is the rmtree inability to remove git object files. This was previously fixed for some parts of the platformio code by adding a custom fs.rmtree function that chmod's them via onerror hook, but the original Python function is used by the move (and it cannot be overriden)

If problems with PlatformIO Build System:

See ESPurna's
https://github.com/xoseperez/espurna/blob/dev/code/platformio.ini

Additional info

-

[valeros]

Unfortunately, I also cannot reproduce the issue, tried espurna project with Python 3.7,3.8 on Win 8, Win 10.

[mcspr]

I can't add any more info, since I can't reproduce with 3 different Windows machines as well.
Only hunch so far is exec_command(), where Popen'ed process may still be running after function returns (?). But, as discussed in the other issue, that did nothing to solve the installation problem:

platformio-core/platformio/proc.py

Line 112 in 6013903

p = subprocess.Popen(*args, **kwargs)

[knopserl]

@valeros, @ivankravets I had a quite comprehensive discussion and some "debugging" with mcspr, who supported me well. You can follow the discussion, what we've tried and what I did and the output and errors of PIO.
xoseperez/espurna#2006

I uninstalled VS Code, PIO, PYTHON, deleted directories and reinstalled everything required, but with always the same errors.
I'm using the latest Windows 10 Pro version 1909 and latest VS Code and latest PIO.
I run out of ideas and after weeks, I'm giving up. I hope you can help, based on the error log/output.

[oscarsan1]

Hi,
I don't kwow what happens in you case but I also use Windows 10 Pro version 1909 and latest VS Code and latest PIO and don't have any problems.
I folow the instructions in the wiki for instalation.

[knopserl]

@oscarsan1 so did I, and I uninstalled/reinstalled it multiple times. I'm using the latest PYTHON 3.8 (only 32Bit available) and installed for all users. But it's good to know that it "could" work in principle - just not for me :-( now

[knopserl]

@ALL I spent a lot of effort in further testing and installed VS Code + Platformio now on two different PC's with Windows 10 Pro 1909 and I did 3 different installation on 3 different disks with different users following the platformio instructions.
So in total I did certainly 5+ installation from scratch and all result in the same errors.
It always breaks with same error in shutil.py, Permission error [WinError 5] line 607 os.ulink(fullname).
I can build other projects but according to @mcspr one like espurna which uses lib_deps = and temporary lib creations and removal fail always. So I have no clue what to do and what I might do wrong on all 5 installation on two different PC's using also different users.

[ivankravets]

Do you have antivirus tools? They can block PlatformIO.

[knopserl]

I have deactivated the AV tools and also excluded the directories. I try now to deinstall all AV's on one PC (only Microsoft Defender - default on Windows 10). I can also try then to disable Microsoft Defender (having no protection) for just one build. But it's not acceptable to have no virus protection at all on a PC.
And, files can be created, but it looks like not removed/renamed or deleted?
I think there might be also another flaw with the shuitil.py, As sometimes it says that the file is locked by another process.

I can confirm now, that with all antivirus installed and only Windows defnder running, there error is exact the same. Then I even switched off alos the last protection, the Windows Defender and have NO antivirus protection, I get the same error. The problematic script code is shutil.py line 607.

[knopserl]

@ivankravets I did not give up and did further testing.
Here are my latest findings:
When I do a Build task without any environment/device selection it takes the wemos-d1mini-relayshield and that build works. But that default devices seem to do no cloning,
When I use any other devices/env. it fails with the access denied error.

Now I tried your previous recommendation and changed to espurna/code directory and executed:
platformio.exe run --environment blitzwolf-bwshpx
Surprisingly, that worked now too.
So it seem to be a VScode + platformio problem.
Any further ideas how to get that further investigated and solved?
I opened also a new issue/bug in the vscode github project. Let's see if they react and accept this as in vscode issue.

[Bmooij]

Root cause: platformio/platformio-vscode-ide#1995