Can't lock Pipfile including editable dependencies in non utf-8 Windows

[yukihiko-shinoda]

I already sent pull request to fix this issue.

Issue description

Latest version of Pipenv can't lock Pipfile including editable dependencies.
This issue occurs only in non utf-8 Windows.
(I'm using cp932 / Shift-JIS: (日本語: Japanese))

This issue may be affecting development process in Japan, Korea, and China.
cf. Windows code page - Wikipedia

Cause is that removing read only file fails.
In most case, editable dependencies includes local Git repository.
And also in most case, Git repository includes read only file.

Expected result

Succeed to lock.

Actual result

Failed to lock.

shinoda@shinoda-PC MINGW64 ~/workspace/issue-pipenv
$ pipenv lock --verbose
Locking [dev-packages] dependencies…
           Building requirements...
Resolving dependencies...
[ ===] Locking...
                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done
Traceback (most recent call last):
  File "c:\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python38\Scripts\pipenv.exe\__main__.py", line 7, in <module>
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\python38\lib\site-packages\pipenv\cli\command.py", line 368, in lock
    do_lock(
  File "c:\python38\lib\site-packages\pipenv\core.py", line 1103, in do_lock
    venv_resolve_deps(
  File "c:\python38\lib\site-packages\pipenv\utils.py", line 1323, in venv_resolve_deps
    c = resolve(cmd, sp)
  File "c:\python38\lib\site-packages\pipenv\utils.py", line 1163, in resolve
    echo(c.err.strip(), err=True)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\delegator.py", line 151, in err
    return self._pexpect_out
  File "C:\Python38\Lib\site-packages\pipenv\vendor\delegator.py", line 121, in _pexpect_out
    result += self.subprocess.read()
  File "C:\Python38\Lib\site-packages\pipenv\vendor\pexpect\spawnbase.py", line 444, in read
    self.expect(self.delimiter)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\pexpect\spawnbase.py", line 343, in expect
    return self.expect_list(compiled_pattern_list,
  File "C:\Python38\Lib\site-packages\pipenv\vendor\pexpect\spawnbase.py", line 372, in expect_list
    return exp.expect_loop(timeout)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\pexpect\expect.py", line 169, in expect_loop
    incoming = spawn.read_nonblocking(spawn.maxread, timeout)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\pexpect\popen_spawn.py", line 93, in read_nonblocking
    buf += self._decoder.decode(incoming, final=False)
UnicodeDecodeError: 'cp932' codec can't decode byte 0x82 in position 197: illegal multibyte sequence
Exception in thread Exception in thread Thread-11Thread-12:
:
Traceback (most recent call last):
Traceback (most recent call last):
  File "c:\python38\lib\threading.py", line 932, in _bootstrap_inner
  File "c:\python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()    self.run()

  File "c:\python38\lib\threading.py", line 870, in run
  File "c:\python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)    
self._target(*self._args, **self._kwargs)
  File "c:\python38\lib\subprocess.py", line 1366, in _readerthread
  File "c:\python38\lib\subprocess.py", line 1366, in _readerthread
    buffer.append(fh.read())
  File "c:\python38\lib\codecs.py", line 322, in decode
    buffer.append(fh.read())
      File "c:\python38\lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: : 'utf-8' codec can't decode byte 0x83 in position 48: invalid start byte
'utf-8' codec can't decode byte 0x8c in position 2: invalid start byte
Traceback (most recent call last):
  File "c:\python38\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] アクセスが拒否されました。: 'C:\\Users\\shinoda\\AppData\\Local\\Temp\\reqlib-src9jen7hr3\\issue-pipenv\\read_only_file.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\contextmanagers.py", line 205, in spinner
    yield _spinner
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 618, in run
    return _create_subprocess(
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 542, in _create_subprocess
    c.out, c.err = c.communicate()
  File "c:\python38\lib\subprocess.py", line 1024, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "c:\python38\lib\subprocess.py", line 1416, in _communicate
    stdout = stdout[0]
IndexError: list index out of range

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "c:\python38\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] アクセスが拒否されました。: 'C:\\Users\\shinoda\\AppData\\Local\\Temp\\reqlib-src9jen7hr3\\issue-pipenv\\read_only_file.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\path.py", line 433, in rmtree
    shutil.rmtree(directory, ignore_errors=ignore_errors, onerror=onerror)
  File "c:\python38\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\python38\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python38\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\path.py", line 496, in handle_remove_readonly
    set_write_bit(path)
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\path.py", line 383, in set_write_bit
    c = run(
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 618, in run
    return _create_subprocess(
  File "C:\Python38\Lib\site-packages\pipenv\vendor\vistir\misc.py", line 542, in _create_subprocess
    c.out, c.err = c.communicate()
  File "c:\python38\lib\subprocess.py", line 1024, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "c:\python38\lib\subprocess.py", line 1416, in _communicate
    stdout = stdout[0]
IndexError: list index out of range

Note that, "アクセスが拒否されました。" means "Access denied.".

Steps to replicate

1.
Create Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pipenvissue = {editable = true, path = "."}

[packages]

2.
Create setup.py:

#!/usr/bin/env python
"""This module implements build settings."""

from setuptools import find_packages, setup  # type: ignore


def main():
    """This function implements build settings."""
    setup(
        name="pipenvissue",
        version="1.0.0",
        description="",
        long_description="",
        author="Yukihiko Shinoda",
        author_email="yuk.hik.future@gmail.com",
        packages=find_packages(),
        python_requires="",
        install_requires=[],
        dependency_links=[],
        classifiers=[],
    )


if __name__ == "__main__":
    main()

3.
Create file which has read only permission to simulate local git repository.
In most case, git repository includes read only file.

Note that, "読み取り専用" means "Read-only".

4.
Create Package directory:

issue-pipenv
+---pipenvissue
|   +---__init__.py
+---Pipfile
+---setup.py
+---read_only_file.txt

5.
Run pipenv lock:

pipenv lock

---

shinoda@shinoda-PC MINGW64 ~/workspace/issue-pipenv
$ pipenv --support

$ pipenv --support

Pipenv version: '2020.6.2'

Pipenv location: 'c:\\python38\\lib\\site-packages\\pipenv'

Python location: 'c:\\python38\\python.exe'

Python installations found:

• 3.8.3: C:\Python38\python.exe
• 3.7.7: C:\Python37\python.exe
• 3.6.8: C:\Python36\python.exe
• 3.5.4: C:\Python35\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.3',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.18362',
 'python_full_version': '3.8.3',
 'python_version': '3.8',
 'sys_platform': 'win32'}

System environment variables:

• ALLUSERSPROFILE
• ANT_HOME
• APPDATA
• COLORTERM
• COMMONPROGRAMFILES
• COMPUTERNAME
• COMSPEC
• CHOCOLATEYINSTALL
• CHOCOLATEYLASTPATHUPDATE
• CHOCOLATEYTOOLSLOCATION
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• DRIVERDATA
• EXEPATH
• FPS_BROWSER_APP_PROFILE_STRING
• FPS_BROWSER_USER_PROFILE_STRING
• FP_NO_HOST_CHECK
• GIT_ASKPASS
• GS_PATH
• HOME
• HOMEDRIVE
• HOMEPATH
• INTEL_DEV_REDIST
• LANG
• LOCALAPPDATA
• LOGONSERVER
• M2
• M2_REPO
• MAVEN_OPTS
• MIC_LD_LIBRARY_PATH
• MSYS
• MSYSTEM
• NODE_PATH
• NODIST_PREFIX
• NODIST_X64
• NUMBER_OF_PROCESSORS
• ORIGINAL_XDG_CURRENT_DESKTOP
• OS
• ONEDRIVE
• ONEDRIVECONSUMER
• PATH
• PATHEXT
• PLINK_PROTOCOL
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMFILES
• PS1
• PSMODULEPATH
• PUBLIC
• PWD
• PROGRAMDATA
• PROGRAMFILES(X86)
• PROGRAMW6432
• SESSIONNAME
• SHLVL
• SLACK_WEBHOOK_URL
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TERM
• TERM_PROGRAM
• TERM_PROGRAM_VERSION
• TMP
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VBOX_MSI_INSTALL_PATH
• VS140COMNTOOLS
• VSCODE_GIT_ASKPASS_MAIN
• VSCODE_GIT_ASKPASS_NODE
• VSCODE_GIT_IPC_HANDLE
• WINDIR
• _
• ASL.LOG
• WINDOWS_TRACING_FLAGS
• WINDOWS_TRACING_LOGFILE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\shinoda\bin;C:\Python38\Scripts;C:\Python38;C:\Python37\Scripts;C:\Python37;C:\Python36\Scripts;C:\Python36;C:\Python35\Scripts;C:\Python35;C:\tools\Anaconda3\Scripts;C:\tools\Anaconda3;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler;C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Live;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files (x86)\Windows Live\Shared;C:\ProgramData\chocolatey\bin;C:\Program Files\Java\jdk1.8.0_74\bin;C:\Program Files\Java\jdk1.8.0_92\bin;C:\Program Files\Java\jdk1.8.0_102\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files\Java\jdk1.8.0_112\bin;C:\Program Files\Java\jdk1.8.0_121\bin;C:\tools\go\bin;C:\Program Files\Java\jdk1.8.0_131\bin;C:\Program Files\Java\jdk1.8.0_141\bin;C:\Program Files\gs\gs9.21\bin;C:\Program Files\gs\gs9.21\lib;C:\Program Files\Java\jdk1.8.0_152\bin;C:\Program Files\Java\jdk1.8.0_162\bin;C:\WINDOWS\System32\OpenSSH;C:\Program Files\Java\jdk1.8.0_191\bin;C:\Program Files (x86)\Common Files\Sony Shared\FeliCaLibrary;C:\Program Files\Common Files\Sony Shared\FeliCaLibrary;C:\Program Files (x86)\Common Files\Sony Shared\FeliCaNFCLibrary;C:\Program Files\Common Files\Sony Shared\FeliCaNFCLibrary;C:\Program Files\Microsoft VS Code\bin;C:\Program Files (x86)\Nodist\bin;C:\Program Files\Java\jdk1.8.0_201\bin;C:\Program Files\Java\jdk1.8.0_211\bin;C:\Program Files\Java\jdk1.8.0_221\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit;C:\tools\php73;C:\tools\php74;C:\PROGRA~1\JPKI;C:\Program Files\TortoiseGit\bin;C:\Program Files (x86)\Streamlink\bin;C:\HashiCorp\Vagrant\bin;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Git\cmd;C:\Users\shinoda\AppData\Roaming\Python\Python38\Scripts;C:\Program Files\Oracle\VirtualBox;C:\Users\shinoda\AppData\Local\Microsoft\WindowsApps;C:\Users\shinoda\go\bin;C:\Program Files\GPAC;C:\Users\shinoda\AppData\Local\Microsoft\WindowsApps;C:\Users\shinoda\go\bin;C:\Users\shinoda\AppData\Local\Programs\Microsoft VS Code Insiders\bin;C:\Users\shinoda\AppData\Local\GitHubDesktop\bin
• LANG: en_US.UTF-8
• PWD: C:/Users/shinoda/workspace/issue-pipenv

---

Contents of Pipfile ('C:\Users\shinoda\workspace\issue-pipenv\Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pipenvissue = {editable = true, path = "."}

[packages]

[matteius]

Thanks for your contribution! This should be solved on pipenv=2022.8.17 ... let us know if it is not.

[yukihiko-shinoda]

I confirmed that this issue is fixed.
However, fixed from pipenv=2020.8.13, maybe locking algorithm is changed in 2020.6.2 -> 2020.8.13.