Default encoding for .env files

[feakuru]

I had some Russian symbols in my .env file while launching pipenv shell from Windows 10 and got the following result:

$ pipenv shell
Loading .env environment variables…
Traceback (most recent call last):
  File "c:\program files (x86)\python36-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files (x86)\python36-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files (x86)\Python36-32\Scripts\pipenv.exe\__main__.py", line 9, in <module>
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\cli.py", line 564, in shell
    core.load_dot_env()
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\core.py", line 155, in load_dot_env
    dotenv.load_dotenv(denv, override=True)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 250, in load_dotenv
    return DotEnv(f, verbose=verbose).set_as_environment_variables(override=override)
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 94, in set_as_environment_variables
    for k, v in self.dict().items():
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 73, in dict
    values = OrderedDict(self.parse())
  File "c:\program files (x86)\python36-32\lib\site-packages\pipenv\vendor\dotenv\main.py", line 80, in parse
    for line in f:
  File "c:\program files (x86)\python36-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 426: character maps to <undefined>

If it helps, here's the help output

$ python -m pipenv.help

$ python -m pipenv.help output

Pipenv version: '11.10.0'

Pipenv location: 'C:\\Program Files (x86)\\Python36-32\\lib\\site-packages\\pipenv'

Python location: 'C:\\Program Files (x86)\\Python36-32\\python.exe'

Other Python installations in PATH:

• 3.6: C:\Program Files (x86)\Python36-32\python.exe

• 3.6.4: C:\Program Files (x86)\Python36-32\python.exe

• 3.6.4: C:\Windows\py.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.4',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.16299',
 'python_full_version': '3.6.4',
 'python_version': '3.6',
 'sys_platform': 'win32'}

System environment variables:

• ACLOCAL_PATH
• ALLUSERSPROFILE
• APPDATA
• COMMONPROGRAMFILES
• COMPUTERNAME
• COMSPEC
• CONFIG_SITE
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• DISPLAY
• EXEPATH
• FPS_BROWSER_APP_PROFILE_STRING
• FPS_BROWSER_USER_PROFILE_STRING
• FSHARPINSTALLDIR
• HOME
• HOMEDRIVE
• HOMEPATH
• HOSTNAME
• INFOPATH
• LANG
• LEIN_JAVA_CMD
• LOCALAPPDATA
• LOGONSERVER
• MANPATH
• MINGW_CHOST
• MINGW_PACKAGE_PREFIX
• MINGW_PREFIX
• MSYSTEM
• MSYSTEM_CARCH
• MSYSTEM_CHOST
• MSYSTEM_PREFIX
• NUMBER_OF_PROCESSORS
• ORIGINAL_PATH
• ORIGINAL_TEMP
• ORIGINAL_TMP
• OS
• ONEDRIVE
• PATH
• PATHEXT
• PKG_CONFIG_PATH
• PLINK_PROTOCOL
• PROCESSOR_ARCHITECTURE
• PROCESSOR_ARCHITEW6432
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMFILES
• PS1
• PSMODULEPATH
• PUBLIC
• PWD
• PROGRAMDATA
• PROGRAMFILES(X86)
• PROGRAMW6432
• SESSIONNAME
• SHELL
• SHLVL
• SSH_ASKPASS
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TERM
• TMP
• TMPDIR
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• WINDIR
• _
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONUNBUFFERED

Pipenv▒specific environment variables:

Debug▒specific environment variables:

• PATH: C:\Users\feana\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\feana\bin;C:\Program Files\Docker\Docker\Resources\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Python36-32\Scripts;C:\Program Files (x86)\Python36-32;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files\Git\cmd;C:\Users\feana\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft VS Code\bin;C:\Users\feana\.lein\bin;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl
• SHELL: C:\Program Files\Git\usr\bin\bash.exe
• LANG: en_US.UTF-8
• PWD: C:/Users/feana

---

Expected result

Correct .env file parsing.

Actual result

Somehow, it thinks the file is encoded in CP1252. Why?..

Steps to replicate

1. Be on Win10
2. Have cyrillic in .env
3. Run pipenv shell

[feakuru]

Actually, now that I've removed the cyrillic symbols, I'm getting an AttributeError: module 'pexpect' has no attribute 'spawn'. Should I open another issue?..

[uranusjr]

@feakuru Yes, this should not happen. Post the traceback and I’ll work on it. Regarding the env file, what encoding do you expect it to be decoded with? UTF-8? (And for your question, it does not know what the file is encoded in, so it simply uses the system default, i.e. the code page encoding on Windows.)

[techalchemy]

What about using io.open or whatever?

[uranusjr]

Dotenv already uses io.open but does not specify an encoding 🤷‍♂️ See the linked python-dotenv issue above. I fixed that without touching dotenv however, by passing it a pre-decoded StringIO instead.

[techalchemy]

@uranusjr so are we good to close this?

[uranusjr]

@techalchemy Urrgh, I never got to work on this in the end… This is still not fixed, sorry.

[trim21]

meet same bug with utf-8 encoded .env file on windows

this is a bug of python-dotenv, they just open .env file and try to readline.

Their solution is that they will add a arguments encoding(default 'utf-8'), which means pipenv need to find .env file's corrent encoding first and pass it to python-dotenv to fix issue like this.

theskumar/python-dotenv#161

[uranusjr]

Pull request is welcomed. Don’t wait for people; take on it yourself if you already know what to do :)

[trim21]

@uranusjr I do want to start a pull request, but theskumar/python-dotenv#161 is not merged yet. Unless I can merge it into vendor before python-dotenv merge this pr.

[uranusjr]

cc @techalchemy to maybe revisit this when we’re about to release the next version.

[matteius]

I believe this was resolved.