UnicodeDecodeError in Masonry with MSYS2 #1027

danyeaw · 2019-04-10T01:34:04Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
OS version and name: Windows 10 with MSYS2 20180531
Poetry version: 0.12.11
Link of a Gist with the contents of your pyproject.toml file: https://github.com/gaphor/gaphor/blob/master/pyproject.toml

Issue

When trying to pip install my app I am getting a UnicodeDecodeError.

$ pip3 install --user .
Processing c:/tools/msys64/home/dyeaw/gaphor
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... error
    Complete output from command C:/tools/msys64/mingw64/bin/python3.exe C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel C:/Users/dyeaw/AppData/Local/Temp/tmp237x9yfr:
    Traceback (most recent call last):
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 64, in prepare_metadata_for_build_wheel
        hook = backend.prepare_metadata_for_build_wheel
    AttributeError: module 'poetry.masonry.api' has no attribute 'prepare_metadata_for_build_wheel'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 207, in <module>
        main()
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 67, in prepare_metadata_for_build_wheel
        config_settings)
      File "C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py", line 95, in _get_wheel_metadata_from_wheel
        whl_basename = backend.build_wheel(metadata_directory, config_settings)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\api.py", line 40, in build_wheel
        poetry, SystemEnv(Path(sys.prefix)), NullIO(), Path(wheel_directory)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\wheel.py", line 48, in make_in
        wb = WheelBuilder(poetry, env, io, target_dir=directory, original=original)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\wheel.py", line 38, in __init__
        super(WheelBuilder, self).__init__(poetry, env, io)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\builders\builder.py", line 42, in __init__
        self._meta = Metadata.from_package(self._package)
      File "C:/Users/dyeaw/AppData/Local/Temp/pip-build-env-n028mbl_/overlay/lib/python3.7/site-packages\poetry\masonry\metadata.py", line 50, in from_package
        meta.description = f.read()
      File "C:/tools/msys64/mingw64/lib/python3.7\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 7221: character maps to <undefined>

    ----------------------------------------
Command "C:/tools/msys64/mingw64/bin/python3.exe C:/tools/msys64/mingw64/lib/python3.7/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel C:/Users/dyeaw/AppData/Local/Temp/tmp237x9yfr" failed with error code 1 in C:/Users/dyeaw/AppData/Local/Temp/pip-req-build-clnw4_9m

It looks like Poetry is not using an encoding when it calls open() to load the description.

I am opening the issue here after exploring the cause in MSYS2 first: msys2/MINGW-packages#5155

The text was updated successfully, but these errors were encountered:

drunkwcodes · 2019-04-10T22:56:37Z

Save your pyproject.toml in utf-8 encoding again should be alright?

To handle different encodings, it needs to depend on chardet without a guarentee resolving this problem.

Reference: https://stackoverflow.com/questions/436220/how-to-determine-the-encoding-of-text

danyeaw · 2019-04-10T23:55:44Z

@drunkwcodes My pyproject.toml, and the rest of my project is in UTF-8. The issue is that Poetry is calling open() with no encoding specified, and the default encoding for Windows is cp1252.

drunkwcodes · 2019-04-11T04:47:40Z

Sorry. Here it is. Please try to change msys2's encoding setting to utf-8.
Or use chcp 65001 command.

https://stackoverflow.com/questions/28481410/how-to-set-console-encoding-in-msys

danyeaw · 2019-04-11T12:31:30Z

@drunkwcodes Thanks for trying to research possible solutions. The correct default/preferred encoding of the contents of files for Windows users is cp1252. This is what is returned when I call locale.getpreferredencoding(). It is not a solution for me to change my default encoding. This issue here is that Poetry is opening pyproject.toml, which should be UTF-8 for all users, without specifying the encoding to use.

drunkwcodes · 2019-04-12T02:08:13Z

@danyeaw Sorry for not having a solution for you to change your default encoding, sir. What a lovely night.

joar · 2019-04-12T11:55:49Z

I believe #1030 is related to this issue. I don't think it's safe to assume that the locale will always be set to UTF-8 on systems where poetry (or packages using poetry, such as pendulum) might be installed.

vlcinsky · 2019-05-07T12:46:06Z

We (me and @rhorenov) had problem to install pendulum==2.0.4 on MS Windows with encoding windows-1250.

If the package was to be built (e.g. pendulum), it failed on special characters in metadata generated by poetry as it was opening files for build using default console encoding.

Installing packages from wheels worked well.

I am going to create PR for master to fix this issue (adding explicit encoding where files are open for writing in text mode).

…try#1027) This is minimal set of modifications, which fixed the problem. There are still more places, where files are open in text mode without explicit encoding, thus possibly failing on systems not using UTF-8 console encoding.

…1088) * Fix encoding for files created for building package (issue #1027) This is minimal set of modifications, which fixed the problem. There are still more places, where files are open in text mode without explicit encoding, thus possibly failing on systems not using UTF-8 console encoding. * test_api.py use explicit encoding for files. * get-poetry.py: fix explicit (utf-8) encoding for files open in text mode * tests: fix explicit (utf-8) encoding for files open in text mode * poetry init: use explicit utf-8 for created pyproject.toml * installed: use explicit utf-8 for text files * layouts: use explicit utf-8 for open text files * spdx: explicit utf-8 for open text files * pypi_repository: explicit utf-8 for open text files * explicit utf-8 for json * explicit utf-8 for metadata * explicit utf-8 for requires.txt * open files with explicit encoding in py27 * fix creation of poetry.bat on Windows * Make get-poetry.py compatible to py2/3 * Blacked two files to pass linter test * remove extranous encoding related comments * rewert broken move of import related to ansible * removed extraneous comments about opening files in UTF-8 * rewert ansible.release import (keep order of vars unchanged) * Revert to original content (as it comes from external project) * all vendored setup.py files in fixtures reverted back to org

vlcinsky · 2019-07-10T16:52:15Z

Can @danyeaw or anyone else check, that the latest poetry version 0.12.17 is not not suffering this problem?

danyeaw · 2019-07-10T17:21:37Z

Closed by commit e9a13f8 👍

vlcinsky · 2019-07-10T18:33:19Z

Greate @danyeaw

Can you close the issue? I do not seem to have such option.

github-actions · 2024-03-03T16:06:03Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

danyeaw mentioned this issue Apr 11, 2019

Fix UnicodeDecodeError caused by opening README with no encoding #1029

Closed

2 tasks

joar mentioned this issue Apr 12, 2019

poetry 0.12.12 breaks pendulum install on Python 2.7 #1030

Closed

3 tasks

vlcinsky mentioned this issue May 7, 2019

Review of (explicit) encoding for files being created in text mode #1087

Closed

3 tasks

cjerdonek mentioned this issue Jun 19, 2019

pip 19.1.1 fails to install pendulum 2.0.4 pypa/pip#6566

Closed

danyeaw closed this as completed Jul 11, 2019

github-actions bot locked as resolved and limited conversation to collaborators Mar 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UnicodeDecodeError in Masonry with MSYS2 #1027

UnicodeDecodeError in Masonry with MSYS2 #1027

danyeaw commented Apr 10, 2019

drunkwcodes commented Apr 10, 2019

danyeaw commented Apr 10, 2019

drunkwcodes commented Apr 11, 2019 •

edited

danyeaw commented Apr 11, 2019

drunkwcodes commented Apr 12, 2019

joar commented Apr 12, 2019

vlcinsky commented May 7, 2019

vlcinsky commented Jul 10, 2019

danyeaw commented Jul 10, 2019

vlcinsky commented Jul 10, 2019

github-actions bot commented Mar 3, 2024

UnicodeDecodeError in Masonry with MSYS2 #1027

UnicodeDecodeError in Masonry with MSYS2 #1027

Comments

danyeaw commented Apr 10, 2019

Issue

drunkwcodes commented Apr 10, 2019

danyeaw commented Apr 10, 2019

drunkwcodes commented Apr 11, 2019 • edited

danyeaw commented Apr 11, 2019

drunkwcodes commented Apr 12, 2019

joar commented Apr 12, 2019

vlcinsky commented May 7, 2019

vlcinsky commented Jul 10, 2019

danyeaw commented Jul 10, 2019

vlcinsky commented Jul 10, 2019

github-actions bot commented Mar 3, 2024

drunkwcodes commented Apr 11, 2019 •

edited