Review of (explicit) encoding for files being created in text mode

[vlcinsky]

• OS version and name: Windows and other systems not using UTF-8 default encoding
• Poetry version: 0.12.15

Resolving issue #1027 by PR #1085 we have found a pattern, which might cause poetry to fail on systems, which are not using default utf-8 encoding.

The pattern is opening files for writing in text mode without explicit encoding.

Possibly correct default encoding

Some files are written and used on the same system, thus shall be fine using their own default encoding for writing and reading the content.

Here are the candidates:

get-poetry.py|564 col 18| with open(os.path.join(POETRY_BIN, "poetry.bat"), "w") as f:
get-poetry.py|573 col 14| with open(os.path.join(POETRY_BIN, "poetry"), "w") as f:
get-poetry.py|585 col 14| with open(os.path.join(POETRY_HOME, "env"), "w") as f:
get-poetry.py|606 col 18| with open(profile, "r") as f:
get-poetry.py|610 col 22| with open(profile, "a") as f:
get-poetry.py|703 col 18| with open(profile, "r") as f:
get-poetry.py|719 col 18| with open(profile, "w") as f:

Files which shall use explicit encoding

Some files shall have specific encoding by default. Using default encoding is probably an error.

Candidates in get-poetry

get-poetry.py|390 col 18| with open(os.path.join(POETRY_LIB, "poetry", "__version__.py")) as f:

Candidates in core code:

poetry/spdx/__init__.py|29 col 10| with open(licenses_file) as f:
poetry/spdx/updater.py|23 col 14| with open(file, "w") as f:
poetry/spdx/updater.py|30 col 17| with urlopen(url) as r:
poetry/installation/pip_installer.py|196 col 18| with open(setup, "w") as f:
poetry/layouts/src.py|17 col 27| with package_init.open("w") as f:
poetry/layouts/layout.py|143 col 28| with tests_default.open("w") as f:
poetry/layouts/layout.py|155 col 21| with poetry.open("w") as f:
poetry/layouts/standard.py|17 col 27| with package_init.open("w") as f:
poetry/json/__init__.py|22 col 10| with open(schema) as f:
poetry/puzzle/provider.py|349 col 39| with requires.open() as f:
poetry/repositories/pypi_repository.py|536 col 35| with requires.open() as f:
poetry/repositories/pypi_repository.py|548 col 35| with requires.open() as f:
poetry/masonry/metadata.py|49 col 33| with package.readme.open() as f:
poetry/console/commands/init.py|161 col 46| with (Path.cwd() / "pyproject.toml").open("w") as f:

Candidates in test suite:

tests/console/conftest.py|179 col 22| with p.file.path.open() as f:
tests/console/conftest.py|187 col 22| with p.file.path.open("w") as f:
tests/repositories/test_pypi_repository.py|36 col 22| with fixture.open() as f:
tests/repositories/test_legacy_repository.py|32 col 22| with fixture.open() as f:
tests/utils/fixtures/setups/ansible/setup.py|57 col 10| with open(SYMLINK_CACHE, "w") as f:
tests/utils/fixtures/setups/ansible/setup.py|66 col 14| with open(SYMLINK_CACHE, "r") as f:
tests/utils/fixtures/setups/ansible/setup.py|142 col 10| with open(file_name, "r") as f:
tests/utils/fixtures/setups/sqlalchemy/setup.py|115 col 6| with open(
tests/utils/fixtures/setups/sqlalchemy/setup.py|120 col 6| with open(os.path.join(os.path.dirname(__file__), "README.rst")) as r_file:

tests/utils/fixtures/setups/pyyaml/setup.py|218 col 20| data = open(cache).read().strip()
tests/utils/fixtures/setups/pyyaml/setup.py|225 col 9| open(src, "w").write(ext.feature_check)
tests/utils/fixtures/setups/pyyaml/setup.py|246 col 13| open(cache, "w").write("0\n")
tests/utils/fixtures/setups/pyyaml/setup.py|271 col 13| open(cache, "w").write("0\n")
tests/utils/fixtures/setups/pyyaml/setup.py|273 col 9| open(cache, "w").write("1\n")
tests/fixtures/git/github.com/demo/no-version/setup.py|9 col 10| with open(os.path.join(os.path.dirname(__file__), "demo", "__init__.py")) as f:

Next actions

• review, if get-poetry.py shall really rely on default system encoding
• fix it in branch based on master (intended for soon release as 0.12.x)
• agree on how to push these changes to develop branch

[vlcinsky]

@sdispater PR #1088 shall resolve this issue. I did the changes against master branch as I hope, it will resolve couple of subtle bugs and number of open issues could get down (what would require some review of currently reported issues).

Automated tests are passing.

If you accept it, propose, who and how shall manage getting these changes into develop branch.

[saidbouras]

For Windows users (using ubuntu wsl too), is there any workaround ? Or do we have to wait for a new release ? I think it's a great flaw for developers working on Windows (corporate company use to work on Windows).
I wanted to leverage Poetry on my company for python projects but now I will have to switch back to pipenv....

[vlcinsky]

@saidbouras did you try the latest poetry version?

According to https://github.com/sdispater/poetry/blob/master/CHANGELOG.md version 0.12.17 shall include fix to this issue.

In fact, I will close the issue as I consider it resolved.

[saidbouras]

@vlcinsky thanks for your fast response.
I am currently using version 0.12.17 and when I try to update project I get this error below :
[UnicodeEncodeError] 'ascii' codec can't encode character u'\u2018' in position 2289: ordinal not in range(128)

[vlcinsky]

@saidbouras the message could be related to printing (e.g. UTF-8 based) text to console, which has problems to handle that.

Could you provide more details? Was there any stack trace around?

[saidbouras]

Sorry, I am beginning in python in general, but I investigated a bit and it seems that it's related to a dependency : sasl=0.2.1.
When I delete this dependency poetry successfully create the wheel artefact.
There is maybe a verbose mode for command poetry install...I will check that and put results log here.

Thanks !

EDIT : below the trace.

Package operations: 3 installs, 0 updates, 0 removals

  - Installing sasl (0.2.1)

[UnicodeEncodeError]
'ascii' codec can't encode character u'\u2018' in position 2278: ordinal not in range(128)

Exception trace:
 /home/said/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /home/said/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /home/said/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /home/said/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /home/said/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /home/said/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /home/said/.poetry/lib/poetry/console/commands/update.py in handle() at line 41
   return installer.run()
 /home/said/.poetry/lib/poetry/installation/installer.py in run() at line 73
   self._do_install(local_repo)
 /home/said/.poetry/lib/poetry/installation/installer.py in _do_install() at line 290
   self._execute(op)
 /home/said/.poetry/lib/poetry/installation/installer.py in _execute() at line 306
   getattr(self, "_execute_{}".format(method))(operation)
 /home/said/.poetry/lib/poetry/installation/installer.py in _execute_install() at line 331
   self._installer.install(operation.package)
 /home/said/.poetry/lib/poetry/installation/pip_installer.py in install() at line 91
   self.run(*args)
 /home/said/.poetry/lib/poetry/installation/pip_installer.py in run() at line 112
   return self._env.run("python", "-m", "pip", *args, **kwargs)
 /home/said/.poetry/lib/poetry/utils/env.py in run() at line 539
   return super(VirtualEnv, self).run(bin, *args, **kwargs)
 /home/said/.poetry/lib/poetry/utils/env.py in run() at line 388
   raise EnvCommandError(e, input=input_)
 /home/said/.poetry/lib/poetry/utils/env.py in __init__() at line 99
   e.cmd, e.returncode, decode(e.output)

[vlcinsky]

Thanks @saidbouras

According to provided stacktrace there seem to be a problem to install sasl in general. This issue is very likely not related to the issue of encoding.

I would propose:

• try to install sasl==0.2.1 using pip - did you succeed? Did it fail? If so, report the issue to sasl. The cause could be either problem in sasl lib, or missing headers for the package needed for compilation...
• if pip works, try again poetry. Is it working now? If it is still failing? If so, create new issue in poetry as it is probably not related to the encoding described in current one.

Good luck in world of Python - it is amazing one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.