Cannot run under POSIX locale

[jonapich]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: Ubuntu 18.04

• Poetry version: 1.0.0

Issue

Running poetry under a POSIX locale will raise this error:

[UnicodeEncodeError]
'ascii' codec can't encode character '\xa0' in position 30: ordinal not in range(128)

locales:

LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"**
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Workaround: Set the locale to something else... Specifically for this image, the C.UTF-8 locale is available:

export LC_ALL=C.UTF-8

Only then, will poetry correctly display its menu.

[jonapich]

The source of the problem: poetry attempts to display the following:

USAGE
  poetry [-h] [-q] [-v [<...>]] [-V] [--ansi] [--no-ansi] [-n] <command> [<arg1>] ... [<argN>]

However, clikit (cleo?) prints the character \xa0 between [-v and [<...>]] instead of a regular space. This causes the UnicodeEncodeError with the following stacktrace:

[UnicodeEncodeError]
'ascii' codec can't encode character '\xa0' in position 30: ordinal not in range(128)

Traceback (most recent call last):
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/api/command/command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/handler/help/help_text_handler.py", line 29, in handle
    usage.render(io)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/ui/help/abstract_help.py", line 31, in render
    layout.render(io, indentation)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/ui/layout/block_layout.py", line 42, in render
    element.render(io, self._indentations[i] + indentation)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/ui/components/labeled_paragraph.py", line 70, in render
    + '
'"
  File "/root/.poetry/lib/poetry/_vendor/py3.6/cleo/io/io_mixin.py", line 55, in write
    super(IOMixin, self).write(string, flags)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/api/io/io.py", line 58, in write
    self._output.write(string, flags=flags)
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/api/io/output.py", line 61, in write
    self._stream.write(to_str(formatted))
  File "/root/.poetry/lib/poetry/_vendor/py3.6/clikit/io/output_stream/stream_output_stream.py", line 24, in write
    self._stream.write(string)

...is this a bug in the clikit/cleo stuff?

Additionally I found this character in poetry/mixology/version_solver.py on line 288 and 289.

[jaepil-choi]

Having a similar issue with UnicodeEncodeError. \xa0 is causing the issue on my end, too. After extensive search on Google, it seems like a common Python encoding issue

My issue: #1887

[george-hawkins]

There have been a number of bugs logged related to this, e.g. #1427, #1887 and this one. So it seems it's not entirely uncommon for people to hit this issue.

@abn closed #1427 explaining that this wasn't a Poetry issue but was a user issue, i.e. the user should understand what's gone wrong and know that they need to update their LC_ALL or LANG environment variables to enable UTF-8 handing.

OK, fair enough the change needs to be made outside Poetry, but I think it wouldn't be unreasonable for Poetry to actively check for this issue and fail in a rather less obscure way. It's a little disconcerting as a first time user if the very first thing you try with Poetry fails like this:

$ poetry --help
Poetry version 1.0.5

USAGE

[UnicodeEncodeError]
'ascii' codec can't encode character '\xa0' in position 30: ordinal not in range(128)

I suggest at the very least that something like the following is added to the __init__ method for Application in console/application.py:

if locale.getpreferredencoding(False) == "ANSI_X3.4-1968":
    sys.exit("UTF-8 support is required but ASCII has been configured as the preferred encoding.")

Ideally, one would add more details about setting LC_ALL or LANG or provide a link to more information in a trouble-shooting section of the documentation. The Click package (that's used to provide the command line interface for Flask and other projects) does something like this.

Or one could do what many packages do and actively fix the problem, e.g. with something like this (see main.py:34 in the Peru project):

def force_utf8_in_ascii_mode_hack():
    '''In systems without a UTF8 locale configured, Python will default to
    ASCII mode for stdout and stderr. This causes our fancy display to fail
    with encoding errors. In particular, you run into this if you try to run
    peru inside of Docker. This is a hack to force emitting UTF8 in that case.
    Hopefully it doesn't break anything important.'''
    if sys.stdout.encoding == 'ANSI_X3.4-1968':
        sys.stdout = open(
            sys.stdout.fileno(), mode='w', encoding='utf8', buffering=1)
        sys.stderr = open(
            sys.stderr.fileno(), mode='w', encoding='utf8', buffering=1)

Note: in Python 3.7 they introduced io.TextIOWrapper.reconfigure which allows you to reconfigure the encoding for an existing stream as long as you haven't already outputted data to the stream in question (but obviously Poetry needs to support pre-3.7 version of Python as well).

[txels]

I see various open issues related to poetry not properly parsing non-ascii names. I am not sure which one to report my error to... On my ubuntu 20.10, even with every locale set as UTF-8, I get issues with my name not being parsed properly. The value comes from .gitconfig which is also utf-8 encoded.

 $ LC_ALL=en_GB.UTF-8 poetry init

This command will guide you through creating your pyproject.toml config.

Package name [wishlist2]:  
Version [0.1.0]:  
Description []:  
Author [Carles Barrobés <carles@barrobes.com>, n to skip]:  
Invalid author string. Must be in the format: John Smith <john@example.com>

Oddly, typing it again on the terminal, poetry accepts it.

Author [Carles Barrobés <carles@barrobes.com>, n to skip]:  
Invalid author string. Must be in the format: John Smith <john@example.com>
Author [Carles Barrobés <carles@barrobes.com>, n to skip]:  Carles Barrobés <carles@barrobes.com>
License []:  

...later on I have other issues with virtualenvs, but that's a different topic, will report separately.

[dimbleby]

seems to work fine today:

$ locale
LANG=
LANGUAGE=
LC_CTYPE=POSIX
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

$ poetry --help

Description:
  Lists commands.

Usage:
  list [options] [--] [<namespace>]

...

I guess this can be closed

[neersighted]

Indeed, the immediate reproduction no longer works and I think we should close this issue. I am aware of some more latent issues with locales under the surface, but we likely should track/fix those separately.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.