Poe 0.16.5 doesn't work on some non-utf-8 Windows environment

[privet-kitty]

Hi. Thank you for maintaining this awesome task-runner. I ran into a problem on Windows with Japanese language environment. The minimal reproducible example would be as follows:

1. Put the following pyproject.toml under a directory whose path contains multi-byte characters (e.g. C:\Users\username\テスト).

   [tool.poetry]
   name = "test-package"
   version = "0.1.0"
   description = ""
   authors = []
   
   [tool.poetry.dependencies]
   python = "^3.10"
   
   [tool.poetry.group.dev.dependencies]
   poethepoet = "^0.16.5"
   
   [build-system]
   requires = ["poetry-core"]
   build-backend = "poetry.core.masonry.api"
   
   [tool.poe.tasks]
   python = "python --version"

2. Set up the environment inside the directory bypoetry config virtualenvs.in-project true --local and poetry install.
3. Run poe python and see it fail.
   • I guess it succeeds on a pure utf-8 environment. To forcibly reproduce the error, setting PYTHONIOENCODING="cp932" in advance would be a way, for example.

The error would be as follows.

Poe => python --version
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\user\テスト\.venv\Scripts\poe.exe\__main__.py", line 7, in <module>
  File "C:\Users\user\テスト\.venv\Lib\site-packages\poethepoet\__init__.py", line 33, in main
    result = app(cli_args=sys.argv[1:])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\テスト\.venv\Lib\site-packages\poethepoet\app.py", line 67, in __call__
    return self.run_task() or 0
           ^^^^^^^^^^^^^^^
  File "C:\Users\user\テスト\.venv\Lib\site-packages\poethepoet\app.py", line 98, in run_task
    return self.task.run(context=context, extra_args=self.ui["task"][1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\テスト\.venv\Lib\site-packages\poethepoet\task\base.py", line 239, in run
    return self._handle_run(
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\user\テスト\.venv\Lib\site-packages\poethepoet\task\cmd.py", line 44, in _handle_run
    return context.get_executor(self.invocation, env, self.options).execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\テスト\.venv\Lib\site-packages\poethepoet\executor\poetry.py", line 27, in execute
    poetry_env = self._get_poetry_virtualenv()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\テスト\.venv\Lib\site-packages\poethepoet\executor\poetry.py", line 76, in _get_poetry_virtualenv
    .decode()
     ^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 15: invalid start byte

The cause is, bytes.decode() assumes utf-8 when no encoding is given, but what poetry env info -p outputs is not necessarily in utf-8. In my environment, the correct encoding is cp932 when the python output is piped. It appears to be a traditional problem on sys.stdout.encoding, which python "wisely" decides depending on the execution environment and the output stream.¹

I confirmed that the example above failed on 0.16.5 but not 0.16.4. I guess it's just because the recent changes (15edd33) have made it so that _get_poetry_virtualenv is always called. It should be an inapparent problem in older versions too.

As for the solution, it would be a way, setting PYTHONIOENCODING to utf-8 before invoking poetry env info -p (67707aa). I'm not confident about how decent this workaround is, but anyway that resolved my issue.

Footnotes

1. For your information, the related stackoverflow posts would be Why do the sys.stdout.encoding differ when output is piped (in Python2.x)? or Setting the correct encoding when piping stdout in Python ↩

[nat-n]

Hi @privet-kitty,

Thanks for reporting this issue and suggesting a solution. I plan to include your fix in an upcoming release.

I guess that environment variable should also be set/respected when running tasks that capture output in the executor like this. Would you foresee any downside with that?

[nat-n]

@privet-kitty This feature is now available as part of v0.17.0.

[privet-kitty]

@nat-n Thank you for merging!

I guess that environment variable should also be set/respected when running tasks that capture output in the executor like this. Would you foresee any downside with that?

Actually I don't quite understand the necessity that you care PYTHONIOENCODING either in save_task_output or exec_via_subproc, because users can easily control the I/O of their tasks, and more importantly they know that they can without being taught (unlike poetry env info -p problem) . However, I find no obvious downside and I'm not against that.

[nat-n]

@privet-kitty yes, users can control it (still), though my intention is that this way they're less likely to have to think about it. Thanks for the feedback. :)