[BUG] UnicodeEncodeError on Windows with ruler.

[tobiasraabe]

Describe the bug

The issue started appearing in my test suite, and I could boil it down to this reproducible example. I am using rich 12.5.1.

# Content of script.py.

from rich.console import Console


if __name__ == '__main__':
    console = Console()
    console.rule()  # or console.print("─")

# Content of run.py.

import subprocess

result = subprocess.run(("python", "rich_script.py"), check=True)

and then I am executing pytest run.py, which results in

Click to expand

================================================= test session starts =================================================
platform win32 -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: C:\Users\TobiasR\git\pytask, configfile: tox.ini
plugins: anyio-3.6.1, cov-3.0.0, forked-1.4.0, xdist-2.5.0
collected 0 items / 2 errors

======================================================= ERRORS ========================================================
_______________________________________________ ERROR collecting run.py _______________________________________________
run.py:3: in <module>
    result = subprocess.run(("python", "rich_script.py"), check=True)
C:\tools\miniconda3\envs\pytask\lib\subprocess.py:524: in run
    raise CalledProcessError(retcode, process.args,
E   subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
--------------------------------------------------- Captured stderr ---------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\TobiasR\git\pytask\rich_script.py", line 6, in <module>
    console.rule()
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1553, in rule
    self.print(rule)
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1646, in print
    with self:
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 848, in __exit__
    self._exit_buffer()
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 806, in _exit_buffer
    self._check_buffer()
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1999, in _check_buffer
    legacy_windows_render(
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_windows_renderer.py", line 17, in legacy_windows_render
    term.write_styled(text, style)
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 442, in write_styled
    self.write_text(text)
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 403, in write_text
    self.write(text)
  File "C:\tools\miniconda3\envs\pytask\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-78: character maps to <undefined>
_______________________________________________ ERROR collecting run.py _______________________________________________
run.py:3: in <module>
    result = subprocess.run(("python", "rich_script.py"), check=True)
C:\tools\miniconda3\envs\pytask\lib\subprocess.py:524: in run
    raise CalledProcessError(retcode, process.args,
E   subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
--------------------------------------------------- Captured stderr ---------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\TobiasR\git\pytask\rich_script.py", line 6, in <module>
    console.rule()
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1553, in rule
    self.print(rule)
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1646, in print
    with self:
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 848, in __exit__
    self._exit_buffer()
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 806, in _exit_buffer
    self._check_buffer()
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\console.py", line 1999, in _check_buffer
    legacy_windows_render(
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_windows_renderer.py", line 17, in legacy_windows_render
    term.write_styled(text, style)
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 442, in write_styled
    self.write_text(text)
  File "C:\tools\miniconda3\envs\pytask\lib\site-packages\rich\_win32_console.py", line 403, in write_text
    self.write(text)
  File "C:\tools\miniconda3\envs\pytask\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-78: character maps to <undefined>
=============================================== short test summary info ===============================================
ERROR run.py - subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
ERROR run.py - subprocess.CalledProcessError: Command '('python', 'rich_script.py')' returned non-zero exit status 1.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================== 2 errors in 0.72s ==================================================

The error does not occur when I

• switch to rich 12.4.4 or ...
• set PYTHONIOENCODING to utf-8 or ...
• execute python run.py

Platform

Click to expand

The issue occurred only on Windows and I am running Powershell Core embedded in Windows Terminal.

python -m rich.diagnose
pip freeze | grep rich

╭───────────────────────── <class 'rich.console.Console'> ─────────────────────────╮
│ A high level console interface.                                                  │
│                                                                                  │
│ ╭──────────────────────────────────────────────────────────────────────────────╮ │
│ │ <console width=120 ColorSystem.TRUECOLOR>                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                  │
│     color_system = 'truecolor'                                                   │
│         encoding = 'utf-8'                                                       │
│             file = <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> │
│           height = 30                                                            │
│    is_alt_screen = False                                                         │
│ is_dumb_terminal = False                                                         │
│   is_interactive = True                                                          │
│       is_jupyter = False                                                         │
│      is_terminal = True                                                          │
│   legacy_windows = False                                                         │
│         no_color = False                                                         │
│          options = ConsoleOptions(                                               │
│                        size=ConsoleDimensions(width=120, height=30),             │
│                        legacy_windows=False,                                     │
│                        min_width=1,                                              │
│                        max_width=120,                                            │
│                        is_terminal=True,                                         │
│                        encoding='utf-8',                                         │
│                        max_height=30,                                            │
│                        justify=None,                                             │
│                        overflow=None,                                            │
│                        no_wrap=False,                                            │
│                        highlight=None,                                           │
│                        markup=None,                                              │
│                        height=None                                               │
│                    )                                                             │
│            quiet = False                                                         │
│           record = False                                                         │
│         safe_box = True                                                          │
│             size = ConsoleDimensions(width=120, height=30)                       │
│        soft_wrap = False                                                         │
│           stderr = False                                                         │
│            style = None                                                          │
│         tab_size = 8                                                             │
│            width = 120                                                           │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭── <class 'rich._windows.WindowsConsoleFeatures'> ───╮
│ Windows features available.                         │
│                                                     │
│ ╭─────────────────────────────────────────────────╮ │
│ │ WindowsConsoleFeatures(vt=True, truecolor=True) │ │
│ ╰─────────────────────────────────────────────────╯ │
│                                                     │
│ truecolor = True                                    │
│        vt = True                                    │
╰─────────────────────────────────────────────────────╯
╭────── Environment Variables ───────╮
│ {                                  │
│     'TERM': None,                  │
│     'COLORTERM': None,             │
│     'CLICOLOR': None,              │
│     'NO_COLOR': None,              │
│     'TERM_PROGRAM': None,          │
│     'COLUMNS': None,               │
│     'LINES': None,                 │
│     'JUPYTER_COLUMNS': None,       │
│     'JUPYTER_LINES': None,         │
│     'JPY_PARENT_PID': None,        │
│     'VSCODE_VERBOSE_LOGGING': None │
│ }                                  │
╰────────────────────────────────────╯
platform="Windows"

[willmcgugan]

Not the first time we've been bitten by Windows CP-1252 encoding.

The solution may be to force utf8

[darrenburns]

I can't reproduce this. Running the same Python version, Rich version, tried legacy/non-legacy terminals, tried forcing the encoding to cp1252 and utf-8, tried with an without running in a subprocess. Every time the rule renders without an error. I'm not sure where else the difference could lie 🤔

It's interesting that you've supplied the example as a subprocess - is that because you can only reproduce the issue when you run it in a subprocess?

I noticed in the "Platform" information you shared, legacy_windows = False, yet the code path which raises the error implies that legacy_windows = True. I wonder if for some reason, the console reports legacy Windows in the subprocess but not in the parent process? Regardless, I can't reproduce even when I force legacy and cp1252 on my end, so I'm stumped at the moment!

Also, comparing the version that works with you with the latest version, I can't see any relevant (encoding-related) changes at all: v12.4.4...v12.5.1

[tobiasraabe]

Yes, this error only occurs when I am using pytest to execute the subprocess script with pytest run.py. python run.py works. So, for me, it seemed like pytest influences which console settings are detected in the subprocess.

[ubaumann]

I also had it after an update in my windows tests in github
https://github.com/InfrastructureAsCode-ch/nornir_rich/runs/7483512482?check_suite_focus=true

I am using PYTHONIOENCODING: utf-8 at the moment and it works fine but for me on windows pytest . worked well. So I have the problam only in the workflow on GitHub

[willmcgugan]

For tests, you probably want to set legacy_windows to True or False, so you aren't at the mercy of the environment. Ditto with the other parameters that auto-detect.

[github-actions]

Did I solve your problem?

Why not buy the devs a coffee to say thanks?

[mattip]

I am seeing this in a azure pipelines run of a conda-forge feedstock. Should I open another issue? Relevant bits:

Python 3.8, rich 12.6.0-pyhd8ed1ab_0 from conda-forge.

ConsoleOptions

From this log

| | render_options = ConsoleOptions(                                        | |
| |                      size=ConsoleDimensions(width=79, height=25),       | |
| |                      legacy_windows=True,                               | |
| |                      min_width=1,                                       | |
| |                      max_width=79,                                      | |
| |                      is_terminal=False,                                 | |
| |                      encoding='cp1252',                                 | |
| |                      max_height=25,                                     | |
| |                      justify=None,                                      | |
| |                      overflow=None,                                     | |
| |                      no_wrap=None,                                      | |
| |                      highlight=None,                                    | |
| |                      markup=None,                                       | |
| |                      height=None                                        | |
| |                  )                                                      | |

when rendering this string

text = '.                                                  \u2554\u2584\u2593\u2593\u2593\u2593\u2584'

[ManchurianMan]

This is still an issue. In CP1252 encoding (which is what the :! operator in nvim uses) console.print(Rule()) and console.rule() throw an error when no title is provided. They work fine with a title provided.