Unicode: use "target encoding" while transcoding for output

[penguinolog]

• apply Canvas.content method API normalization and annotation
• • Overloaded methods should accept arguments as base class method
• • Overloaded methods can use own defaults

Checklist

• I've ensured that similar functionality has not already been implemented
• I've ensured that similar functionality has not earlier been proposed and declined
• I've branched off the master or python-dual-support branch
• I've merged fresh upstream into my branch recently
• I've ran tox successfully in local environment
• I've included docstrings and/or documentation and/or examples for my code (if this is a new feature)

[coveralls]

Pull Request Test Coverage Report for Build 7813470994

• -5 of 14 (64.29%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.002%) to 70.89%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
urwid/display/curses.py	1	2	50.0%
urwid/display/_raw_display_base.py	0	4	0.0%

Totals
Change from base Build 7813019446:	0.002%
Covered Lines:	8774
Relevant Lines:	12462

---

💛 - Coveralls

[penguinolog]

this buggy line screams: we should review Canvas logic and ultimately switch to the Unicode strings in canvas contents.
Currently we have standard chain for 99,9% of widgets str -> bytes -> str (terminal encoding independent).
During transcoding we're using "target encoding", but in fact it useful only in input processing (not everywhere: in windows and with curses we already can have decoded data).

[wardi]

Hmm, this code was originally working with bytes so that applications could be running with a non-utf-8 encoding without converting to Unicode and back (potentially losing data in the process) but I guess everything in raw display uses Unicode now?

[penguinolog]

$ python
Python 3.11.4 (main, Dec  7 2023, 15:43:41) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
>>> sys.stdout.write("test str")
test str8
>>> sys.stdout.write(b"test bytes")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: write() argument must be str, not bytes
>>> sys.stdin
<_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>

IO is now using strings, so we have to use decode/encode to operate bytes.
Curses is using strings and can return keystrokes as strings and decoded mouse input.
Windows raw display has access natively to str and bytes input from the same object

[penguinolog]

Hmm, this code was originally working with bytes so that applications could be running with a non-utf-8 encoding without converting to Unicode and back (potentially losing data in the process) but I guess everything in raw display uses Unicode now?

Output in unicode was as minimum in version 2.0 (and even ignored target encoding): https://github.com/urwid/urwid/blob/release-2.0.0/urwid/raw_display.py#L848

[penguinolog]

@wardi in python 3 we have STDOUT, which is always "Unicode". And it's not always buffered (so we cannot use buffer for output).