Unable to print unicode characters when running repl with mpremote

[XuanTongYao]

Port, board and/or hardware

RP2 Pico

MicroPython version

MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico with RP2040

Reproduction

1. Create a file called tt.py
2. Write the following contents

print("你好")

3. Run this command in the same directory:mpremote mount .
4. import tt

Expected behaviour

Expected to print "你好".

Observed behaviour

PS D:\PROJECTS\Github\MicroPython-uFont> mpremote mount .
Local directory . is mounted at /remote
Connected to MicroPython at COM6
Use Ctrl-] or Ctrl-x to exit this shell
>
MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import tt
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\YHW\AppData\Local\Programs\Python\Python312\Scripts\mpremote.exe\__main__.py", line 7, in <module>
  File "C:\Users\YHW\AppData\Local\Programs\Python\Python312\Lib\site-packages\mpremote\main.py", line 536, in main
    do_repl(state, argparse_repl().parse_args([]))
  File "C:\Users\YHW\AppData\Local\Programs\Python\Python312\Lib\site-packages\mpremote\repl.py", line 89, in do_repl
    do_repl_main_loop(
  File "C:\Users\YHW\AppData\Local\Programs\Python\Python312\Lib\site-packages\mpremote\repl.py", line 54, in do_repl_main_loop  
    console_out_write(console_data_out)
  File "C:\Users\YHW\AppData\Local\Programs\Python\Python312\Lib\site-packages\mpremote\repl.py", line 83, in console_out_write  
    console.write(b)
  File "C:\Users\YHW\AppData\Local\Programs\Python\Python312\Lib\site-packages\mpremote\console.py", line 119, in write
    buf = buf.decode() if isinstance(buf, bytes) else buf
          ^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data

Additional Information

Obviously this error occurs on the PC, so I guess there must be something wrong with the mpremote source code. The error is also triggered when printing other unicode characters, such as Japanese, emoji, etc.

As far as I know, many people use mpremote mount . to quickly demo their code, and I'd like to, but something went wrong.

Is there any way to avoid this problem?
I could convert unicode characters to hexadecimal and print them, but that's pointless.
Printing unicode characters works perfectly fine with tools like thonny.

Code of Conduct

Yes, I agree

[massimosala]

Hi @XuanTongYao

Are you using a windows or linux pc ?

Here I document some caveats regarding the Windows "dos" console and utf8 support.

Windows registry: at this location there is the list of codepages and fonts:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Console\TrueTypeFont

---

European languages

• Check into the registry there is a font for the codepage 00, like Consolas

• Windows 7: before using mpremote on the command line type this command

CHCP 65001

to switch the console to utf8.

---

Japanese or other far east languages

• Configure the registry with a font for the codepage 936 (for example)

• Windows 10: before using mpremote on the command line type this command

CHCP 936

to switch the console to code page 936
Simplified Chinese

(at the moment I haven't a windows 7 box to try this)

---

If this doesn't work... it means mpremote should be modified. This afternoon I spent one hour modifying its source code.
I tried different tweaks; I need to talk with the mpremote programmer to share the ideas.

[XuanTongYao]

@massimosala
Thanks for your reply.

I was using vscode on windows 11 when I encountered this problem.

For the help you provided, I made some attempts such as using cmd or powershell directly without vscode, switching the code page to 65001, changing fonts, etc.

Unfortunately, it still doesn't work on my pc.

---

I also tried modifying the source code, and very strangely, if you add print directly to the mpremote source code to output unicode characters, it works fine.

I noticed that n is sometimes 2 when receiving unicode characters from the slave in repl.py, line 35.

This means that if the print is used in the slave's code to output a character that is not ASCII but is less than or equal to 0xFFFF in UTF-8 encoding, it will likely work fine.

PS D:\PROJECTS\Github\MicroPython-uFont> mpremote mount .
Local directory . is mounted at /remote
Connected to MicroPython at COM6
Use Ctrl-] or Ctrl-x to exit this shell
>
MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import tt
Здравствуй, мир.
>>>

I'm guessing that perhaps there is a problem with the process of using the serial port to receive data from the slave and process it.

[XuanTongYao]

Hi @massimosala
I would like to correct my mistake, in line 34 of repl.py, the value of n is always 2 when receiving data from the print().

From my understanding of this code, n represents the byte length of the data.

        try:
            n = state.transport.serial.inWaiting()
        except OSError as er:
            if er.args[0] == 5:  # IO error, device disappeared
                print("device disconnected")
                break

A single byte character with a length less than 2 is never truncated, so it can be displayed normally.

If it's a character that is 2 bytes long, you have to make sure that its start byte is in exactly an even position, i.e., aligned to 2, for it to display properly.

But n never seems to be greater than 2, so unicode characters of length 3 bytes are truncated and can't be displayed.

Modified code

On line 41 and following of repl.py.

I fixed it in the following somewhat silly way, and surprisingly it prints unicode characters correctly.

        if n > 0:
            dev_data_in = state.transport.serial.read(n)
            # Modified code
            if n == 2:
                # Read the truncated data
                if dev_data_in[1] & 0b11111000 == 0b11110000:
                    dev_data_in += state.transport.serial.read(3)
                elif dev_data_in[0] & 0b11111000 == 0b11110000:
                    dev_data_in += state.transport.serial.read(2)
                elif dev_data_in[1] & 0b11110000 == 0b11100000:
                    dev_data_in += state.transport.serial.read(2)
                elif dev_data_in[0] & 0b11110000 == 0b11100000:
                    dev_data_in += state.transport.serial.read(1)
                elif dev_data_in[1] & 0b11100000 == 0b11000000:
                    dev_data_in += state.transport.serial.read(1)
            # Modified code
                
            if dev_data_in is not None:
                if escape_non_printable:
                    # Pass data through to the console, with escaping of non-printables.
                    console_data_out = bytearray()
                    for c in dev_data_in:
                        if c in (8, 9, 10, 13, 27) or 32 <= c <= 126:
                            console_data_out.append(c)
                        else:
                            console_data_out.extend(b"[%02x]" % c)
                else:
                    console_data_out = dev_data_in
                console_out_write(console_data_out)

Console log

PS D:\PROJECTS\Github\MicroPython-uFont> mpremote mount .
Local directory . is mounted at /remote
Connected to MicroPython at COM6
Use Ctrl-] or Ctrl-x to exit this shell
>
MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico with RP2040
Type "help()" for more information.
>>> import tt
Hello, world!
Здравствуй, мир
你好,世界！
>>>

[massimosala]

Wow! A solution is a solution... but what a mess :-)

I review my changes and try to find the minimal one to resolve this.
I will let you know, here.
Ciao

[XuanTongYao]

I think I've found the source of the problem.

There are 3 files involved in this problem: repl.py, transport_serial.py, and console.py.

The reason for this bug

First, let's explain why the value of n is always 2 when receiving data from print().
In fact, it's not only when receiving data from print(), but also when displaying text in the console.

In the try block, the function corresponding to inWaiting() is at line 1206 of transport_serial.py, which checks if there is data input to state.transport.serial, and if there is, appends one byte of data to state.transport.serial.buf and returns the length of buf.

Then in the following if n > 0:, read(n) reads n bytes of data from the buf and moves them out of the buf.

But when transferring ASCII characters, the value of n should be 1, not 2.

The problem lies in the first statement in the while loop, console_in.waitchar(state.transport.serial), which also executes inWaiting().
It updates the buf without reading it with the read(), which results in 1 byte of data already in the buf before the inWaiting() is executed in the try block, and if there is still data waiting to be entered into state.transport.serial, the value of n will be 2.

Personally, I think that in the strict sense, this code has been working fine with bugs. (Only ASCII)

# repl.py , line 6

def do_repl_main_loop(
    state, console_in, console_out_write, *, escape_non_printable, code_to_inject, file_to_inject
):
    while True:
        console_in.waitchar(state.transport.serial)
        c = console_in.readchar()
        if c:
            if c in (b"\x1d", b"\x18"):  # ctrl-] or ctrl-x, quit
                break
            elif c == b"\x04":  # ctrl-D
                # special handling needed for ctrl-D if filesystem is mounted
                state.transport.write_ctrl_d(console_out_write)
            elif c == b"\x0a" and code_to_inject is not None:  # ctrl-j, inject code
                state.transport.serial.write(code_to_inject)
            elif c == b"\x0b" and file_to_inject is not None:  # ctrl-k, inject script
                console_out_write(bytes("Injecting %s\r\n" % file_to_inject, "utf8"))
                state.transport.enter_raw_repl(soft_reset=False)
                with open(file_to_inject, "rb") as f:
                    pyfile = f.read()
                try:
                    state.transport.exec_raw_no_follow(pyfile)
                except TransportError as er:
                    console_out_write(b"Error:\r\n")
                    console_out_write(er)
                state.transport.exit_raw_repl()
            else:
                state.transport.serial.write(c)

        try:
            n = state.transport.serial.inWaiting()
        except OSError as er:
            if er.args[0] == 5:  # IO error, device disappeared
                print("device disconnected")
                break

        if n > 0:
            dev_data_in = state.transport.serial.read(n)
            if dev_data_in is not None:
                if escape_non_printable:
                    # Pass data through to the console, with escaping of non-printables.
                    console_data_out = bytearray()
                    for c in dev_data_in:
                        if c in (8, 9, 10, 13, 27) or 32 <= c <= 126:
                            console_data_out.append(c)
                        else:
                            console_data_out.extend(b"[%02x]" % c)
                else:
                    console_data_out = dev_data_in
                console_out_write(console_data_out)

# transport_serial.py , line 1174 in class SerialIntercept

    def _check_input(self, blocking):
        if blocking or self.orig_serial.inWaiting() > 0:
            c = self.orig_serial.read(1)
            if c == b"\x18":
                # a special command
                c = self.orig_serial.read(1)[0]
                self.orig_serial.write(b"\x18")  # Acknowledge command
                PyboardCommand.cmd_table[c](self.cmd)
            elif not VT_ENABLED and c == b"\x1b":
                # ESC code, ignore these on windows
                esctype = self.orig_serial.read(1)
                if esctype == b"[":  # CSI
                    while not (0x40 < self.orig_serial.read(1)[0] < 0x7E):
                        # Looking for "final byte" of escape sequence
                        pass
            else:
                self.buf += c

    def inWaiting(self):
        self._check_input(False)
        return len(self.buf)

    def read(self, n):
        while len(self.buf) < n:
            self._check_input(True)
        out = self.buf[:n]
        self.buf = self.buf[n:]
        return out

# console.py , line 98
    def waitchar(self, pyb_serial):
        while not (self.inWaiting() or pyb_serial.inWaiting()):
            time.sleep(0.01)

---

How to fix it.

Since I don't know much about the workings behind transport_serial.py and console.py, the code I fixed was just adding new logic rather than modifying it.

The main thing is to check whether the buf is modified after console_in.waitchar(state.transport.serial) is executed, if it is modified, then determine whether the current buf is ASCII or UTF-8 characters, if not, then the same logic as before.

def do_repl_main_loop(
    state, console_in, console_out_write, *, escape_non_printable, code_to_inject, file_to_inject
):
    while True:
        buf_len = len(state.transport.serial.buf)
        console_in.waitchar(state.transport.serial)
        c = console_in.readchar()
        if c:
            if c in (b"\x1d", b"\x18"):  # ctrl-] or ctrl-x, quit
                break
            elif c == b"\x04":  # ctrl-D
                # special handling needed for ctrl-D if filesystem is mounted
                state.transport.write_ctrl_d(console_out_write)
            elif c == b"\x0a" and code_to_inject is not None:  # ctrl-j, inject code
                state.transport.serial.write(code_to_inject)
            elif c == b"\x0b" and file_to_inject is not None:  # ctrl-k, inject script
                console_out_write(bytes("Injecting %s\r\n" % file_to_inject, "utf8"))
                state.transport.enter_raw_repl(soft_reset=False)
                with open(file_to_inject, "rb") as f:
                    pyfile = f.read()
                try:
                    state.transport.exec_raw_no_follow(pyfile)
                except TransportError as er:
                    console_out_write(b"Error:\r\n")
                    console_out_write(er)
                state.transport.exit_raw_repl()
            else:
                state.transport.serial.write(c)

        try:
            if buf_len != len(state.transport.serial.buf):
                # If the buf is modified, read a byte from it.
                first_byte = state.transport.serial.read(1)
                if first_byte.isascii():
                    n = 0
                elif first_byte[0] & 0b1110_0000 == 0b1100_0000:
                    # UTF-8 2 bytes
                    n = 1
                elif first_byte[0] & 0b1111_0000 == 0b1110_0000:
                    # UTF-8 3 bytes
                    n = 2
                elif first_byte[0] & 0b1111_1000 == 0b1111_0000:
                    # UTF-8 4 bytes
                    n = 3
            else:
                first_byte = b""
                n = state.transport.serial.inWaiting()
        except OSError as er:
            if er.args[0] == 5:  # IO error, device disappeared
                print("device disconnected")
                break

        if n > 0 or first_byte != b"":
            dev_data_in = first_byte+state.transport.serial.read(n)
            if dev_data_in is not None:
                if escape_non_printable:
                    # Pass data through to the console, with escaping of non-printables.
                    console_data_out = bytearray()
                    for c in dev_data_in:
                        if c in (8, 9, 10, 13, 27) or 32 <= c <= 126:
                            console_data_out.append(c)
                        else:
                            console_data_out.extend(b"[%02x]" % c)
                else:
                    console_data_out = dev_data_in
                console_out_write(console_data_out)

[massimosala]

Hi

I found the minimal changes to support multi-byte characters and, as command-line options, all serial ports.
Apply the attached patches to the original source files.
Try it and let me know.

If it is OK, I will contact the developer of mpremote
mpremote_diff.zip

[XuanTongYao]

Hi @massimosala
Your code is working perfectly.

There's a little problem with the diff file, it has a CRLF end-of-line sequence, the original repository file has a LF end-of-line sequence.

[dpgeorge]

It looks like this only happens on Windows with mpremote, due to the fact that ConsoleWindows.write is trying to convert bytes to str where the bytes has a partial unicode/utf-8 character.

Probably the best fix is to use sys.stdout.buffer.write instead of sys.stdout.write and write the bytes directly. Then no conversion to str is needed.