Problem identified in 2014 with editor arduino/Arduino#2430, but May 2015 comment contains reference to limitation with serial monitor encoding support

I did some research on UTF-8 behavior today. I found this code in arduino-core/src/processing/app/Serial.java which does the conversion from the incoming bytes to the strings displayed by the UI:

  public synchronized void serialEvent(SerialPortEvent serialEvent) {
    if (serialEvent.isRXCHAR()) {
      try {
        byte[] buf = port.readBytes(serialEvent.getEventValue());
        if (buf.length > 0) {
          String msg = new String(buf);
          char[] chars = msg.toCharArray();
          message(chars, chars.length);
        }
      } catch (SerialPortException e) {
        errorMessage("serialEvent", e);
      }
    }
  }

The String-constructor used decodes the incoming bytes according to the particular platform's default charset, its documentation says.

I think the standard should be to always use UTF-8. So a sketch developed and programmed on a Mac works with a Linux or Windows box and vice versa. If we later want to add HEX display, as proposed by #1727, here would be one place to do it.

There is a catch here: In our context, there is no guarantee whatsoever that the bytes that come in do us the favor to split neatly at character boundaries. It is quite feasible they include only the first byte of a character that is encoded into several bytes, and the other bytes come later, with the next call.

In this case, the String-constructor is documented to exhibit undocumented behavior :-) .

In actual tests, I indeed see erratic output whenever I use non-ASCII characters in my sketches. Sometimes it works, sometimes it doesn't. (This is Linux, and I'm almost certain the platform encoding is UTF-8.)

The String-constructor documentation advises to use a CharsetDecoder instead.

I think that's good advice. This would give control over the encoding used. Clean UTF-8 decoding even in the split character case is a feature included in CharsetDecoder.

Hi,
I also recently stumbled across that limitation. The Serial Monitor now supports reading in UTF-8, so writing should also default to UTF-8.