Python 2 does not keep encoding when converting .ino

[mcspr]

Configuration

Operating system:
Ubuntu 19.10

PlatformIO Version (platformio --version):
PlatformIO, version 4.2.2a1

Description of problem

pio run with python3 and python2 produce different binaries when source code contains strings with non-ASCII symbols.

Steps to Reproduce

1. Use main.ino in the project.
2. Add const char celsius[] = "-----> °C"; and use it in the .ino file
3. Check that the main.ino is stored in UTF-8 encoding
4. Build with PIO installed in Python 2 environment

Actual Results

Checking that encoding is correct:

$ file src/main.ino
src/main.ino: C source, UTF-8 Unicode text

Log from pio run:

Converting main.ino
Unicode decode error has occurred, please remove invalid (non-ASCII or non-UTF8) characters from /home/builder/py2-test/src/main.ino.cpp file

Afterwards, checking firmware.elf. celcius string is stored as b0 43, which is invalid utf-8 sequence:

$ xxd .pio/build/d1_mini/firmware.elf | grep -- '----->'
00001610: 2d2d 2d2d 2d3e 20b0 4300 0d0a 006e 616e  -----> .C....nan
                           ^^ ^^

This causes issues when strings are used as-is in, for example, WebSocket communication.
See xoseperez/espurna#2163

Expected Results

Same version of PIO, but running with python3:

$ xxd .pio/build/d1_mini/firmware.elf | grep -- '----->'
00001610: 2d2d 2d2d 2d3e 20c2 b043 000d 0a00 6e61  -----> ..C....na
                           ^^ ^^^^

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:d1_mini]
platform = espressif8266
board = d1_mini
framework = arduino
src_build_flags = -g

Source file to reproduce issue:

(main.ino)
#include <Arduino.h>

const char celsius[] = "-----> °C";

void setup() {
        Serial.begin(115200);
        Serial.println(celsius);
}

void loop() {}

Additional info

Using PIO installed with Python 3 solves the issue.

Using .cpp or .h to store the string is a workaround. Raw bytes instead of strings is also a possible workaround.

[ivankravets]

Thanks for the report! Please re-test with pio upgrade --dev.

[mcspr]

Can confirm that now it works with python2. Thank you!