Support unicode characters longer than a byte

[microbit-carlos]

At the moment the String.charCodeAt() method is used to get the user script character codes, and it's return value is stored in a Uint8Array.
If the value returned is larger than a byte, as it would happen with some UTF-8 characters, then some of that information is lost and the wrong character is encoded into the hex file.

PythonEditor/python-main.js

Lines 84 to 92 in 3d30f7f

	// add header, pad to multiple of 16 bytes
	data = new Uint8Array(4 + script.length + (16 - (4 + script.length) % 16));
	data[0] = 77; // 'M'
	data[1] = 80; // 'P'
	data[2] = script.length & 0xff;
	data[3] = (script.length >> 8) & 0xff;
	for (var i = 0; i < script.length; ++i) {
	data[4 + i] = script.charCodeAt(i);
	}

This is easy to reproduce, simply create a hex file with a UTF-8 character larger than a byte, download the hex, and load it back into the editor.

# UFT-8 character longer than a byte: Σ

Becomes:

# UFT-8 character longer than a byte: £

As the chracter 0x03A3 (Σ) has been encoded as 0xA3 (£)

[PandaWhoCodes]

I would love to see a work around for this without the memory contraints bring increased

[microbit-carlos]

Unfortunately I don't think that would be possible, if a unicode character takes more than a byte then there really isn't much we can do there. As it currently stands right now each character is only storing is least significant byte and discarding the rest, and because of this data loss we are saving the wrong characters, and that is a worse outcome than using more flash space.

[microbit-carlos]

UTF-8 characters now work in the Editor since the filesystem was merged to the beta editor.