Ensure UTF-8 vs ASCII is used where appropriate

[Poikilos]

Sometimes UTF-8 or ASCII is specified (default is utf-8) for encode or decode. It isn't clear which is correct for the protocol(s). tcpsocket.py has a mixture of both.

:edit:

• fix instances of "ascii" or "ASCII" for encode/decode
• fix low-level encode/decode (chr or ord calls)
• Add Cyrillic tests for SNIP and MemoryService byte list to str methods

[Poikilos]

I am asking the OpenLCB team about this now. I am also reading that the first 128 characters are the same, and I'm not sure if extended characters would ever occur. Greek or slavic node names??

[bobjacobsen]

It depends on where the encode or decode is being done. For CAN-format transfer over CAN proper, or over TCP, those frames are ASCII and ASCII is appropriate for their encode/decoder (though UTF-8 probably wouldn't hurt). Display of CD, SNIP and CDI content should be encoded/decoded as UTF-8.

[bobjacobsen]

Which 128 characters are you referring to? The first part of the configuration memory? That might contain UTF-8. (Those strings should be defined by byte length, not character count, because UTF-8 has multibyte characters)

[Poikilos]

I mean I read in learning materials that 0 to 127 in the ASCII character set are the same as 0 to 127 in the UTF-8 character set, but after that they differ. That's why I alluded to Cyrillic (Greek and slavic) maybe being the only time we would/could see or test this.

[Poikilos]

@bakerstu says that they use tinyutf8 for everything in C++ to ensure compatibility with European languages.

@balazsracz says the LCC standard says to use UTF-8 which I didn't recall

[Poikilos]

I can work on this

[bobjacobsen]

I can work on this

Great. Thank you!

Looks like there are a few places that need to be checked:

% grep -r encode . | grep utf

% grep -r encode . | grep ascii
./openlcb/canbus/seriallink.py:        msg = string.encode('ascii')
./openlcb/canbus/tcpsocket.py:        msg = string.encode('ascii')
./openlcb/snip.py:        mfgArray = '{:.40}'.format(self.manufacturerName).encode('ascii')
./openlcb/snip.py:        mdlArray = '{:.40}'.format(self.modelName).encode('ascii')
./openlcb/snip.py:        hdvArray = '{:.20}'.format(self.hardwareVersion).encode('ascii')
./openlcb/snip.py:        sdvArray = '{:.20}'.format(self.softwareVersion).encode('ascii')
./openlcb/snip.py:        upnArray = '{:.62}'.format(self.userProvidedNodeName).encode('ascii')
./openlcb/snip.py:            '{:.63}'.format(self.userProvidedDescription).encode('ascii')
./openlcb/memoryservice.py:        strToUInt8 = value.encode('ascii')

% grep -r decode . | grep utf  
./openlcb/canbus/seriallink.py:        return (b''.join(chunks)).decode("utf-8")
./openlcb/canbus/tcpsocket.py:        return b''.join(chunks).decode("utf-8")

% grep -r decode . | grep ascii

I think seriallink.py and tcpsocket.py are OK (though they could be decoding with ascii without any issue).

The snip.py and memoryservice.py encodings don't look right. They should probably be changed to utf-8 and some tests made.

Bob

[Poikilos]

I think seriallink.py and tcpsocket.py are OK (though they could be decoding with ascii without any issue).

Using chr doesn't notice encoding, so casting int list to bytes or bytearray then using decode("utf-8") will be reliable. Coming in next commit.

• future idea: Consider using bytearray #58