Reading a string prefixed with a length in code points, not bytes

[rgov]

Suppose I have the string "π" which is encoded using UTF-8 as the byte sequence [0xCF, 0x80]. This is serialized as a length-prefixed string, but the length is given in terms of characters or code points, rather than bytes. So instead of a length of 2 bytes, it is written with a length of 1 character.

How is this expressed in construct?

[arekbulski]

I think it is impossible to know from chat length.

[rgov]

I worked out that you can modify bytes.decode() to truncate on the first decoding error, rather than raise an exception:

# Register a new decoding error handler for truncating the input at
# the first error, rather than throwing an exception.
def truncate_on_error(exc):
    if isinstance(exc, UnicodeDecodeError):
        return '', len(exc.object)
    raise exc

codecs.register_error('truncate', truncate_on_error)

# Example
b'hello world\x80asdf'.decode('utf-8', 'truncate') == 'hello world'

Then if you could greedily read bytes until you reach the desired string length, or exhaust the stream, you could achieve the desired behavior.

[rgov]

Anyway, in a way isn't this exactly what open('foo.txt', 'r', encoding='utf-8').read(3) does?

Edit: Answer: surprisingly no, this doesn't work.

[rgov]

Here's my attempt. It reads one byte at a time until it can decode the string.

class CharLenString(construct.Construct):
    '''
    Like a length-prefixed PascalString, but the length prefix is the number
    of characters, not bytes.

    This is a challenge with encodings that can use multiple code units to
    represent a single character, because we don't know how many bytes to read
    ahead of time.
    '''
    def __init__(self, lengthfield, encoding):
        super().__init__()
        self.lengthfield = lengthfield
        self.encoding = encoding

    def _parse(self, stream, context, path):
        # Parse length field
        length = self.lengthfield._parsereport(stream, context, path)

        # Read a byte at a time until we can decode enough characters
        data = bytearray()
        string = ''
        while len(string) < length:
            data.extend(construct.stream_read(stream, 1, path))
            try:
                string += data.decode(self.encoding)
                data.clear()
            except UnicodeDecodeError:
                pass

        # Confirm that we got the right length after all
        if len(string) != length:
            raise construct.StringError('Invalid string')

        return string

    def _build(self, obj, stream, context, path):
        data = obj.encode(self.encoding)

        # Write the length field then the encoded data
        self.lengthfield._build(len(data), stream, context, path)
        construct.stream_write(stream, data, len(data), path)

        # Return the string that appears in the context dict
        return obj

    def _sizeof(self, context, path):
        raise construct.SizeofError()

Test case:

d = Sequence(CharLenString(Byte, 'utf-8'), Byte)
o = d.parse( b'\x04' + '美味しい'.encode('utf-8') + b'\x41')
print('Parsed:', o)
s = d.build(o)
print('Serialized:', s)

I'm not sure this approach works across all encodings, though.