Bug in UTF-8 decoding while reading only part of a (text) blob

[Indy2222]

• Package Name: azure-storage-blob
• Package Version: 12.19.0
• Operating System: Linux
• Python Version: 3.10

Describe the bug

When downloading only part of a UTF-8 decoded text blob, the SDK sometimes raises UnicodeDecodeError with error message 'utf-8' codec can't decode byte 0xe2 in position ...: unexpected end of data. See the code example below.

downloader = (
    client
    .get_blob_client(some_txt_blob)
    .download_blob(timeout=30, encoding='utf-8-sig')
)
# The line below sometimes raises
downloader.read(10 * 1024 * 1024)

I think that this line

azure-sdk-for-python/sdk/storage/azure-storage-blob/azure/storage/blob/_download.py

Line 643 in 5ee1d7a

return data.decode(self._encoding)

assumes that the accumulated data (variable data) always starts and ends on a UTF-8 character boundary. Depending on the blob content (e.g. presence of non-ASCII characters) and the value passed to the read() method, the assumption may be untrue.

Expected behavior

Ideally, the API should accept the number of characters to download, in the case of text decoding, instead of the number of bytes. Alternatively, it could fetch a few extra bytes if needed. At a minimum, reading a specific number of bytes (or characters) combined with text decoding should not be allowed; for example, an explicit exception should be raised, and this behavior should be documented, or the API should not even allow it.

Additional context

We need to gradually download and parse a large CSV file/blob. Loading the whole file into memory and parsing it all at once is not feasible.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[l0lawrence]

Hi @Indy2222 thanks for the feedback, we will investigate and get back to you asap.

[jalauzon-msft]

Hi @Indy2222 Martin, thanks for bringing this to our attention and for you investigation. This does indeed seem to be an issue we overlooked when adding the read method. I think you are right that ultimately arbitrary length reads should not really be supported if encoding is specified however the tricky part is we can't just remove the support as that would be a breaking change. Maybe we can read a few extra bytes or something but that could get pretty complex. It may be that the best we can do now is generate a loud warning (maybe even deprecation warning) if we see people trying to use read(size=x) with encoding specified.

We will chat internally about how to best handle this and work on making some change. Thanks again.