Use non-BOM encodings #2370
Use non-BOM encodings #2370
Conversation
filmor
force-pushed
the
fix-bom-strings
branch
2 times, most recently
from
49db3bf
to
07f65c7
Compare
The documentation of the used `PyUnicode_DecodeUTF16` states that not passing `*byteorder` or passing a 0 results in the first two bytes, if they are the BOM (U+FEFF, zero-width no-break space), to be interpreted and skipped, which is incorrect when we convert a known "non BOM" string, which all strings from C# are.
|
@filmor can you ELI5? For someone not familiar with intricacies of BOM, but aware of byte order issues. My biggest question is if this change has any potential to introduce bugs to handling strings that actually have BOM? E.g. imagine a scenario when someone serialized and persisted something with BOM using 3.0.3, but after this change in 3.0.4 if they read it back BOM will be in their string data. |
|
It's the other way round. Strings that are being passed between Python and .NET are UTF16 in the respective native byte order (usually LE), without a BOM. The functions that we were using for the conversions (in particular |
Use non-BOM encodings for both C#->Python and Python->C#, as the byteorder is always the native one and the BOM is neither never or always used.
Fixes #2369.