PYTHON-3048 Fixed issue with invalid UTF-8 string.

[bcwarner]

I think there was a missing switch case in isLegalUTF8 for characters beginning with 0xED, based on what I saw in this implementation. There are some other apparent differences, but I think for those other cases, they probably have the same behavior already.

Results:

In [1]: from bson import *

In [2]: encode({"a":Regex(b'\xed\xbc\xad','')})
---------------------------------------------------------------------------
InvalidStringData                         Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 encode({"a":Regex(b'\xed\xbc\xad','')})

File ~/opt/anaconda3/lib/python3.9/site-packages/pymongo-4.2.0.dev2-py3.9-macosx-10.9-x86_64.egg/bson/__init__.py:947, in encode(document, check_keys, codec_options)
    944 if not isinstance(codec_options, CodecOptions):
    945     raise _CODEC_OPTIONS_TYPE_ERROR
--> 947 return _dict_to_bson(document, check_keys, codec_options)

InvalidStringData: regex patterns must be valid UTF-8

[ShaneHarvey]

Nice find, however it might be a better idea to remove the encoding_helpers.c file entirely and rely on Python's builtin UTF-8 decoding behavior instead. It looks like we only use this logic when encoding regex fields.

[blink1073]

This line can be removed entirely

[ShaneHarvey]

As Bernie mentioned above we need to keep the NULL byte checking behavior. For BSON keys we perform the same check like this:
https://github.com/mongodb/mongo-python-driver/blob/be3008aa11/bson/_cbsonmodule.c#L1230-L1239

[ShaneHarvey]

LGTM!

[ShaneHarvey]

It would be interesting to see if there's any performance cost of using PyUnicode_DecodeUTF8 instead of the encoding helpers. My guess is the answer is no except docs with many regex fields ({...'r1': Regex(...), ..., 'r1000': Regex(...)}) or docs with large regex fields (eg a 1MB regex: {'r': Regex(b'1'*1024*1024)}). Could you spend some time investigating using timeit and post your findings?

[bcwarner]

We found that using the Python standard library significantly speeds up the results. For 1 million bytes, the Unicode C implementation used about 4.19 ms ± 2.89 µs, and the Python Unicode implementation takes about 145 µs ± 947 ns to run.

[ShaneHarvey]

We found that using the Python standard library significantly speeds up the results. For 1 million bytes, the Unicode C implementation used about 4.19 ms ± 2.89 µs, and the Python Unicode implementation takes about 145 µs ± 947 ns to run.

NBD only a modest 30x speed up. LGTM!