Add optional parameter 'ignore', 'replace' or 'backslashreplace' in string the decoding function

[rudaporto]

I've found some issues with some records stored in a MySQL database.
At this line, an UnicodeDecodeError exception is raised when I'm fetching data from a specific table:

https://github.com/PyMySQL/mysqlclient-python/blob/3de469db63c5da47d61ac79bba5dadc0d22bde5c/MySQLdb/connections.py#L231

I know that the issue is caused by some malformed data store in MySQL that cannot be decoded as UTF-8 since they were incorrectly encoded in the first place.

But it would be nice if the driver has a better way to deal with these cases by simply adding an option in the connection defining the string decode configuration: 'strict', 'ignore', 'replace' or 'backslashreplace'.

So, basically I'm proposing to add the following option to the connection class.

    :param str decode_errors='strict':
        The default value is 'strict' but it can accept 'ignore' or 'replace' in PY2
        And in PY3 it can be also set as 'backslashreplace'

And the string_decoder implementation would be:

def string_decoder(s):
    return s.decode(db.encoding, errors=decode_errors)

References:

• https://docs.python.org/2/howto/unicode.html
• https://docs.python.org/3/howto/unicode.html

[methane]

Use use_unicode=False option.

[rudaporto]

@methane Thanks for the quick response.

Is that bad to be able to parametrize the decode behavior?

Sure, I can get bytes and do the decode by myself but the idea was exactly to not do that since it can be not that simple without all the context.

[methane]

I can't maintain more option&behavior. This project is too complicated to me.
You can write your own wrapper instead.

[rudaporto]

@methane yes, I see since it's too hard to customize anything or sublcass and just override one method or parameter.

So the focus is to improve PyMySQL to make it fast so we do not need this driver anymore?

[methane]

I won't add such option to PyMySQL too.
PyMySQL is complicated too because of PyMySQL aimed to compatible with this project
and this project is complicated.

I don't add any "convenient for 1% user" options. It makes more hard to find bugs.

My goal is less compexity, less maintenance effort, smaller code, less bugs.
Any convenient feature should be implemented in upper layer.

[rudaporto]

I don't add any "convenient for 1% user" options. It makes more hard to find bugs.

I usually agree with this, but I'm not asking to add any fancy and complicated parameter.
It's just a parameter to customize the call to the std str(PY2) or bytes(PY3) decode method.

It was my assumption that other users could benefit from it, that is why I created this request, not for my convenience.

I will create my wrapper and override this function after the connection instance is created, no problem.

Thanks anyway!

[devl00p]

I bumped into that issue too and I'm agree an option would be so nice. I had to patch the connection.converter at runtime to overwrite the function.