Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ignore the UnicodeDecodeError when some bytes cannot be decoded with … #583

Closed
wants to merge 1 commit into from

Conversation

wzzz
Copy link

@wzzz wzzz commented Apr 21, 2023

sometime the function cursor.fetchall() raise exceptions like follower:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 0: invalid continuation byte

this exception is very annoying if you don't fix the records which include the Garbled character.

…the specified charset

use env variable MYC_USE_DECODE_ERR_REP_MODE to switch on/off, if MYC_USE_DECODE_ERR_REP_MODE is 1, then use replace mode, or NULL mode
} else {
//fprintf(stderr, "decoding with %s\n", encoding);
return PyUnicode_Decode(rowitem, length, encoding, NULL);
return PyUnicode_Decode(rowitem, length, encoding, decode_err_mode);

Check notice

Code scanning / CodeQL

Commented-out code

This comment appears to contain commented-out code.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's existed before my modifcation.

if (encoding == utf8) {
//fprintf(stderr, "decoding with utf8!\n");
return PyUnicode_DecodeUTF8(rowitem, length, NULL);
return PyUnicode_DecodeUTF8(rowitem, length, decode_err_mode);

Check notice

Code scanning / CodeQL

Commented-out code

This comment appears to contain commented-out code.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's existed before my modifcation.

@codecov
Copy link

codecov bot commented Apr 21, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (f5a2f3d) 86.00% compared to head (e8ffcdc) 86.00%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #583   +/-   ##
=======================================
  Coverage   86.00%   86.00%           
=======================================
  Files           6        6           
  Lines         550      550           
=======================================
  Hits          473      473           
  Misses         77       77           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@methane
Copy link
Member

methane commented Apr 23, 2023

I don't maintain such code.
Use use_unicode=False.

@methane methane closed this Apr 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants