Question about UnicodeDecodeError

[SincerelyUnique]

(1) Error code:
df_result = pd.read_sql_query(sql, engine)

(2) Error message:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 41: invalid start byte

(3) Traceback:
site-packages\pandas\io\sql.py, line 314 site-packages\pandas\io\sql.py, line 1108 site-packages\sqlalchemy\engine\result.py, line 1216 site-packages\sqlalchemy\engine\base.py, line 1475 site-packages\sqlalchemy\engine\result.py, line 1211 site-packages\sqlalchemy\engine\result.py, line 1161

(4) Error Reason:
DB encoding is iso-8859-1, nls_lang is AMERICAN_AMERICA.WE8ISO8859P1,
but db table has uninterruptedly space character(encoding is 0xa0), so utf-8 can't decode it

(5) Solution:
change sqlalchemy version to 1.2.6, it's ok
pip install SQLAlchemy==1.2.6

(6) So my question is what's the difference between version 1.2.6 and version 1.3.10, could you help me?

[zzzeek]

there is an encoding_errors flag that will be in 1.3.11:

https://docs.sqlalchemy.org/en/13/changelog/changelog_13.html#change-ecabed6215423c25dbaabffd240cc534

try out from git:

https://github.com/sqlalchemy/sqlalchemy/archive/rel_1_3.zip

then set encoding_errors='ignore' in create_engine()

pls confirm it works thanks

[zzzeek]

#4799

[zzzeek]

1.2 is not using SQLAlchemy's decoder, it uses cx_Oracle's, which I am beginning to suspect implicitly ignores encoding errors, even though this is configurable what cx_Oracle version are you using please

[zzzeek]

oh also is this python 2 you are using. in python 3 there's no difference

[SincerelyUnique]

@zzzeek Thanks for your help, it works very well. I will list my operation here.
Install:

1. python setup.py install
2. easy_install SQLAlchemy-1.3.11.dev0-py2.7-win-amd64.egg
3. pip list (sqlalchemy version is 1.3.11.dev0)

Try:
engine = create_engine(ProdConfig.SQLALCHEMY_DATABASE_URI, encoding_errors='ignore')
It works fine!

[zzzeek]

are you using python 2 or 3 ?

[SincerelyUnique]

Python2.7.15

[zzzeek]

ok that makes sense then, wait for 1.3.11