Unicode bug

[DownGoat]

Exception in thread Thread-12:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in bootstrap_inner
self.run()
File "/home/sis13/Dropbox/PyCharm/FeedServer/feedserver/feedstorer.py", line 98, in run
add_entry(entry, feed.id)
File "/home/sis13/Dropbox/PyCharm/FeedServer/feedserver/feedstorer.py", line 28, in add_entry
stored_ent = db_session.query(Entry).filter_by(link=entry.get("link")).first()
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/orm/query.py", line 2145, in first
ret = list(self[0:1])
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/orm/query.py", line 2012, in __getitem
return list(res)
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/orm/loading.py", line 72, in instances
rows = [process[0](row, None) for row in fetch]
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/orm/loading.py", line 447, in instance
populate_state(state, dict, row, isnew, only_load_props)
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/orm/loading.py", line 301, in populate_state
populator(state, dict_, row)
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/orm/strategies.py", line 150, in fetch_col
dict_[key] = row[col]
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/engine/result.py", line 89, in getitem
return processor(self._row[index])
File "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.8.1-py2.7.egg/sqlalchemy/processors.py", line 66, in process
return decoder(value, errors)[0]
File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 254-255: unexpected end of data

[abourget]

Have you found the solution for this bug ? I get the same thing, and I'm really not sure what's going on.. it seems it's in SQLAlchemy, but I'm not sure..

[DownGoat]

I have not written any code to solve this, but the problem is that the data in the feed is not unicode, and the data loaded from the feed has to be converted to unicode somewhere before being passed to any DB stuff.

[DownGoat]

Some more exceptions, saving them here for later.

/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:331: Warning: Data truncated for column 'description' at row 1
  cursor.execute(statement, parameters)
2014-05-10 16:28:47,113 - feedserver - CRITICAL - Unicode error?
Traceback (most recent call last):
  File "/home/dwg/ownCloud/PyCharm/FeedServer/feedserver/feedstorer.py", line 31, in add_entry
    stored_ent = db_session.query(Entry).filter_by(link=entry.get("link")).first()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2156, in first
    ret = list(self[0:1])
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2023, in __getitem__
    return list(res)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2348, in instances
    rows = [process[0](row, None) for row in fetch]
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 2120, in _instance
    populate_state(state, dict_, row, isnew, only_load_props)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 1974, in populate_state
    populator(state, dict_, row)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 150, in fetch_col
    dict_[key] = row[col]
UnicodeDecodeError: 'utf8' codec can't decode byte 0xca in position 1023: unexpected end of data

687474703a2f2f7777772e7265646469742e636f6d2f722f636f6e6c616e67732f636f6d6d656e74732f3235376d75792f616e5f6578616d706c655f6f665f6d795f66697273745f6469616c6563745f666f725f796f7272612f

Exception in thread Thread-12:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/dwg/ownCloud/PyCharm/FeedServer/feedserver/feedstorer.py", line 108, in run
    add_entry(entry, feed.id)
  File "/home/dwg/ownCloud/PyCharm/FeedServer/feedserver/feedstorer.py", line 57, in add_entry
    logger.debug(u"Updating entry with id: {0}".format(entry.get("id")))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 57: ordinal not in range(128)