Skip to content
Permalink
Browse files

More unicode/utf8 fixes

  • Loading branch information
strk committed Jun 6, 2017
1 parent 4f4d63a commit 14ab5eb0bda9a5993b9ab15a3b7de1af9df8dcba
Showing with 7 additions and 7 deletions.
  1. +7 −7 python/plugins/db_manager/db_plugins/connector.py
@@ -78,7 +78,7 @@ def _execute(self, cursor, sql):
if cursor is None:
cursor = self._get_cursor()
try:
cursor.execute(unicode(sql))
cursor.execute(str(sql))

except self.connection_error_types() as e:
raise ConnectionError(e)
@@ -98,7 +98,7 @@ def _execute_and_commit(self, sql):
def _get_cursor(self, name=None):
try:
if name is not None:
name = unicode(name).encode('ascii', 'replace').replace('?', "_")
name = str(name).encode('ascii', 'replace').replace('?', "_")
self._last_cursor_named_id = 0 if not hasattr(self,
'_last_cursor_named_id') else self._last_cursor_named_id + 1
return self.connection.cursor("%s_%d" % (name, self._last_cursor_named_id))
@@ -181,16 +181,16 @@ def _get_cursor_columns(self, c):

@classmethod
def quoteId(self, identifier):
if hasattr(identifier, '__iter__'):
if hasattr(identifier, '__iter__') and not isinstance(identifier, str):
ids = list()
for i in identifier:
if i is None or i == "":
continue
ids.append(self.quoteId(i))
return u'.'.join(ids)

identifier = unicode(
identifier) if identifier is not None else unicode() # make sure it's python unicode string
identifier = str(
identifier) if identifier is not None else str() # make sure it's python unicode string
return u'"%s"' % identifier.replace('"', '""')

@classmethod
@@ -204,12 +204,12 @@ def quoteString(self, txt):
txts.append(self.quoteString(i))
return u'.'.join(txts)

txt = unicode(txt) if txt is not None else unicode() # make sure it's python unicode string
txt = str(txt) if txt is not None else str() # make sure it's python unicode string
return u"'%s'" % txt.replace("'", "''")

@classmethod
def getSchemaTableName(self, table):
if not hasattr(table, '__iter__'):
if not hasattr(table, '__iter__') and not isinstance(table, str):
return (None, table)
if isinstance(table, str):
table = table.split('.')

10 comments on commit 14ab5eb

@jusabatier

This comment has been minimized.

Copy link

@jusabatier jusabatier replied Jun 26, 2017

Since this commit, I get this error when trying to import data via 'cadastre' module in a PostGIS database :

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 104: ordinal not in range(128) 
Traceback (most recent call last):
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_dialogs.py", line 1090, in processImport
    qi.importMajic()
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_import.py", line 419, in importMajic
    self.executeSqlScript(s, False, item.has_key('constraints'))
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_import.py", line 1064, in executeSqlScript
    self.executeSqlQuery(sql, ignoreError)
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_import.py", line 1115, in executeSqlQuery
    c = self.connector._execute_and_commit(sql)
  File "C:/PROGRA~1/QGIS2~1.18/apps/qgis/./python/plugins\db_manager\db_plugins\connector.py", line 95, in _execute_and_commit
    self._execute(None, sql)
  File "C:/PROGRA~1/QGIS2~1.18/apps/qgis/./python/plugins\db_manager\db_plugins\connector.py", line 81, in _execute
    cursor.execute(str(sql))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 104: ordinal not in range(128)


Version de Python : 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] 
Version de QGIS : 2.18.10 Las Palmas, 59e0f78 

It look like the use of the str function in the connector create this error.

Have you some solution to fix this ? Is it a configuration problem with my environment ?
I tried both on a Deian and Windows platform and each time with the 2.18 version of QGis I have this error.

@Gustry

This comment has been minimized.

Copy link
Contributor

@Gustry Gustry replied Jun 26, 2017

Hum, yes, users with some accents in the attribute table may have some problems with this commit I think

@jusabatier

This comment has been minimized.

Copy link

@jusabatier jusabatier replied Jun 26, 2017

To my knowledge, this addon doesn't generate tables with accent in their attributes.

Just add comments on columns with accents : https://github.com/3liz/QgisCadastrePlugin/blob/master/scripts/plugin/commun_create_metier.sql#L1142

Maybe this is the problem.

But I think it have to be solved since it's a regression.

@mdouchin

This comment has been minimized.

Copy link
Contributor

@mdouchin mdouchin replied Jul 5, 2017

In fact, the "cadastre" french plugin uses dbmanager tools to run queries. Il would have to adapt the code to be able to manage QGIS before and after this commit.
3liz/QgisCadastrePlugin#106

@mdouchin

This comment has been minimized.

Copy link
Contributor

@mdouchin mdouchin replied Jul 5, 2017

It seems I can find a workaround by replacing in my code L 461 of cadastre_dialogs.py
c = connector._execute(None,unicode(sql))
by
c = connector._execute(None,unicode(sql).encode('utf-8'))

@jusabatier Please continue the issue conversation here for the cadastre plugin
3liz/QgisCadastrePlugin#106

@borysiasty

This comment has been minimized.

Copy link
Member

@borysiasty borysiasty replied Jul 13, 2017

@strk could you take a look at https://issues.qgis.org/issues/16833 please?
What was wrong with the previous unicode()? Looks like a mistaken backport from master/Py3? :)

@strk

This comment has been minimized.

Copy link
Contributor Author

@strk strk replied Jul 17, 2017

@strk

This comment has been minimized.

Copy link
Contributor Author

@strk strk replied Jul 18, 2017

Revert PR is in #4882

@strk

This comment has been minimized.

Copy link
Contributor Author

@strk strk replied Jul 19, 2017

Reverting the commit triggers an infinite loop, see https://travis-ci.org/qgis/QGIS/jobs/255027295

@strk

This comment has been minimized.

Copy link
Contributor Author

@strk strk replied Jul 19, 2017

NOTE: infinite loop only occurs with python3, python2 is fine with the revert. We need a solution working for both.

Please sign in to comment.
You can’t perform that action at this time.