Skip to content

Commit

Permalink
More unicode/utf8 fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
@strk
strk committed Jun 6, 2017
1 parent 4f4d63a commit 14ab5eb
Showing 1 changed file with 7 additions and 7 deletions.
14 changes: 7 additions & 7 deletions python/plugins/db_manager/db_plugins/connector.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ def _execute(self, cursor, sql):
if cursor is None:
cursor = self._get_cursor()
try:
cursor.execute(unicode(sql))
cursor.execute(str(sql))

except self.connection_error_types() as e:
raise ConnectionError(e)
Expand All @@ -98,7 +98,7 @@ def _execute_and_commit(self, sql):
def _get_cursor(self, name=None):
try:
if name is not None:
name = unicode(name).encode('ascii', 'replace').replace('?', "_")
name = str(name).encode('ascii', 'replace').replace('?', "_")
self._last_cursor_named_id = 0 if not hasattr(self,
'_last_cursor_named_id') else self._last_cursor_named_id + 1
return self.connection.cursor("%s_%d" % (name, self._last_cursor_named_id))
Expand Down Expand Up @@ -181,16 +181,16 @@ def _get_cursor_columns(self, c):

@classmethod
def quoteId(self, identifier):
if hasattr(identifier, '__iter__'):
if hasattr(identifier, '__iter__') and not isinstance(identifier, str):
ids = list()
for i in identifier:
if i is None or i == "":
continue
ids.append(self.quoteId(i))
return u'.'.join(ids)

identifier = unicode(
identifier) if identifier is not None else unicode() # make sure it's python unicode string
identifier = str(
identifier) if identifier is not None else str() # make sure it's python unicode string
return u'"%s"' % identifier.replace('"', '""')

@classmethod
Expand All @@ -204,12 +204,12 @@ def quoteString(self, txt):
txts.append(self.quoteString(i))
return u'.'.join(txts)

txt = unicode(txt) if txt is not None else unicode() # make sure it's python unicode string
txt = str(txt) if txt is not None else str() # make sure it's python unicode string
return u"'%s'" % txt.replace("'", "''")

@classmethod
def getSchemaTableName(self, table):
if not hasattr(table, '__iter__'):
if not hasattr(table, '__iter__') and not isinstance(table, str):
return (None, table)
if isinstance(table, str):
table = table.split('.')
Expand Down

10 comments on commit 14ab5eb

@jusabatier
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this commit, I get this error when trying to import data via 'cadastre' module in a PostGIS database :

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 104: ordinal not in range(128) 
Traceback (most recent call last):
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_dialogs.py", line 1090, in processImport
    qi.importMajic()
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_import.py", line 419, in importMajic
    self.executeSqlScript(s, False, item.has_key('constraints'))
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_import.py", line 1064, in executeSqlScript
    self.executeSqlQuery(sql, ignoreError)
  File "C:/Users/julien.sabatier/.qgis2/python/plugins\cadastre\cadastre_import.py", line 1115, in executeSqlQuery
    c = self.connector._execute_and_commit(sql)
  File "C:/PROGRA~1/QGIS2~1.18/apps/qgis/./python/plugins\db_manager\db_plugins\connector.py", line 95, in _execute_and_commit
    self._execute(None, sql)
  File "C:/PROGRA~1/QGIS2~1.18/apps/qgis/./python/plugins\db_manager\db_plugins\connector.py", line 81, in _execute
    cursor.execute(str(sql))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 104: ordinal not in range(128)


Version de Python : 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] 
Version de QGIS : 2.18.10 Las Palmas, 59e0f78

It look like the use of the str function in the connector create this error.

Have you some solution to fix this ? Is it a configuration problem with my environment ?
I tried both on a Deian and Windows platform and each time with the 2.18 version of QGis I have this error.

@Gustry
Copy link
Contributor

@Gustry Gustry commented on 14ab5eb Jun 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, yes, users with some accents in the attribute table may have some problems with this commit I think

@jusabatier
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To my knowledge, this addon doesn't generate tables with accent in their attributes.

Just add comments on columns with accents : https://github.com/3liz/QgisCadastrePlugin/blob/master/scripts/plugin/commun_create_metier.sql#L1142

Maybe this is the problem.

But I think it have to be solved since it's a regression.

@mdouchin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, the "cadastre" french plugin uses dbmanager tools to run queries. Il would have to adapt the code to be able to manage QGIS before and after this commit.
3liz/QgisCadastrePlugin#106

@mdouchin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems I can find a workaround by replacing in my code L 461 of cadastre_dialogs.py
c = connector._execute(None,unicode(sql))
by
c = connector._execute(None,unicode(sql).encode('utf-8'))

@jusabatier Please continue the issue conversation here for the cadastre plugin
3liz/QgisCadastrePlugin#106

@borysiasty
Copy link
Member

@borysiasty borysiasty commented on 14ab5eb Jul 13, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@strk could you take a look at https://issues.qgis.org/issues/16833 please?
What was wrong with the previous unicode()? Looks like a mistaken backport from master/Py3? :)

@strk
Copy link
Contributor Author

@strk strk commented on 14ab5eb Jul 17, 2017 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@strk
Copy link
Contributor Author

@strk strk commented on 14ab5eb Jul 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert PR is in #4882

@strk
Copy link
Contributor Author

@strk strk commented on 14ab5eb Jul 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverting the commit triggers an infinite loop, see https://travis-ci.org/qgis/QGIS/jobs/255027295

@strk
Copy link
Contributor Author

@strk strk commented on 14ab5eb Jul 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NOTE: infinite loop only occurs with python3, python2 is fine with the revert. We need a solution working for both.

Please sign in to comment.