Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bytes error in mysql module #45026

Closed
angeloudy opened this issue Dec 15, 2017 · 27 comments

Comments

Projects
None yet
@angeloudy
Copy link
Contributor

commented Dec 15, 2017

Description of Issue/Question

I recently switch from python2.7 to python 3.6 and a lot of modules are stopped working.

The first issue is pip: #44980
Second issue is mysql

Function: mysql_grants.present
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/local/lib/python3.6/site-packages/salt/state.py", line 1843, in call
                  **cdata['kwargs'])
                File "/usr/local/lib/python3.6/site-packages/salt/loader.py", line 1795, in wrapper
                  return f(*args, **kwargs)
                File "/usr/local/lib/python3.6/site-packages/salt/states/mysql_grants.py", line 150, in present
                  grant, database, user, host, grant_option, escape, **connection_args
                File "/usr/local/lib/python3.6/site-packages/salt/modules/mysql.py", line 1778, in grant_exists
                  grants = user_grants(user, host, **connection_args)
                File "/usr/local/lib/python3.6/site-packages/salt/modules/mysql.py", line 1745, in user_grants
                  tmp = grant[0].split(' IDENTIFIED BY')[0]
              TypeError: a bytes-like object is required, not 'str'
     Started: 10:50:28.599004
    Duration: 11.843 ms
     Changes:

Third issue is with http

 Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/local/lib/python3.6/site-packages/salt/state.py", line 1843, in call
                  **cdata['kwargs'])
                File "/usr/local/lib/python3.6/site-packages/salt/loader.py", line 1795, in wrapper
                  return f(*args, **kwargs)
                File "/usr/local/lib/python3.6/site-packages/salt/states/http.py", line 153, in wait_for_successful_query
                  raise caught_exception  # pylint: disable=E0702
                File "/usr/local/lib/python3.6/site-packages/salt/states/http.py", line 144, in wait_for_successful_query
                  ret = query(name, wait_for=wait_for, **kwargs)
                File "/usr/local/lib/python3.6/site-packages/salt/states/http.py", line 96, in query
                  if match in data.get('text', ''):
              TypeError: a bytes-like object is required, not 'str'
     Started: 10:27:10.800930
    Duration: 301579.96 ms

All the issues I am having have similar errors:

TypeError: a bytes-like object is required, not 'str'

Can someone look at this?

Setup

FreeBSD 11.1

Steps to Reproduce Issue

  1. install package py36-salt
  2. to reproduce pip issue, create a state file with pip.installed and added it to top.sls and run salt state.highstate
  3. to reproduce mysql issue, create a state file with mysql state and run it from highstate.

Versions Report

Salt Version:
Salt: 2017.7.2

Dependency Versions:
cffi: 1.11.2
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: 1.2.5
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.3 (default, Dec 11 2017, 00:25:34)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.3
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.2
ZMQ: 4.2.2

System Versions:
dist:
locale: UTF-8
machine: amd64
release: 11.1-RELEASE-p4
system: FreeBSD
version: Not Installed

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Dec 18, 2017

@angeloudy can you please share a sanitized version of your state files to help replicate this issue?

@Ch3LL Ch3LL added the Info Needed label Dec 18, 2017

@Ch3LL Ch3LL added this to the Blocked milestone Dec 18, 2017

@drel4o

This comment has been minimized.

Copy link

commented Mar 15, 2018

Hello there,

I am experiencing the same problem, so here is an example of a failing state:

test_db grants:
mysql_grants.present:
- host: localhost
- grant: all
- database: 'test_db.*'
- user: {{ pillar['db']['user'] }}

The database and the user exist in mysql.

Salt Version:
Salt: 2017.7.4

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.4
mysql-python: 1.3.10
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.4+ (default, Feb 12 2018, 08:25:03)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.3
System Versions:
dist: Ubuntu 18.04 bionic
locale: UTF-8
machine: x86_64
release: 4.13.0-32-generic
system: Linux
version: Ubuntu 18.04 bionic

Thank you in advance. :)
D

@angeloudy

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2018

This was not an issue of salt, but the issue of python mysql library.
I fixed it with the following patch

--- MySQLdb/cursors.py.orig
+++ MySQLdb/cursors.py
@@ -21,7 +21,17 @@
 else:
     text_type = str

-
+def convert_to_str(var):
+    if isinstance(var,tuple):
+        return tuple(convert_to_str(item) for item in var)
+    if isinstance(var,list):
+        return ([convert_to_str(item) for item in var])
+    elif isinstance(var,dict):
+        return {convert_to_str(key):convert_to_str(value) for key,value in var.items()}
+    elif isinstance(var,bytes):
+        return var.decode('utf-8')
+    else:
+        return var
 #: Regular expression for :meth:`Cursor.executemany`.
 #: executemany only supports simple bulk insert.
 #: You can use it to load large dataset.
@@ -443,6 +453,9 @@
         else:
             result = self._rows
         self.rownumber = len(self._rows)
+        if not PY2:
+            db = self._get_db()
+            result = tuple(convert_to_str(result))
         return result

     def scroll(self, value, mode='relative'):

@angeloudy angeloudy closed this May 3, 2018

@angeloudy angeloudy reopened this May 3, 2018

@angeloudy

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2018

I am still not sure if this needs to be fixed in mysqlclient library or in salt.

@angeloudy angeloudy changed the title python 3 not fully supported bytes error in mysql module May 3, 2018

@methane

This comment has been minimized.

Copy link

commented Jun 28, 2018

What is connection_args?
mysqlclient returns Unicode normally.

@methane

This comment has been minimized.

Copy link

commented Jun 28, 2018

You need to pass charset="utf8mb4" or "utf8" to connection args.

@methane

This comment has been minimized.

Copy link

commented Jun 29, 2018

I found, salt disables unicode for PY2, but it is disabled for PY3 too.

salt/salt/modules/mysql.py

Lines 316 to 323 in 7d41e03

# MySQLdb states that this is required for charset usage
# but in fact it's more than it's internally activated
# when charset is used, activating use_unicode here would
# retrieve utf8 strings as unicode() objects in salt
# and we do not want that.
#_connarg('connection_use_unicode', 'use_unicode')
connargs['use_unicode'] = False
_connarg('connection_charset', 'charset')

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Jul 3, 2018

is this still an issue on 2018.3? we added a lot of improvements with unicode in this version

@methane

This comment has been minimized.

Copy link

commented Jul 3, 2018

I think so, because same configs are in 2018.3.2 tag.
https://github.com/saltstack/salt/blob/2018.3.2/salt/modules/mysql.py#L318-L325

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2018

And if you set that to True the failure goes away?

@methane

This comment has been minimized.

Copy link

commented Jul 6, 2018

I think so, but I'm not salt user.
I'm a maintainer of mysqlclient-python.

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2018

ping @regilero looks like you set this to False. Is there a specific reason you see to not change it to True?

@methane

This comment has been minimized.

Copy link

commented Jul 10, 2018

I suppose he did it for PY2.

use_unicode=True is OK for PY2 if the code is written to use unicode string.
But if not, it should be use_unicode=(not PY2).

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2018

k thanks for clarifying. looks like this will need to get fixed up. Will label as a bug.

@Ch3LL Ch3LL added Bug High Severity P4 and removed Info Needed labels Jul 11, 2018

@Ch3LL Ch3LL modified the milestones: Blocked, Approved Jul 11, 2018

@andreasnuesslein

This comment has been minimized.

Copy link

commented Nov 15, 2018

i believe i have the same issue, for me it helped to str() the relevant return-values from MySQLdb's cur.fetchall(), like so:

diff --git a/salt/modules/mysql.py b/salt/modules/mysql.py
index 889de1c..1e32556 100644
--- a/mysql.py
+++ b/mysql.py
@@ -1766,8 +1766,8 @@ def user_grants(user,
     ret = []
     results = cur.fetchall()
     for grant in results:
-        tmp = grant[0].split(' IDENTIFIED BY')[0]
-        if 'WITH GRANT OPTION' in grant[0] and 'WITH GRANT OPTION' not in tmp:
+        tmp = str(grant[0]).split(' IDENTIFIED BY')[0]
+        if 'WITH GRANT OPTION' in str(grant[0]) and 'WITH GRANT OPTION' not in tmp:
             tmp = '{0} WITH GRANT OPTION'.format(tmp)
         ret.append(tmp)
     log.debug(ret)
@zerthimon

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2018

The diff provided by @andreasnuesslein solves the issue!
Thanks a lot!

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2018

does it work when using salt.utils.stringutils.to_str() as that will handle cases for both python2 and python3.

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2018

actually salt.utils.stringutils.to_unicode() should be used when you are attempting to simply convert a value from a non-string type to string type

@Ch3LL

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2018

but @terminalmage can verify my thinking here

@terminalmage

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2018

Fixed by #50823.

@morph027

This comment has been minimized.

Copy link

commented Dec 19, 2018

Any chance we can get this backported to 2017.7 too? ;)

@terminalmage

This comment has been minimized.

Copy link
Contributor

commented Dec 19, 2018

Unlikely, since A) there are no more 2017.7 releases planned, and B) the helper function used to make this fix doesn't exist in that release branch.

@morph027

This comment has been minimized.

Copy link

commented Dec 19, 2018

Okay...as the fix from @andreasnuesslein works, i'm going to check how i can incorporate it into salt.

@ernestas-s

This comment has been minimized.

Copy link

commented Feb 25, 2019

I use onliner to fix this on daily basis with v2018.3.3
sed -i '1756,1758{s/grant\[0\]/str\(grant\[0\]\)/}' /usr/lib/python3/dist-packages/salt/modules/mysql.py

@terminalmage

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

@saltstack/team-core This should be able to be closed since the fix was merged.

@waynew

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

👍

@waynew waynew closed this Mar 26, 2019

@pkramme

This comment has been minimized.

Copy link

commented Apr 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.