docker_service python 3 compat #30696

dj4ngo · 2017-09-21T13:17:20Z

SUMMARY

Fix Python 3 Compatibility docker_service

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

docker_service

ANSIBLE VERSION

ansible 2.5.0 (docker_services_py35_compat d295a6b1f2) last updated 2017/09/21 15:12:16 (GMT +200)

ansibot · 2017-09-21T13:21:50Z

cc @chouseknecht
click here for bot help

ansibot · 2017-09-21T13:28:43Z

The test ansible-test sanity --test pep8 [?] failed with the following errors:

lib/ansible/modules/cloud/docker/docker_service.py:514:15: E201 whitespace after '['
lib/ansible/modules/cloud/docker/docker_service.py:514:34: E202 whitespace before ']'

click here for bot help

…t_redirected_output from cleanup_redirection_tempfiles since output is not used

…method Don't use --no-color option because replaced by --no-ansi to ensure compatibility The full traceback is: Traceback (most recent call last): File "/tmp/ansible_y084ix0x/ansible_module_docker_service.py", line 1068, in <module> main() File "/tmp/ansible_y084ix0x/ansible_module_docker_service.py", line 1063, in main result = ContainerManager(client).exec_module() File "/tmp/ansible_y084ix0x/ansible_module_docker_service.py", line 657, in exec_module result = self.cmd_up() File "/tmp/ansible_y084ix0x/ansible_module_docker_service.py", line 753, in cmd_up cleanup_redirection_tempfiles(out_redir_name, err_redir_name) File "/tmp/ansible_y084ix0x/ansible_module_docker_service.py", line 515, in cleanup_redirection_tempfiles get_redirected_output(err_name) File "/tmp/ansible_y084ix0x/ansible_module_docker_service.py", line 524, in get_redirected_output new_line = re.sub(r'\x1b\[.+m', '', line.encode('ascii')) File "/usr/lib/python3.5/re.py", line 182, in sub return _compile(pattern, flags).sub(repl, string, count) TypeError: cannot use a string pattern on a bytes-like object

mscherer · 2017-09-21T13:52:18Z

shipit

dj4ngo · 2017-09-21T13:55:28Z

@jctanner @kassiansun (as randomly selected maintainer of cloud/docker namespace) could you please review this pull request ? At least two shipit comments (from maintainers or namespace maintainers or core team members) are required in order to automerge this pull request.

mscherer · 2017-09-21T16:50:17Z

linked to #30672

dj4ngo · 2017-09-21T19:24:46Z

related to issue #30354

kassiansun · 2017-09-22T00:40:18Z

@abadger This MR contains further cleanups, should we merge this instead of #30672?

abadger · 2017-09-22T01:52:17Z

lib/ansible/modules/cloud/docker/docker_service.py



 def get_redirected_output(path_name):
    output = []
    with open(path_name, 'r') as fd:
        for line in fd:
            # strip terminal format/color chars
-            new_line = re.sub(r'\x1b\[.+m', '', line.encode('ascii'))
+            new_line = re.sub(r'\x1b\[.+m', '', to_text(line))


This portion is wrong. On python3 it doesn't do anything (because line is already text). On python2 it makes a byte string pattern compare to a text string. If there's no nonascii characters I think that will work okay but it will definitely fail to do the right thing if nonascii does creep into the data.

abadger · 2017-09-22T01:54:03Z

The python3 compatibility here is not correct. The other changes look good but I'm not a docker user or maintainer so I've been trying not to touch it too much.

kassiansun · 2017-09-22T01:56:30Z

@abadger So for line in f returns bytes string on python 2.x? Then line needs to be converted with to_text as re.sub don't accept bytes string as argument on 2.7.x (higher versions) and 3.x (lower versions).

abadger · 2017-09-22T02:42:47Z

That's incorrect. Both python2 and python3 accept both byte strings and text strings as arguments. What you shouldn't do is mix them together.

$ python2          *[python3-port-try2]  (19:41:18)
Python 2.7.13 (default, May 10 2017, 20:04:28) 
[GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.sub(b'test', b'', b'this is a test case')
'this is a  case'

$ python3          *[python3-port-try2]  (19:41:57)
Python 3.5.4 (default, Aug 23 2017, 18:32:05) 
[GCC 6.4.1 20170727 (Red Hat 6.4.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.sub(b'test', b'', b'this is a test case')
b'this is a  case'

abadger · 2017-09-22T02:45:05Z

lib/ansible/modules/cloud/docker/docker_service.py

            output.append(new_line)
-    fd.close()


Okay, I just tested this and it looks like fd.close() does not cause an error. It's useless to have it there but not strictly necessary to remove it.

kassiansun · 2017-09-22T02:46:11Z

This is not the case with the source code:

Python 3.4.3 (default, Sep 14 2016, 12:36:27)
>>> re.sub(r'\x1b\[.+m', '', b'this is a test case')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/re.py", line 179, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: can't use a string pattern on a bytes-like object

abadger · 2017-09-22T02:49:08Z

@kassiansun You are doing exactly what I told you you can't do. You are mixing text strings r'\x1b\[.+m' and '' with a byte string: b'this is a test case'.

kassiansun · 2017-09-22T02:50:16Z

But this is the case with source code, we need to make sure the third one is text string

kassiansun · 2017-09-22T02:53:23Z

lib/ansible/modules/cloud/docker/docker_service.py



 def get_redirected_output(path_name):
    output = []
    with open(path_name, 'r') as fd:
        for line in fd:
            # strip terminal format/color chars
-            new_line = re.sub(r'\x1b\[.+m', '', line.encode('ascii'))
+            new_line = re.sub(r'\x1b\[.+m', '', to_text(line))


There's no need to do to_text here, re.sub takes str as argument, bytes or text

@abadger This behavior of re.sub is not consistent among python 3.x... But I think for line in f always return line as text str?

@kassiansun python-3.x's re.sub has not changed. The basic rules of using it are even the same in python-2.x but python-2.x sometimes allows you to cheat (when the data you are operating on is ascii-only).

You also are incorrect about the type of line in for line in f. That depends on what f returns on iteration. In this case, f is a file which is opened in the default read mode. On Python-2.x, iterating f returns byte strings. On python3, it returns text strings. If desired, you can force it to return byte strings on both python2 and python3 by opening the file in binary mode ('rb') but there is no way to force it to return text strings on both python2 and python3.

Thanks, that's the point I want to get. Your code is consistent among 2.x and 3.x, because for line in f returns line consistent with r''.

abadger · 2017-09-22T03:02:52Z

@dj4ngo I've rebased and merged your cleanup commit on top of my fix for the python3 compat problem. The commit is here: 901bc2c

@kassiansun No. We need to make sure that all three of the strings are the same type (either all three bytes or all three text). My change, now merged does that. The change here creates a mismatch on Python2.

Additionally, this change would have changed the type of the string returned from get_redirected_output from a byte string to a text string on Python2. That could cause tracebacks in the calling code if it wasn't careful of how it combined the byte string with the

ansibot added ci_verified Changes made in this PR are causing tests to fail. needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. and removed community_review In order to be merged, this PR must follow the community review workflow. labels Sep 21, 2017

dj4ngo added 2 commits Sep 21, 2017

Clean code, fd.close not required since open as fd if used; remove ge…

bae2070

…t_redirected_output from cleanup_redirection_tempfiles since output is not used

dj4ngo force-pushed the docker_services_py35_compat branch from d295a6b to 177c609 Compare Sep 21, 2017

ansibot removed the ci_verified Changes made in this PR are causing tests to fail. label Sep 21, 2017

mscherer removed the needs_triage Needs a first human triage before being processed. label Sep 21, 2017

ansibot added community_review In order to be merged, this PR must follow the community review workflow. and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. labels Sep 21, 2017

abadger reviewed Sep 22, 2017

View changes

kassiansun suggested changes Sep 22, 2017

View changes

abadger closed this Sep 22, 2017

ansibot added bug This issue/PR relates to a bug. and removed bugfix_pull_request labels Mar 6, 2018

ansible locked and limited conversation to collaborators Apr 26, 2019

docker_service python 3 compat #30696

docker_service python 3 compat #30696

dj4ngo commented Sep 21, 2017

ansibot commented Sep 21, 2017

ansibot commented Sep 21, 2017

mscherer commented Sep 21, 2017

dj4ngo commented Sep 21, 2017

mscherer commented Sep 21, 2017

dj4ngo commented Sep 21, 2017

kassiansun commented Sep 22, 2017

abadger Sep 22, 2017

abadger commented Sep 22, 2017

kassiansun commented Sep 22, 2017

abadger commented Sep 22, 2017

abadger Sep 22, 2017

kassiansun commented Sep 22, 2017

abadger commented Sep 22, 2017

kassiansun commented Sep 22, 2017 •

edited

kassiansun Sep 22, 2017

kassiansun Sep 22, 2017

abadger Sep 22, 2017

kassiansun Sep 22, 2017

abadger commented Sep 22, 2017

docker_service python 3 compat #30696

docker_service python 3 compat #30696

Conversation

dj4ngo commented Sep 21, 2017

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

ansibot commented Sep 21, 2017

ansibot commented Sep 21, 2017

mscherer commented Sep 21, 2017

dj4ngo commented Sep 21, 2017

mscherer commented Sep 21, 2017

dj4ngo commented Sep 21, 2017

kassiansun commented Sep 22, 2017

abadger Sep 22, 2017

Choose a reason for hiding this comment

abadger commented Sep 22, 2017

kassiansun commented Sep 22, 2017

abadger commented Sep 22, 2017

abadger Sep 22, 2017

Choose a reason for hiding this comment

kassiansun commented Sep 22, 2017

abadger commented Sep 22, 2017

kassiansun commented Sep 22, 2017 • edited

kassiansun Sep 22, 2017

Choose a reason for hiding this comment

kassiansun Sep 22, 2017

Choose a reason for hiding this comment

abadger Sep 22, 2017

Choose a reason for hiding this comment

kassiansun Sep 22, 2017

Choose a reason for hiding this comment

abadger commented Sep 22, 2017

kassiansun commented Sep 22, 2017 •

edited