Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ansible 2.6 / python3: CallError: builtins.TypeError: the JSON object must be str, not 'bytes' #374

Closed
candlerb opened this issue Sep 18, 2018 · 5 comments

Comments

@candlerb
Copy link
Contributor

@candlerb candlerb commented Sep 18, 2018

  • mitogen master plus #372 patch
  • ubuntu 16.04.5, lxd 2.0.11
  • python 2.7.9 on host, python 3.5.1 in lxd container
  • ansible 2.6.4 from ppa

Test case:

==> test.yml <==
- hosts: gold-master
  tasks:
    - raw: echo hello world

==> hosts.yml <==
all:
  vars:
    ansible_python_interpreter: /usr/bin/python3
  hosts:
    gold-master:
      ansible_connection: lxd

==> ansible.cfg <==
[defaults]
inventory = hosts.yml
#strategy_plugins = /usr/local/share/mitogen/ansible_mitogen/plugins/strategy
#strategy = mitogen_linear

gold-master is an ubuntu 16.04 lxd container.

Works with mitogen commented out. With mitogen enabled, I get:

$ ansible-playbook test.yml

PLAY [gold-master] ******************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was:     s.__class__.__name__))
fatal: [gold-master]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
	to retry, use: --limit @/home/ubuntu/ansible/misc/test.retry

PLAY RECAP **************************************************************************************************************************
gold-master                : ok=0    changed=0    unreachable=0    failed=1

With -vvv there is an exception traceback shown:

[pid 3603] 11:13:12.040260 D mitogen: mitogen.core.Stream(u'unix_client.3622').on_disconnect()
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 139, in run
    res = self._execute()
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 584, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/mixins.py", line 116, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/normal.py", line 46, in run
    result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/mixins.py", line 334, in _execute_module
    timeout_secs=self.get_task_timeout_secs(),
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/planner.py", line 484, in invoke
    kwargs=planner.get_kwargs(),
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/connection.py", line 428, in call
    return self._rethrow(recv)
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/connection.py", line 415, in _rethrow
    return recv.get().unpickle()
  File "/usr/local/share/mitogen-candlerb-371/mitogen/core.py", line 595, in unpickle
    raise obj
CallError: builtins.TypeError: the JSON object must be str, not 'bytes'
  File "<stdin>", line 1993, in _dispatch_one
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/target.py", line 353, in run_module
    impl = klass(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 633, in __init__
    super(NewStyleRunner, self).__init__(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 578, in __init__
    super(ScriptRunner, self).__init__(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 445, in __init__
    super(ProgramRunner, self).__init__(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 252, in __init__
    self.args = json.loads(json_args)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))

Looks like a bytes to str conversion is missing somewhere.

I tried a band-aid of json.loads(str(json_args)), but this then gives a different error UPDATE: see below.

@RonnyPfannschmidt
Copy link

@RonnyPfannschmidt RonnyPfannschmidt commented Sep 18, 2018

please try to show the content of the json-value as well

@candlerb
Copy link
Contributor Author

@candlerb candlerb commented Sep 18, 2018

It was my mistake, I should have used decode() not str().

With the following sticking plaster it works:

--- /usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py.orig	2018-09-18 11:21:52.414854084 +0000
+++ /usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py	2018-09-18 11:56:20.106790770 +0000
@@ -249,7 +249,7 @@
         self.service_context = service_context
         self.econtext = econtext
         self.detach = detach
-        self.args = json.loads(json_args)
+        self.args = json.loads(json_args.decode("UTF-8"))
         self.temp_dir = temp_dir
         self.extra_env = extra_env
         self.env = env

However, I imagine that really it's the responsibility of the caller of this function to pass json_args as as a string, not bytes.

FWIW, I did also log json_args. The value was:

b'{"_ansible_version": "2.6.4", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p"], "_ansible_no_log": false, "gather_timeout": 10, "_ansible_module_name": "setup", "_ansible_debug": false, "_ansible_verbosity": 3, "_ansible_keep_remote_files": false, "_ansible_syslog_facility": "LOG_USER", "_ansible_socket": null, "gather_subset": "all", "_ansible_diff": false, "_ansible_remote_tmp": "~/.ansible/tmp", "_ansible_shell_executable": "/bin/sh", "_ansible_check_mode": false, "_ansible_tmpdir": "/root/.ansible/tmp/ansible_mitogen_l9nzvee2/worker-4388-7ff4cc7b1650"}'
@dw
Copy link
Member

@dw dw commented Sep 19, 2018

Thanks for reporting this. I'll add something to CI, because that change needs tested in both directions.

@dw
Copy link
Member

@dw dw commented Oct 3, 2018

The problem is specific to Python>2,<3.6:

Changed in version 3.6: s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.

This is a minor pain, I need to add another CI target to cover 3.5. There is an annoying balance between speedy but inaccurate or dog-slow (and thus forgotten/ignored) CI

@dw
Copy link
Member

@dw dw commented Nov 7, 2018

Sorry for the crazy delay - discovered a can of worms around Python 3.x in the process! You may wish to consider subscribing to #426 in the meantime.

The fix for this was committed in 16911c9.

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!

@dw dw closed this Nov 7, 2018
dw added a commit that referenced this issue Nov 7, 2018
dw added a commit that referenced this issue Nov 23, 2018
dw added a commit that referenced this issue Jan 19, 2019
dw added a commit that referenced this issue Jan 19, 2019
d2eb01f tests: pin idna to last supporting 2.6-compat version.
b90889c tests: pin idna to last supporting 2.6-compat version.
9da6e6a tests: don't call Router.shutdown() twice.
b2b7e7b tests: file_service_test fixes
04755c3 issue #426: tighten up PushFileService types.
8fa3c74 issue #426: RouterMonitor format incorrect for 3->2 forward.
d8b9634 issue #426: PushFileService missing to_text() call.
1e9f344 issue #426: big hack so reset_connection has task_var access
18bfde5 issue #444: update Changelog.
835bead tests: allow running scripts from any subdir.
81c93e1 ci: remove duplicate /usr/bin/time call
ca9ae45 issue #426: TemporaryEnvironment must coerce to Unicode.
4bc0d0e issue #426: apply_mode_spec() must handle bytes.
a8921bb tests: fix scaling in fork_histogram
374a361 docs: try to fix CSS difference between local and rtfd
861be2e docs: wrap text around logos
b084d83 docs: fit SVG viewbox to ansible logo
98d06e2 docs: delete shame.rst to make room for new chapters.
5f3244a docs: import pcaps (using LFS) to regenerate charts.
6936b93 tests: import fork_histogram.py.
5a96d13 issue #426: fix all.yml sorting, one more delegate_to
bd82fa1 issue #426: fix low_level_execute_command.yml breakage.
a6e6bc4 issue #426: to_text filter.
d15f533 Turn on Travis build notifcations, but send them to IRC.
9d87f03 issue #426: disable Ansible smart transport.
49d37bf issue #426: remove LANG and LC_ALL during tests.
a6e6fd1 issue #426: more 2->3 test fixes.
67f710f issue #426: use delegate_to in fixup_perms2 and copy.yml
a67a436 docs: add #374 to Changelog.
2b229a6 docs: add thanks entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants