Skip to content

ansible 2.6 / python3: CallError: builtins.TypeError: the JSON object must be str, not 'bytes' #374

Closed

Description

@candlerb

Test case:

==> test.yml <==
- hosts: gold-master
  tasks:
    - raw: echo hello world

==> hosts.yml <==
all:
  vars:
    ansible_python_interpreter: /usr/bin/python3
  hosts:
    gold-master:
      ansible_connection: lxd

==> ansible.cfg <==
[defaults]
inventory = hosts.yml
#strategy_plugins = /usr/local/share/mitogen/ansible_mitogen/plugins/strategy
#strategy = mitogen_linear

gold-master is an ubuntu 16.04 lxd container.

Works with mitogen commented out. With mitogen enabled, I get:

$ ansible-playbook test.yml

PLAY [gold-master] ******************************************************************************************************************

TASK [Gathering Facts] **************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was:     s.__class__.__name__))
fatal: [gold-master]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
	to retry, use: --limit @/home/ubuntu/ansible/misc/test.retry

PLAY RECAP **************************************************************************************************************************
gold-master                : ok=0    changed=0    unreachable=0    failed=1

With -vvv there is an exception traceback shown:

[pid 3603] 11:13:12.040260 D mitogen: mitogen.core.Stream(u'unix_client.3622').on_disconnect()
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 139, in run
    res = self._execute()
  File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 584, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/mixins.py", line 116, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/normal.py", line 46, in run
    result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/mixins.py", line 334, in _execute_module
    timeout_secs=self.get_task_timeout_secs(),
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/planner.py", line 484, in invoke
    kwargs=planner.get_kwargs(),
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/connection.py", line 428, in call
    return self._rethrow(recv)
  File "/usr/local/share/mitogen-candlerb-371/ansible_mitogen/connection.py", line 415, in _rethrow
    return recv.get().unpickle()
  File "/usr/local/share/mitogen-candlerb-371/mitogen/core.py", line 595, in unpickle
    raise obj
CallError: builtins.TypeError: the JSON object must be str, not 'bytes'
  File "<stdin>", line 1993, in _dispatch_one
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/target.py", line 353, in run_module
    impl = klass(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 633, in __init__
    super(NewStyleRunner, self).__init__(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 578, in __init__
    super(ScriptRunner, self).__init__(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 445, in __init__
    super(ProgramRunner, self).__init__(**kwargs)
  File "master:/usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py", line 252, in __init__
    self.args = json.loads(json_args)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))

Looks like a bytes to str conversion is missing somewhere.

I tried a band-aid of json.loads(str(json_args)), but this then gives a different error UPDATE: see below.

Activity

  1. RonnyPfannschmidt commented on Sep 18, 2018

    @RonnyPfannschmidt

    please try to show the content of the json-value as well

  2. candlerb commented on Sep 18, 2018

    @candlerb
    ContributorAuthor

    It was my mistake, I should have used decode() not str().

    With the following sticking plaster it works:

    --- /usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py.orig	2018-09-18 11:21:52.414854084 +0000
    +++ /usr/local/share/mitogen-candlerb-371/ansible_mitogen/runner.py	2018-09-18 11:56:20.106790770 +0000
    @@ -249,7 +249,7 @@
             self.service_context = service_context
             self.econtext = econtext
             self.detach = detach
    -        self.args = json.loads(json_args)
    +        self.args = json.loads(json_args.decode("UTF-8"))
             self.temp_dir = temp_dir
             self.extra_env = extra_env
             self.env = env
    

    However, I imagine that really it's the responsibility of the caller of this function to pass json_args as as a string, not bytes.

    FWIW, I did also log json_args. The value was:

    b'{"_ansible_version": "2.6.4", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p"], "_ansible_no_log": false, "gather_timeout": 10, "_ansible_module_name": "setup", "_ansible_debug": false, "_ansible_verbosity": 3, "_ansible_keep_remote_files": false, "_ansible_syslog_facility": "LOG_USER", "_ansible_socket": null, "gather_subset": "all", "_ansible_diff": false, "_ansible_remote_tmp": "~/.ansible/tmp", "_ansible_shell_executable": "/bin/sh", "_ansible_check_mode": false, "_ansible_tmpdir": "/root/.ansible/tmp/ansible_mitogen_l9nzvee2/worker-4388-7ff4cc7b1650"}'
    
  3. dw commented on Sep 19, 2018

    @dw
    Member

    Thanks for reporting this. I'll add something to CI, because that change needs tested in both directions.

  4. dw commented on Oct 3, 2018

    @dw
    Member

    The problem is specific to Python>2,<3.6:

    Changed in version 3.6: s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.

    This is a minor pain, I need to add another CI target to cover 3.5. There is an annoying balance between speedy but inaccurate or dog-slow (and thus forgotten/ignored) CI

  5. dw commented on Nov 7, 2018

    @dw
    Member

    Sorry for the crazy delay - discovered a can of worms around Python 3.x in the process! You may wish to consider subscribing to #426 in the meantime.

    The fix for this was committed in 16911c9.

    This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

    Thanks for reporting this!

  6. added a commit that references this issue on Nov 7, 2018
  7. added a commit that references this issue on Nov 23, 2018
  8. added 2 commits that reference this issue on Jan 19, 2019
  9. added a commit that references this issue on Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions