blockinfile errors with Python3: TypeError: startswith first arg must be bytes or a tuple of bytes, not str

[Napsty]

Summary

When I try to run a blockinfile task with Python3 on the target, the following error is showing up:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: startswith first arg must be bytes or a tuple of bytes, not str
fatal: [target.example.com]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 10.10.1.151 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637750429.0-17572-233380936890987/AnsiballZ_blockinfile.py\", line 102, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637750429.0-17572-233380936890987/AnsiballZ_blockinfile.py\", line 94, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637750429.0-17572-233380936890987/AnsiballZ_blockinfile.py\", line 40, in invoke_module\r\n    runpy.run_module(mod_name='ansible.modules.blockinfile', init_globals=None, run_name='__main__', alter_sys=True)\r\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_blockinfile_payload_4goytdsn/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 268, in <module>\r\n  File \"/tmp/ansible_blockinfile_payload_4goytdsn/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 215, in main\r\nTypeError: startswith first arg must be bytes or a tuple of bytes, not str\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

If I set the ansible_python_interpreter of the target to Python 2.7, the task works.

Maybe related: #76027

Issue Type

Bug Report

Component Name

blockinfile

Ansible Version

$ ansible --version
ansible 2.9.27
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/srv/ansible/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.17 (default, Jul 20 2020, 15:37:01) [GCC 7.5.0]

Configuration

$ ansible-config dump --only-changed
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/srv/ansible/inventory']
DEFAULT_ROLES_PATH(/etc/ansible/ansible.cfg) = ['/srv/ansible/roles']
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False

OS / Environment

Ubuntu 18.04

Steps to Reproduce

Simple blockinfile task:

  - name: FILEBEAT - Configure Docker Logging
    blockinfile:
      dest: /etc/filebeat/filebeat.yml
      insertafter: 'filebeat.inputs:'
      marker: "# {mark} -- Docker logging configured by Ansible"
      block: |
        - type: docker
          containers.ids:
            - '*'
          fields:
            my_app: docker
            my_env: {{ nzzenv }}
          fields_under_root: true
          exclude_lines: ['ingress']
          ignore_older: 12h

Run playbook with the task results in error:

TASK [FILEBEAT - Configure Docker Logging] ************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: startswith first arg must be bytes or a tuple of bytes, not str
fatal: [target.example.com]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 10.10.1.151 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637737961.16-28686-16453558107895/AnsiballZ_blockinfile.py\", line 102, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637737961.16-28686-16453558107895/AnsiballZ_blockinfile.py\", line 94, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637737961.16-28686-16453558107895/AnsiballZ_blockinfile.py\", line 40, in invoke_module\r\n    runpy.run_module(mod_name='ansible.modules.blockinfile', init_globals=None, run_name='__main__', alter_sys=True)\r\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_blockinfile_payload_ss2veys_/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 268, in <module>\r\n  File \"/tmp/ansible_blockinfile_payload_ss2veys_/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 215, in main\r\nTypeError: startswith first arg must be bytes or a tuple of bytes, not str\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Changing the target in the inventory to use Python 2.7 instead:

target.example.com ansible_ssh_host=10.10.1.151 ansible_python_interpreter=/usr/bin/python2.7

And the task is executed successfully:

TASK [FILEBEAT - Configure Docker Logging] ************************************************************************************************************************************************
changed: [target.example.com]

Expected Results

Task should be executed without an error.

Actual Results

TASK [FILEBEAT - Configure Docker Logging] ************************************************************************************************************************************************
task path: /srv/ansible/playbooks/filebeat/filebeat.yaml:153
<10.10.1.151> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.1.151> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/srv/ansible/.ansible/cp/d446614f06 10.10.1.151 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<10.10.1.151> (0, b'/home/ansible\n', b'')
<10.10.1.151> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.1.151> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/srv/ansible/.ansible/cp/d446614f06 10.10.1.151 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ansible/.ansible/tmp `"&& mkdir "` echo /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164 `" && echo ansible-tmp-1637758062.9004042-21475-249993162057164="` echo /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164 `" ) && sleep 0'"'"''
<10.10.1.151> (0, b'ansible-tmp-1637758062.9004042-21475-249993162057164=/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164\n', b'')
Using module file /srv/ansible/roles/yaegashi.blockinfile/library/blockinfile.py
<10.10.1.151> PUT /srv/ansible/.ansible/tmp/ansible-local-191240mc1szby/tmppg7myivh TO /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py
<10.10.1.151> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/srv/ansible/.ansible/cp/d446614f06 '[10.10.1.151]'
<10.10.1.151> (0, b'sftp> put /srv/ansible/.ansible/tmp/ansible-local-191240mc1szby/tmppg7myivh /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py\n', b'')
<10.10.1.151> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.1.151> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/srv/ansible/.ansible/cp/d446614f06 10.10.1.151 '/bin/sh -c '"'"'chmod u+x /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/ /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py && sleep 0'"'"''
<10.10.1.151> (0, b'', b'')
<10.10.1.151> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.1.151> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/srv/ansible/.ansible/cp/d446614f06 -tt 10.10.1.151 '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-nnxdinoiyaccefgkkdlxkgihdgozsgjb ; /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<10.10.1.151> (1, b'Traceback (most recent call last):\r\n  File "/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py", line 100, in <module>\r\n    _ansiballz_main()\r\n  File "/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py", line 92, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File "/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py", line 40, in invoke_module\r\n    runpy.run_module(mod_name=\'ansible.modules.blockinfile\', init_globals=dict(_module_fqn=\'ansible.modules.blockinfile\', _modlib_path=modlib_path),\r\n  File "/usr/lib/python3.8/runpy.py", line 207, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File "/tmp/ansible_blockinfile_payload_9m42tynx/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py", line 268, in <module>\r\n  File "/tmp/ansible_blockinfile_payload_9m42tynx/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py", line 215, in main\r\nTypeError: startswith first arg must be bytes or a tuple of bytes, not str\r\n', b'Shared connection to 10.10.1.151 closed.\r\n')
<10.10.1.151> Failed to connect to the host via ssh: Shared connection to 10.10.1.151 closed.
<10.10.1.151> ESTABLISH SSH CONNECTION FOR USER: None
<10.10.1.151> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/srv/ansible/.ansible/cp/d446614f06 10.10.1.151 '/bin/sh -c '"'"'rm -f -r /home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/ > /dev/null 2>&1 && sleep 0'"'"''
<10.10.1.151> (0, b'', b'')
The full traceback is:
Traceback (most recent call last):
  File "/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py", line 100, in <module>
    _ansiballz_main()
  File "/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py", line 92, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.blockinfile', init_globals=dict(_module_fqn='ansible.modules.blockinfile', _modlib_path=modlib_path),
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_blockinfile_payload_9m42tynx/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py", line 268, in <module>
  File "/tmp/ansible_blockinfile_payload_9m42tynx/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py", line 215, in main
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
fatal: [target.example.com]: FAILED! => {
    "changed": false,
    "module_stderr": "Shared connection to 10.10.1.151 closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py\", line 100, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py\", line 92, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1637758062.9004042-21475-249993162057164/AnsiballZ_blockinfile.py\", line 40, in invoke_module\r\n    runpy.run_module(mod_name='ansible.modules.blockinfile', init_globals=dict(_module_fqn='ansible.modules.blockinfile', _modlib_path=modlib_path),\r\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\r\n    return _run_module_code(code, init_globals, run_name, mod_spec)\r\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\r\n    _run_code(code, mod_globals, init_globals,\r\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/tmp/ansible_blockinfile_payload_9m42tynx/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 268, in <module>\r\n  File \"/tmp/ansible_blockinfile_payload_9m42tynx/ansible_blockinfile_payload.zip/ansible/modules/blockinfile.py\", line 215, in main\r\nTypeError: startswith first arg must be bytes or a tuple of bytes, not str\r\n",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• lib/ansible/modules/blockinfile.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[mkrizek]

Can you provide a minimal version of /etc/filebeat/filebeat.yml that we can use to reproduce the issue?

Can you try updating to some of the more recent Ansible versions (2.12.0, 2.11.6, 2.10.15) to see if that resolves the issue?

needs_info

[Napsty]

I'm currently working on exactly that. But to my own surprise I have found that certain blockinfile tasks work. For example the following worked on the very same target host which threw the error in the filebeat.yml playbook:

---
- name: ANSIBLE - Blockinfile test
  hosts: '{{ target }}'
  tasks:
  
  ##########################
  # Set facts
  ##########################

  - name: Write a simple block into testfile
    blockinfile:
      dest: /tmp/blockinfile.test
      create: yes
      insertafter: "EOF"
      marker: "# {mark} -- configured by Ansible"
      block: |
        Write some text
        Write some more text
        Write even more text
        And again write some text

  - name: Write a YAML block into testfile
    blockinfile:
      dest: /tmp/blockinfile.test
      create: yes
      insertafter: "EOF"
      marker: "# {mark} -- yaml block"
      block: |
        - type: docker
          containers.ids:
            - '*'
          fields:
            my_app: docker
            my_env: {{ ansible_distribution_release }}
          fields_under_root: true
          exclude_lines: ['ingress']
          ignore_older: 12h

Trying to find out why right now.

I'm using the Ubuntu PPA for Ansible. Seems that newer Ansible versions haven't been packaged, can you confirm?

[Napsty]

OMG. I found it.

The filebeat playbook uses the yaegashi.blockinfile role. This is legacy stuff as we've been using Ansible for many years. I think blockinfile was added to builtin in 2.0.

- name: ANSIBLE - Filebeat installation and configuration
  hosts: '{{ target }}'
  roles:
    - yaegashi.blockinfile
[...]
  - name: FILEBEAT - Configure Docker Logging
    blockinfile:
      dest: /etc/filebeat/filebeat.yml
      insertafter: "filebeat.inputs:"
      marker: "# {mark} -- Docker logging configured by Ansible"
      block: |
        - type: docker
          containers.ids:
            - '*'
          fields:
            my_app: docker
            my_env: {{ nzzenv }}
          fields_under_root: true
          exclude_lines: ['ingress']
          ignore_older: 12h
    when: dockerioinstalled.rc == 0 or dockerceinstalled.rc == 0

I removed the roles: part and the playbook runs through just fine now:

TASK [FILEBEAT - Configure Docker Logging] *****************************************************************************
changed: [target.example.com]

Sorry for the noise!