Templating broken when constructing value for `ansible_ssh_common_args`

[dnmvisser]

Hi, I'm on ansible-core-2.12.2 (thx for all the work in getting that done) and mitogen v0.3.2.

We have some basic jinja inside one of our vars files:

---
# Use the correct jump host
ansible_ssh_common_args: >-
  -o ProxyCommand='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p admin@{{ hostvars.jumphost.public_ip_address }}'

This causes errors:

TASK [Waiting for connection] *********************************************************************************************************************************************************
task path: /Users/dick.visser/git/deploy_dick/data/acc/site.yml:417
[WARNING]: Unhandled error in Python interpreter discovery for host acc_proxy1: EOF on stream; last 100 lines received: ssh: Could not resolve hostname {{: nodename nor servname
provided, or not known  kex_exchange_identification: Connection closed by remote host

If I hardcode it like this:

---
# Use the correct jump host
ansible_ssh_common_args: >-
  -o ProxyCommand='ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p admin@18.13.233.18'

then things work....

The jinja inside the inventory works fine with ansible v3.4.0 (ansible-base 2.10.x)

Any thoughts?

[hungpr0]

I have the same issue with ansible core 2.11.8, mitogen 0.3.2

[Zocker1999NET]

same issue here, Ansible Core 2.10.17, mitogen 0.3.2 & 0.3.1

[Tristan971]

Occured here too with Ansible 2.11.9 and mitogen 0.3.2 (well, master, more specifically) due to this fella https://github.com/kubernetes-sigs/kubespray/blob/master/roles/kubespray-defaults/defaults/main.yaml#L4

[Zocker1999NET]

with a small test inventory, Ansible 2.10 and git bisect, I found commit c61c063 to be the reason for this bug. Unfortunately this commit is quite big, so I try to manually find the exact reason to maybe find a bugfix / workaround …

[hungpr0]

with a small test inventory, Ansible 2.10 and git bisect, I found commit c61c063 to be the reason for this bug. Unfortunately this commit is quite big, so I try to manually find the exact reason to maybe find a bugfix / workaround …

It may be ansible_mitogen/transport_config.py

[Zocker1999NET]

It may be ansible_mitogen/transport_config.py

You're right, any other change of this commit does not affect the outcome of my tests. But, it may also be that an internal change in Ansible that (also) causes this bug, but I'm not quite sure:

While trying to find the smallest partial revert of commit c61c063, I detected a difference in the result of my small ping test depending on the version of Ansible used.

Beginning from tag v0.3.2, after applying the diff at the end (which reverts the commit partially), running ansible -m ping host with a small test inventory works for Ansible 2.10 as expected but stops working for Ansible 5.4.0 (core 2.12.3) with the same error message:

host | UNREACHABLE! => {
    "changed": false,
    "msg": "EOF on stream; last 100 lines received:\nssh: Could not resolve hostname {%: Name or service not known\r",
    "unreachable": true
}

So partially reverting this change does work for older Ansible versions (~ 2.10) but not for newer ones (~ 5.4.0 / 2.12.3).

This is the diff from the mention above:

diff --git a/ansible_mitogen/transport_config.py b/ansible_mitogen/transport_config.py
index 4babbde3..344c3d84 100644
--- a/ansible_mitogen/transport_config.py
+++ b/ansible_mitogen/transport_config.py
@@ -467,9 +467,9 @@ class PlayContextSpec(Spec):
         return [
             mitogen.core.to_text(term)
             for s in (
-                C.config.get_config_value("ssh_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
-                C.config.get_config_value("ssh_common_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
-                C.config.get_config_value("ssh_extra_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {}))
+                getattr(self._play_context, 'ssh_args', ''),
+                getattr(self._play_context, 'ssh_common_args', ''),
+                getattr(self._play_context, 'ssh_extra_args', '')
             )
             for term in ansible.utils.shlex.shlex_split(s or '')
         ]
@@ -696,9 +696,22 @@ class MitogenViaSpec(Spec):
         return [
             mitogen.core.to_text(term)
             for s in (
-                C.config.get_config_value("ssh_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
-                C.config.get_config_value("ssh_common_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),
-                C.config.get_config_value("ssh_extra_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {}))
+                (
+                    self._host_vars.get('ansible_ssh_args') or
+                    getattr(C, 'ANSIBLE_SSH_ARGS', None) or
+                    os.environ.get('ANSIBLE_SSH_ARGS')
+                    # TODO: ini entry. older versions.
+                ),
+                (
+                    self._host_vars.get('ansible_ssh_common_args') or
+                    os.environ.get('ANSIBLE_SSH_COMMON_ARGS')
+                    # TODO: ini entry.
+                ),
+                (
+                    self._host_vars.get('ansible_ssh_extra_args') or
+                    os.environ.get('ANSIBLE_SSH_EXTRA_ARGS')
+                    # TODO: ini entry.
+                ),
             )
             for term in ansible.utils.shlex.shlex_split(s)
             if s

[guytet]

Encountering the same issue

ansible 2.10.17
mitogen-0.3.2

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=3m -o ForwardAgent=yes
control_path = ~/.ssh/ansible-%%C

ansible_ssh_jumphost: "{{ hostvars[groups['jumphost_servers'][0]]['ansible_host'] }}"
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q {{ ansible_ssh_user }}@{{ ansible_ssh_jumphost }}"'

kex_exchange_identification: Connection closed by remote host

[guytet]

@dnmvisser Have you found a working solution for this issue ? mitogen is a very useful part of our toolset, we'd love to hear if there's a way to make this work. Thank you.

[dnmvisser]

@dnmvisser Have you foung a working solution for this issue ? mitogen is a very useful part of our toolset, we'd love to hear if there's a way to make this work. Thank you.

Nope, I ended up creating files with hard coded IP addresses etc

[guytet]

@dnmvisser Have you foung a working solution for this issue ? mitogen is a very useful part of our toolset, we'd love to hear if there's a way to make this work. Thank you.

Nope, I ended up creating files with hard coded IP addresses etc

We finally got this to work with this combination of versions:

ansible --version
ansible 2.10.17

mitogen-0.3.0rc1

[Zocker1999NET]

I took the time to inspect further and found a difference in the calling of C.config.get_config_value between Ansible and Mitogen.

For getting the configuration of ssh_common_args, Mitogen calls:

mitogen/ansible_mitogen/transport_config.py

Line 478 in 89c0cc9

C.config.get_config_value("ssh_common_args", plugin_type="connection", plugin_name="ssh", variables=self._task_vars.get("vars", {})),

Ansible plugins (here ssh) use a helper AnsiblePlugin.get_option which does (if GitHub does not render the code, click on the links):

https://github.com/ansible/ansible/blob/b104478f171a4030c0cd96ef4d99db65bf04dceb/lib/ansible/plugins/connection/ssh.py#L743-L744

https://github.com/ansible/ansible/blob/b104478f171a4030c0cd96ef4d99db65bf04dceb/lib/ansible/plugins/__init__.py#L55-L62

Intercepting these calls to get_config_value reveals, that the calls from the official ssh plugin sets the argument variables to a dict containing all host variables already resolved (a.k.a. not in their template form after Jinja2). However Mitogen's connection plugin sets the argument to a dict containing the probably the task variables unresolved (a.k.a. in their template form before Jinja2).

Meaning in practice: Given these example host vars:

ansible_ssh_common_args: "{{ other_var }}"
other_var: "--my-option"

Then the argument variables of get_config_value looks like

• {…, "ansible_ssh_common_args": "--my-option", …} if called from Ansible's ssh plugin
• {…, "ansible_ssh_common_args": "{{ other var }}", …} if called from Mitogen's connection plugin

I do not know Ansible's Python code good enough to fix this, probably by resolving the variables properly before passing them to get_config_value, but maybe this helps someone else.

[maxpain]

Any updates?

[maxpain]

@moreati could you please look into this issue?