network_cli connections failing with "could not recover task_vars"

[craigers521]

Running ansible version 2.9.11 on macOS with python 3.8, with latest mitogen version from master, targeting cisco isr routers and other network_cli based connection devices. All playbooks fail with this error:

<10.119.122.254> attempting to start connection
<10.119.122.254> using connection plugin network_cli
<10.119.130.254> attempting to start connection
<10.119.130.254> using connection plugin network_cli
<10.119.122.254> local domain socket does not exist, starting it
<10.119.122.254> control socket path is /Users/cse015/.ansible/pc/92ada5d152
<10.119.122.254> local domain socket listeners started successfully
<10.119.122.254> loaded cliconf plugin ios from path /Users/cse015/python-virtual-environments/ansible29/lib/python3.8/site-packages/ansible/plugins/cliconf/ios.py for network_os ios
<10.119.122.254>
<10.119.122.254> local domain socket path is /Users/cse015/.ansible/pc/92ada5d152
<10.119.130.254> local domain socket does not exist, starting it
<10.119.130.254> control socket path is /Users/cse015/.ansible/pc/77c5115e9c
<10.119.130.254> local domain socket listeners started successfully
<10.119.130.254> loaded cliconf plugin ios from path /Users/cse015/python-virtual-environments/ansible29/lib/python3.8/site-packages/ansible/plugins/cliconf/ios.py for network_os ios
<10.119.130.254>
<10.119.130.254> local domain socket path is /Users/cse015/.ansible/pc/77c5115e9c
fatal: [10.119.122.254]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true
}
fatal: [10.119.130.254]: UNREACHABLE! => {
    "changed": false,
    "msg": "could not recover task_vars. This means some connection settings may erroneously be reset to their defaults. Please report a bug if you encounter this message.",
    "unreachable": true

I also tried running with ansible 2.10 and the latest collectionSupport branch from the s1113950 fork and i was met with a different error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 613, in _execute
    self._connection = self._get_connection(cvars, templar)
  File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 908, in _get_connection
    connection, plugin_load_context = self._shared_loader_obj.connection_loader.get_with_context(
  File "/Users/cse015/.ansible/plugins/mitogen-dev/ansible_mitogen/strategy.py", line 170, in wrap_connection_loader__get
    return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/loader.py", line 826, in get_with_context
    obj.__init__(instance, *args, **kwargs)
  File "/Users/cse015/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 324, in __init__
    super(Connection, self).__init__(
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 290, in __init__
    self._update_connection_state()
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/connection/__init__.py", line 368, in _update_connection_state
    control_path = ssh._create_control_path(
  File "/Users/cse015/.ansible/plugins/mitogen-dev/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 67, in _create_control_path
    return Connection.vanilla_class._create_control_path(*args, **kwargs)
AttributeError: 'get_with_context_result' object has no attribute '_create_control_path'
fatal: [10.119.130.254]: FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""

[s1113950]

@craigers521 thanks for testing this on the #715 branch. Is there a simple way to reproduce this using another network_cli-based task? I don't have a cisco router available to test with.

[isundaylee]

I ran into this same issue AttributeError: 'get_with_context_result' object has no attribute '_create_control_path' for a vyos_config task. Ansible 2.10.6 with Mitogen 0.3.0rc1.

Not sure if this qualifies as a "simple way to reproduce": but Vyos can be installed on a virtual machine, which can be used to test network_cli tasks against.

[s1113950]

@isundaylee do you have a vagrant file or something of the like to create a vyos installation on a vm? I don't have the time to look into reproducing with Vyos at the moment unfortunately.

[lorephoenix]

Hello
I am having a similar issue running Ansible 2.10 using Mitogen 0.30rc1
ansible-galaxy collection list
Collection Version

---

ansible.netcommon 1.4.1
ansible.posix 1.1.1
cisco.ios 1.2.1

The issue occur when gathering facts

• name: "main | Gathering Facts!"
  cisco.ios.ios_facts:
  gather_subset:
  - hardware

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 158, in run
res = self._execute()
File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 613, in _execute
self._connection = self._get_connection(cvars, templar)
File "/usr/lib/python3.6/site-packages/ansible/executor/task_executor.py", line 913, in _get_connection
ansible_playbook_pid=to_text(os.getppid())
File "/home/deployer/ansible/plugins/mitogen-0.3.0rc1/ansible_mitogen/strategy.py", line 168, in wrap_connection_loader__get
return ansible_mitogen.loaders.connection_loader__get(name, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/ansible/plugins/loader.py", line 826, in get_with_context
obj.init(instance, *args, **kwargs)
File "/home/deployer/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 332, in init
play_context, new_stdin, *args, **kwargs
File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/init.py", line 290, in init
self._update_connection_state()
File "/usr/lib/python3.6/site-packages/ansible/plugins/connection/init.py", line 371, in _update_connection_state
self._ansible_playbook_pid
File "/home/deployer/ansible/plugins/mitogen-0.3.0rc1/ansible_mitogen/plugins/connection/mitogen_ssh.py", line 67, in _create_control_path
return Connection.vanilla_class._create_control_path(*args, **kwargs)
AttributeError: 'get_with_context_result' object has no attribute '_create_control_path'
fatal: [hostname]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}

[MrLucifer11]

Hi,

I have the same issue with ansible 2.10.3 and Mitogen 0.30rc1.
Has anyone found a solution please ?

[XuNiLuS]

Hello,

I have the same issue with Ansible 3.4.0 (Based 2.10.13) and Mitogen 0.30rc1.
is there a solution?

[arhue]

Same issue with Ansible 2.13.1 and Mikrotik Router OS 7.5.

[markafarrell]

I have been able to reproduce this issue with the following https://github.com/markafarrell/mitogen-issue-766-repro

[RomyxBaps]

I have you find a solution ? I have the problem for a rsync tasks...