juniper_junos_system fails when (vmhost) rebooting MX204 router

[tb90ka]

Issue Type

• Bug Report

Module Name

juniper_junos_system

$ ansible --version
ansible 2.5.1
  config file = /home/username/repos/git/ansible/ansible.cfg
  configured module search path = [u'/home/username/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.17 (default, Jul 20 2020, 15:37:01) [GCC 7.5.0]

$ pip freeze
ansible==2.5.1
apache-libcloud==2.2.1
apsw==3.16.2.post1
asn1crypto==0.24.0
beautifulsoup4==4.6.0
capirca==1.109
certifi==2018.1.18
chardet==3.0.4
Cheetah==2.4.4
CherryPy==8.9.1
cryptography==2.1.4
cssselect==1.0.3
cssutils==1.0.2
defusedxml==0.5.0
dimclient==0.4.2
dnspython==1.15.0
enum34==1.1.6
feedparser==5.2.1
gdata==2.0.18
html5-parser==0.4.4
html5lib==0.999999999
httplib2==0.9.2
idna==2.6
ipaddress==1.0.17
IPy==0.83
Jinja2==2.10
jmespath==0.9.3
junos-eznc==2.1.7
keyring==10.6.0
keyrings.alt==3.0
lockfile==0.12.2
lxml==4.2.1
Markdown==2.6.9
MarkupSafe==1.0
mechanize==0.2.5
msgpack==0.5.6
mysqlclient==1.3.10
ncclient==0.5.3
ndcli==2.11.0
netaddr==0.7.19
netifaces==0.10.4
numpy==1.13.3
olefile==0.45.1
paramiko==2.0.0
Pillow==5.1.0
psutil==5.4.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycairo==1.16.2
pycrypto==2.6.1
Pygments==2.2.0
pygobject==3.26.1
pykerberos==1.1.14
pyliblzma==0.5.3
pyOpenSSL==17.5.0
pyparsing==2.2.0
pyserial==3.4
python-dateutil==2.6.1
python-ldap==3.0.0
python-yubico==1.3.2
pyusb==1.0.0
pyxdg==0.25
PyYAML==3.12
regex==2017.12.12
repoze.lru==0.7
requests==2.18.4
Routes==2.4.1
scour==0.36
scp==0.10.2
SecretStorage==2.3.1
simplejson==3.13.2
six==1.11.0
SOAPpy==0.12.22
ui-zeus==0.3.0
urllib3==1.22
uTidylib==0.3
webencodings==0.5
WebOb==1.7.3
weboob==1.2
wstools==0.4.3
xmltodict==0.11.0

Ansible roles version

Juniper.Junos, Version 2.4.0

OS / Environment

JUNOS 19.3R1.8

Summary

When trying to reboot a Juniper MX204 router using ansible, it first failed with an error:

    "changed": false, 
    "invocation": {
        "module_args": {
            "action": "reboot", 
            "all_re": false, 
            "at": null, 
            "attempts": null, 
            "baud": null, 
            "console": null, 
            "cs_passwd": null, 
            "cs_user": null, 
            "host": "gw-mx204-test", 
            "in_min": null, 
            "level": null, 
            "logdir": null, 
            "logfile": null, 
            "media": false, 
            "mode": null, 
            "other_re": false, 
            "passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 22, 
            "ssh_config": null, 
            "ssh_private_key_file": null, 
            "timeout": 120, 
            "user": "tbanghard", 
            "vmhost": true
        }
    }, 
    "msg": "The vmhost option can only be used when the action option has the value \"reboot\"."
}

After I patched the .ansible/roles/Juniper.junos/library/juniper_junos_system.py role to:

<     if action == 'reboot' and vmhost is True:
>     if action != 'reboot' and vmhost is True:

... the playbook failed with below error message:

fatal: [gw-mx204-test]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_5j16sqqh/ansible_module_juniper_junos_system.py\", line 399, in <module>\n    main()\n  File \"/tmp/ansible_5j16sqqh/ansible_module_juniper_junos_system.py\", line 356, in main\n    got = junos_module.sw.reboot(in_min, at, all_re, None, vmhost, other_re)\nAttributeError: 'JuniperJunosModule' object has no attribute 'sw'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}

Steps to reproduce

inventory file: lab

[lab]
gw-mx204-test

group_vars/all

---
ansible_python_interpreter: /usr/bin/env python3

connection_settings:
  host: "{{ ansible_host }}"
  port: "22"
  passwd: "{{ password }}"
  timeout: 120

reboot.yaml

---
- name: Reboot Junos OS
  hosts: "gw-mx204-test"
  roles:
    - Juniper.junos
  connection: local
  gather_facts: no

  tasks:
    - name: Reboot router after upgrade
      juniper_junos_system:
        provider: "{{ connection_settings }}"
        all_re: false
        vmhost: true
        action: "reboot"
      register: sw

    - name: print response
      debug:
        var: sw

  vars_prompt:
    - name: password
      prompt: Junos Password
      private: yes
      run_once: yes

Run playbook:

ansible-playbook reboot.yaml -i lab

Expected results

Device reboots and playbook ends.

Actual results

Playbook fails:

$ ansible-playbook reboot.yaml -e "hostname=gw-mx204-test" -vvv
ansible-playbook 2.5.1
  config file = /home/username/git/ansible/ansible.cfg
  configured module search path = [u'/home/username/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.17 (default, Jul 20 2020, 15:37:01) [GCC 7.5.0]
Using /home/username/git/ansible/ansible.cfg as config file
Parsed /home/username/git/ansible/lab inventory source with ini plugin

PLAYBOOK: reboot.yaml *********************************************************************************************************************************************************************************************
1 plays in reboot.yaml
Junos Password: 

PLAY [Reboot Junos OS] ********************************************************************************************************************************************************************************************
META: ran handlers

TASK [Reboot switch after upgrade] ********************************************************************************************************************************************************************************
task path: /home/username/git/ansible/reboot.yaml:10
Using module file /home/username/.ansible/roles/Juniper.junos/library/juniper_junos_system.py
<gw-mx204-test> ESTABLISH LOCAL CONNECTION FOR USER: username
<gw-mx204-test> EXEC /bin/sh -c 'echo ~ && sleep 0'
<gw-mx204-test> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/username/.ansible/tmp/ansible-tmp-1596089007.84-203005424292722 `" && echo ansible-tmp-1596089007.84-203005424292722="` echo /home/username/.ansible/tmp/ansible-tmp-1596089007.84-203005424292722 `" ) && sleep 0'
<gw-mx204-test> PUT /home/username/.ansible/tmp/ansible-local-6456tXfj6f/tmpUqXGp7 TO /home/username/.ansible/tmp/ansible-tmp-1596089007.84-203005424292722/juniper_junos_system.py
<gw-mx204-test> EXEC /bin/sh -c 'chmod u+x /home/username/.ansible/tmp/ansible-tmp-1596089007.84-203005424292722/ /home/username/.ansible/tmp/ansible-tmp-1596089007.84-203005424292722/juniper_junos_system.py && sleep 0'
<gw-mx204-test> EXEC /bin/sh -c '/usr/bin/env python3 /home/username/.ansible/tmp/ansible-tmp-1596089007.84-203005424292722/juniper_junos_system.py && sleep 0'
<gw-mx204-test> EXEC /bin/sh -c 'rm -f -r /home/username/.ansible/tmp/ansible-tmp-1596089007.84-203005424292722/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_5j16sqqh/ansible_module_juniper_junos_system.py", line 399, in <module>
    main()
  File "/tmp/ansible_5j16sqqh/ansible_module_juniper_junos_system.py", line 356, in main
    got = junos_module.sw.reboot(in_min, at, all_re, None, vmhost, other_re)
AttributeError: 'JuniperJunosModule' object has no attribute 'sw'

fatal: [gw-mx204-test]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_5j16sqqh/ansible_module_juniper_junos_system.py\", line 399, in <module>\n    main()\n  File \"/tmp/ansible_5j16sqqh/ansible_module_juniper_junos_system.py\", line 356, in main\n    got = junos_module.sw.reboot(in_min, at, all_re, None, vmhost, other_re)\nAttributeError: 'JuniperJunosModule' object has no attribute 'sw'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
        to retry, use: --limit @/home/username/git/ansible/reboot.retry

PLAY RECAP ********************************************************************************************************************************************************************************************************
gw-mx204-test : ok=0    changed=0    unreachable=0    failed=1

[rahkumar651991]

Hi @tb90ka
You will need to upgrade junos-eznc.
juniper.junos 2.4.0 requires junos-eznc >= 2.5.0. https://github.com/Juniper/ansible-junos-stdlib/releases/tag/2.4.0 .

your current junos-eznc version is 2.1.7. upgrade it and tell if it works fine for you.

[tb90ka]

Hello @rahkumar651991
first of all thanks for your support!

I've upgraded junos-eznc as described:

$ pip freeze | grep junos
junos-eznc==2.5.1

But unfortunately I'm still getting the same error when running the playbook:

The full traceback is:
Traceback (most recent call last):
  File "/home/username/.ansible/tmp/ansible-tmp-1596613954.85-9682-31691744305487/AnsiballZ_juniper_junos_system.py", line 102, in <module>
    _ansiballz_main()
  File "/home/username/.ansible/tmp/ansible-tmp-1596613954.85-9682-31691744305487/AnsiballZ_juniper_junos_system.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/username/.ansible/tmp/ansible-tmp-1596613954.85-9682-31691744305487/AnsiballZ_juniper_junos_system.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible.modules.juniper_junos_system', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.6/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_juniper_junos_system_payload_ea2otdnh/ansible_juniper_junos_system_payload.zip/ansible/modules/juniper_junos_system.py", line 399, in <module>
  File "/tmp/ansible_juniper_junos_system_payload_ea2otdnh/ansible_juniper_junos_system_payload.zip/ansible/modules/juniper_junos_system.py", line 356, in main
AttributeError: 'JuniperJunosModule' object has no attribute 'sw'
fatal: [gw-mx204-test]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/username/.ansible/tmp/ansible-tmp-1596613954.85-9682-31691744305487/AnsiballZ_juniper_junos_system.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/home/username/.ansible/tmp/ansible-tmp-1596613954.85-9682-31691744305487/AnsiballZ_juniper_junos_system.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/username/.ansible/tmp/ansible-tmp-1596613954.85-9682-31691744305487/AnsiballZ_juniper_junos_system.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.juniper_junos_system', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_juniper_junos_system_payload_ea2otdnh/ansible_juniper_junos_system_payload.zip/ansible/modules/juniper_junos_system.py\", line 399, in <module>\n  File \"/tmp/ansible_juniper_junos_system_payload_ea2otdnh/ansible_juniper_junos_system_payload.zip/ansible/modules/juniper_junos_system.py\", line 356, in main\nAttributeError: 'JuniperJunosModule' object has no attribute 'sw'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", 
    "rc": 1
}

I've also tried to remove the role and install it again (via ansible-galaxy) but then I had to patch the roles ( .ansible/roles/Juniper.junos/library/juniper_junos_system.py) code again:

<     if action == 'reboot' and vmhost is True:
>     if action != 'reboot' and vmhost is True:

[tb90ka]

Thanks @rahkumar651991 for fixing this!
I've overwritten the old roles via ansible-galaxy:

$ sudo ansible-galaxy install git+https://github.com/Juniper/ansible-junos-stdlib.git,roles,Juniper.junos --force

Reboot now seems to work, even though I still get a (somewhat strange) error message:

fatal: [gw-mx204-test]: FAILED! => {
    "action": "reboot", 
    "all_re": false, 
    "changed": false, 
    "invocation": {
        "module_args": {
            "action": "reboot", 
            "all_re": false, 
            "at": null, 
            "attempts": null, 
            "baud": null, 
            "console": null, 
            "cs_passwd": null, 
            "cs_user": null, 
            "host": "gw-gsp.nwlab.bs.kae.de.oneandone.net", 
            "in_min": 0, 
            "level": null, 
            "logdir": null, 
            "logfile": null, 
            "media": false, 
            "mode": null, 
            "other_re": false, 
            "passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 22, 
            "ssh_config": null, 
            "ssh_private_key_file": null, 
            "timeout": 120, 
            "user": "$username", 
            "vmhost": true
        }
    }, 
    "media": false, 
    "msg": "Did not find expected RPC response.", 
    "other_re": false, 
    "reboot": true, 
    "vmhost": true
}

The playbook fails, but the device is still rebooting.
Could you please check what's wrong here?

The respective task for this:

    - name: Reboot switch after upgrade
      juniper_junos_system:
        provider: "{{ connection_settings }}"
        vmhost: true
        in_min: 0
        all_re: false
        action: "reboot"
      register: swreboot

Thanks in advance,
tb90ka

[rahkumar651991]

@tb90ka - Can you share the netconf trace on the device for this issue.
you may unicast it to jnpr-community-netdev@juniper.net .
Will check more and get back. Also share the output displayed by the playbook with -v(verbose mode) option.

For enabling netconf trace on device, refer this.
https://www.juniper.net/documentation/en_US/junos/topics/example/netconf-traceoptions-configuring.html

[tb90ka]

@rahkumar651991 information has been provided as requested.

[rahkumar651991]

@tb90ka https://github.com/Juniper/py-junos-eznc/pull/1066/files the change is in PyEZ(junos-eznc) for this. this should fix it. We will release new version for both with the fixes soon.

[tb90ka]

This issues has been fixed meanwhile.

Thanks for your support!