Incompatibility with Ansible 4+ due to API change

[TheMysteriousX]

Issue Type

• Bug Report

Module Name

juniper.device.rpc

juniper.device collection and Python libraries version

$ ansible --version
ansible [core 2.13.1]
  config file = /builds/operations/network/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.10 (main, Mar  2 2022, 04:23:34) [GCC 10.2.1 20210110]
  jinja version = 3.1.2
  libyaml = True

$ pip freeze
absl-py==1.1.0
ansible==6.1.0
ansible-core==2.13.1
bcrypt==3.2.2
capirca==2.0.6
cffi==1.15.1
cfgv==3.3.1
click==8.1.3
cryptography==37.0.4
distlib==0.3.4
filelock==3.7.1
ghp-import==2.1.0
identify==2.5.1
importlib-metadata==4.12.0
ipaddress==1.0.23
Jinja2==3.1.2
junos-eznc==2.6.4
jxmlease==1.0.3
lxml==4.9.1
Markdown==3.3.7
MarkupSafe==2.1.1
mergedeep==1.3.4
mkdocs==1.3.0
mkdocs-material==8.3.9
mkdocs-material-extensions==1.0.3
mock==4.0.3
ncclient==0.6.13
netaddr==0.8.0
nodeenv==1.7.0
packaging==21.3
paramiko==2.11.0
pathspec==0.9.0
platformdirs==2.5.2
ply==3.11
pre-commit==2.20.0
pycparser==2.21
Pygments==2.12.0
pymdown-extensions==9.5
PyNaCl==1.5.0
pyparsing==3.0.9
pyserial==3.5
python-dateutil==2.8.2
PyYAML==6.0
pyyaml_env_tag==0.1
resolvelib==0.8.1
scp==0.14.4
six==1.16.0
toml==0.10.2
transitions==0.8.11
virtualenv==20.15.1
watchdog==2.1.9
xmltodict==0.13.0
yamllint==1.27.1
yamlordereddictloader==0.4.0
zipp==3.8.1

Collection     Version
-------------- -------
juniper.device 1.0.1

OS / Environment

20.4R2, SRX4600

Summary

According to the ansible docs, _check_type_dict() has been removed.

https://docs.ansible.com/ansible/devel/porting_guides/porting_guide_4.html

There is a call to the removed functions in juniper/device/plugins/module_utils/juniper_junos_common.py - this causes a stack trace when calling some RPC's.

Steps to reproduce

- name: Check configuration for errors
  juniper.device.rpc:
    user: "{{ junos_user }}"
    rpc:
      - "open-configuration"
      - "load-configuration"
      - "validate"
      - "close-configuration"
    attrs:
      - {}
      - action: 'override'
        format: 'text'
      - {}
      - {}
    kwargs: 
      - private: true
      - configuration_text: '{{ cnf }}'
      - {}
      - {}
  register: response
  when: compare or check
  vars:
    cnf: "{{ lookup('ansible.builtin.file', candidate_config ) }}"

Expected results

The task is a work in progress, but it shouldn't fail with a stack trace.

Actual results

FAILED! => "changed": false, "module_stderr": "Traceback (most recent call last):
  File \"/root/.ansible/tmp/ansible-tmp-1657850861.8791683-147-98607207419570/AnsiballZ_rpc.py\", line 107, in <module>
    _ansiballz_main()
  File \"/root/.ansible/tmp/ansible-tmp-1657850861.8791683-147-98607207419570/AnsiballZ_rpc.py\", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File \"/root/.ansible/tmp/ansible-tmp-1657850861.8791683-147-98607207419570/AnsiballZ_rpc.py\", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.juniper.device.plugins.modules.rpc', init_globals=dict(_module_fqn='ansible_collections.juniper.device.plugins.modules.rpc', _modlib_path=modlib_path),
  File \"/usr/local/lib/python3.9/runpy.py\", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/local/lib/python3.9/runpy.py\", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File \"/usr/local/lib/python3.9/runpy.py\", line 87, in _run_code
    exec(code, run_globals)
  File \"/tmp/ansible_juniper.device.rpc_payload_lyq4e7_l/ansible_juniper.device.rpc_payload.zip/ansible_collections/juniper/device/plugins/modules/rpc.py\", line 636, in <module>
  File \"/tmp/ansible_juniper.device.rpc_payload_lyq4e7_l/ansible_juniper.device.rpc_payload.zip/ansible_collections/juniper/device/plugins/modules/rpc.py\", line 459, in main
  File \"/tmp/ansible_juniper.device.rpc_payload_lyq4e7_l/ansible_juniper.device.rpc_payload.zip/ansible_collections/juniper/device/plugins/module_utils/juniper_junos_common.py\", line 916, in parse_arg_to_list_of_dicts
AttributeError: 'JuniperJunosModule' object has no attribute '_check_type_dict'
", "module_stdout": "", "msg": "MODULE FAILURE
See stdout/stderr for the exact error", "rc": 1}

[TheMysteriousX]

There's also an issue in the same code path, where safe_eval is called (I noticed this in rpc.py).

By roundtripping the list of dicts created by the task via string then attempting to convert it back to a list of dicts, safe_eval is used.

safe_eval however, doesn't allow string inputs that contain the word 'import' - so trying to load a configuration via rpc calls that has route table imports fails.

[TheMysteriousX]

There's a further issue, that makes it impossible for some RPC's to be called.
If you take e.g. validate:

<rpc>
  <validate>
    <candidate/>
  </validate>
<rpc>

It is not possible to generate this XML with the current code.

[dineshbaburam91]

@TheMysteriousX PyEZ and ansible support only RPCs which are updated in Junos XML explorer. Junos doesn't support the "validate" tag. Kindly validate and run the RPC which is supported by Junos.

https://apps.juniper.net/xmlapi/

[TheMysteriousX]

Thanks for your reply - the use of the validate RPC is not required to trigger these issues, just using an ansible version >4.

i.e.

- name: Check configuration for errors
  juniper.device.rpc:
    user: "{{ junos_user }}"
    rpc:
      - "open-configuration"
      - "load-configuration"
      - "close-configuration"
    attrs:
      - {}
      - action: 'override'
        format: 'text'
      - {}
    kwargs: 
      - private: true
      - configuration_text: '{{ cnf }}'
      - {}
  register: response
  when: compare or check
  vars:
    cnf: "{{ lookup('ansible.builtin.file', candidate_config ) }}"

This task still throws the same stack trace.

Validate is documented here - I'm not sure why it's missing from the xml explorer:
https://www.juniper.net/documentation/us/en/software/junos/netconf/topics/task/netconf-configuration-verifying.html

[chidanandpujar]

Hi ,

Looks to be working fine ,Could you please verify once with latest Ansible collections.

- name: Test juniper.device.rpc module
  hosts: all
  connection: local
  gather_facts: no
  collections:
    - juniper.device

  tasks:
#################
    - name: "Check configuration for errors"
      rpc:
        rpcs:
          - "open-configuration"
          - "load-configuration"
          - "close-configuration"
        attrs:
          - {}
          - action: 'set'
            format: 'text'
          - {}
        kwargs:
          - private: true
          - configuration_set: 'set system syslog file test1 any any'
          - {}
      register: test9
      tags: [ test9 ]

    - name: Check TEST 9
      debug:
        var: test9
      tags: [ test9 ]

ansible-playbook -i inventory pb.juniper_junos_issue_589.yml 

PLAY [Test juniper.device.rpc module] **************************************************************************************************************************************************************

TASK [Check configuration for errors] **************************************************************************************************************************************************************
ok: [test]

TASK [Check TEST 9] ********************************************************************************************************************************************************************************
ok: [test] => {
    "test9": {
        "changed": false,
        "failed": false,
        "results": [
            {
                "attrs": {},
                "changed": false,
                "failed": false,
                "format": "xml",
                "kwargs": {
                    "private": true
                },
                "msg": "The RPC executed successfully.",
                "rpc": "open-configuration",
                "stdout": "",
                "stdout_lines": []
            },
            {
                "attrs": {
                    "action": "set",
                    "format": "text"
                },
                "changed": false,
                "failed": false,
                "format": "xml",
                "kwargs": {
                    "configuration_set": "set system syslog file test1 any any"
                },
                "msg": "The RPC executed successfully.",
                "parsed_output": {
                    "load-configuration-results": {
                        "ok": ""
                    }
                },
                "rpc": "load-configuration",
                "stdout": "<load-configuration-results>\n  <ok/>\n</load-configuration-results>\n",
                "stdout_lines": [
                    "<load-configuration-results>",
                    "  <ok/>",
                    "</load-configuration-results>"
                ]
            },
            {
                "attrs": {},
                "changed": false,
                "failed": false,
                "format": "xml",
                "kwargs": {},
                "msg": "The RPC executed successfully.",
                "rpc": "close-configuration",
                "stdout": "",
                "stdout_lines": []
            }
        ]
    }
}

PLAY RECAP *****************************************************************************************************************************************************************************************
test                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Thanks