ansible.netcommon.httpapi Connection send doesn't consider keyword arguments for open_url

[ArunSaravananBalachandran]

SUMMARY

In dellemc.enterprise_sonic collection for sending requests using 'httpapi' connection, send method of 'Connection' class in ansible.netcommon.httpapi is used.
While sending requests, 'path' and 'data' are passed as positional arguments along with 'headers' and 'method' as keyword arguments to the 'send' method.

• In ansible.netcommon v4.1.0, all the keyword arguments passed to send were in turn passed to open_url for forming the requests to the device.
• But in v5.0.0, the keyword arguments passed to send are not passed to open_url and the requests to the device are missing those required details resulting in failure.

The earlier behavior has been modified by PR #494

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible.netcommon.httpapi

ANSIBLE VERSION

ansible [core 2.13.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/abalac/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/abalac/ansible-env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/abalac/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/abalac/ansible-env/bin/ansible
  python version = 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/abalac/.ansible/collections/ansible_collections
Collection               Version
------------------------ -------
ansible.netcommon        5.0.0
ansible.utils            2.9.0
dellemc.enterprise_sonic 2.0.0

CONFIGURATION

DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 60
HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible.cfg) = 500
PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible.cfg) = 500

OS / ENVIRONMENT

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal

$ python --version
Python 3.8.10

Target Device Firmware: Enterprise SONiC Distribution by Dell Technologies

STEPS TO REPRODUCE

With no VLAN configuration in the device, use the below playbook to configure a VLAN

---                                                                                                                                                  
- name: Ansible vlan resource module config example
  hosts: datacenter
  connection: httpapi
  collections:
    - dellemc.enterprise_sonic
  tasks:
    - name: Add Vlans
      dellemc.enterprise_sonic.sonic_vlans:
        config:
          - vlan_id: 10
        state: merged     

EXPECTED RESULTS

The playbook to configure the specified VLAN in the device completes successfully.

Arguments to open_url from Connection.send for the above playbook with ansible.netcommon v4.1.0 (obtained using persistent_log_messages) - headers and method are populated

2023-03-03 20:31:42,378 p=448696 u=abalac n=ansible | send url 'https://a.b.c.d:443/restconf/data/openconfig-interfaces:interfaces' with data 'None' and kwargs '{'timeout': 500, 'validate_certs': False, 'use_proxy': True, 'headers': {'Content-Type': 'application/yang-data+json', 'Accept': 'application/yang-data+json'}, 'method': 'get', 'force_basic_auth': True, 'url_username': 'admin', 'url_password': 'admin'}'
2023-03-03 20:31:42,379 p=448696 u=abalac n=ansible | send url 'https://a.b.c.d:443/restconf/data/openconfig-interfaces:interfaces' with data '{"openconfig-interfaces:interfaces": {"interface": [{"name": "Vlan10", "config": {"name": "Vlan10"}}]}}' and kwargs '{'timeout': 500, 'validate_certs': False, 'use_proxy': True, 'headers': {'Content-Type': 'application/yang-data+json', 'Accept': 'application/yang-data+json'}, 'method': 'PATCH', 'force_basic_auth': True, 'url_username': 'admin', 'url_password': 'admin'}'
2023-03-03 20:31:42,379 p=448696 u=abalac n=ansible | send url 'https://a.b.c.d:443/restconf/data/openconfig-interfaces:interfaces' with data 'None' and kwargs '{'timeout': 500, 'validate_certs': False, 'use_proxy': True, 'headers': {'Content-Type': 'application/yang-data+json', 'Accept': 'application/yang-data+json'}, 'method': 'get', 'force_basic_auth': True, 'url_username': 'admin', 'url_password': 'admin'}'

ACTUAL RESULTS

Request sent to the device for VLAN configuration is not proper and the execution fails.

ansible-playbook -i hosts sonic_vlan_config.yml  -vvv
ansible-playbook [core 2.13.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/abalac/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/abalac/ansible-env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/abalac/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/abalac/ansible-env/bin/ansible-playbook
  python version = 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /home/abalac/ansible/acl/l3_acls/hosts as it did not pass its verify_file() method
script declined parsing /home/abalac/ansible/acl/l3_acls/hosts as it did not pass its verify_file() method
auto declined parsing /home/abalac/ansible/acl/l3_acls/hosts as it did not pass its verify_file() method
Parsed /home/abalac/ansible/acl/l3_acls/hosts inventory source with ini plugin
redirecting (type: action) dellemc.enterprise_sonic.sonic_vlans to dellemc.enterprise_sonic.sonic
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: sonic_vlan_config.yml *********************************************************************************************************************
1 plays in sonic_vlan_config.yml

PLAY [Ansible vlan resource module config example] **************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************
task path: /home/abalac/ansible/acl/l3_acls/sonic_vlan_config.yml:2
redirecting (type: connection) ansible.builtin.httpapi to ansible.netcommon.httpapi
<a.b.c.d> ESTABLISH LOCAL CONNECTION FOR USER: abalac
<a.b.c.d> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7 `"&& mkdir "` echo /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/ansible-tmp-1677855364.431776-444987-158222360505697 `" && echo ansible-tmp-1677855364.431776-444987-158222360505697="` echo /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/ansible-tmp-1677855364.431776-444987-158222360505697 `" ) && sleep 0'
Using module file /home/abalac/ansible-env/lib/python3.8/site-packages/ansible/modules/setup.py
<a.b.c.d> PUT /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/tmpvanqqsxp TO /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/ansible-tmp-1677855364.431776-444987-158222360505697/AnsiballZ_setup.py
<a.b.c.d> EXEC /bin/sh -c 'chmod u+x /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/ansible-tmp-1677855364.431776-444987-158222360505697/ /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/ansible-tmp-1677855364.431776-444987-158222360505697/AnsiballZ_setup.py && sleep 0'
<a.b.c.d> EXEC /bin/sh -c '/home/abalac/ansible-env/bin/python3 /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/ansible-tmp-1677855364.431776-444987-158222360505697/AnsiballZ_setup.py && sleep 0'
<a.b.c.d> EXEC /bin/sh -c 'rm -f -r /home/abalac/.ansible/tmp/ansible-local-444971aemsgwi7/ansible-tmp-1677855364.431776-444987-158222360505697/ > /dev/null 2>&1 && sleep 0'
ok: [sonic2]
META: ran handlers
redirecting (type: action) dellemc.enterprise_sonic.sonic_vlans to dellemc.enterprise_sonic.sonic

TASK [Add Vlans] ************************************************************************************************************************************
task path: /home/abalac/ansible/acl/l3_acls/sonic_vlan_config.yml:8
redirecting (type: connection) ansible.builtin.httpapi to ansible.netcommon.httpapi
redirecting (type: action) dellemc.enterprise_sonic.sonic_vlans to dellemc.enterprise_sonic.sonic
redirecting (type: action) dellemc.enterprise_sonic.sonic_vlans to dellemc.enterprise_sonic.sonic
The full traceback is:
  File "/home/abalac/.ansible/collections/ansible_collections/dellemc/enterprise_sonic/plugins/module_utils/network/sonic/config/vlans/vlans.py", line 100, in execute_module
    edit_config(self._module, to_request(self._module, requests))
  File "/home/abalac/.ansible/collections/ansible_collections/dellemc/enterprise_sonic/plugins/module_utils/network/sonic/sonic.py", line 140, in edit_config
    return connection.edit_config(commands)
  File "/home/abalac/ansible-env/lib/python3.8/site-packages/ansible/module_utils/connection.py", line 200, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [sonic2]: FAILED! => {
    "changed": false,
    "code": -32603,
    "invocation": {
        "module_args": {
            "config": [
                {
                    "description": null,
                    "vlan_id": 10
                }
            ],
            "state": "merged"
        }
    },
    "msg": "{'ietf-restconf:errors': {'error': [{'error-type': 'protocol', 'error-tag': 'invalid-value', 'error-message': 'Unsupported content-type'}]}, 'code': 415, 'request_data': {'path': 'data/openconfig-interfaces:interfaces', 'method': 'PATCH'}}"
}

PLAY RECAP ******************************************************************************************************************************************
sonic2                     : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Arguments to open_url from Connection.send for the above playbook with ansible.netcommon v5.0.0 (obtained using persistent_log_messages) - headers and method are not populated

2023-03-03 20:30:43,077 p=447421 u=abalac n=ansible | send url 'https://a.b.c.d:443/restconf/data/openconfig-interfaces:interfaces' with data 'None' and kwargs '{'timeout': 500, 'validate_certs': False, 'use_proxy': True, 'headers': {}, 'force_basic_auth': True, 'url_username': 'admin', 'url_password': 'admin'}'
2023-03-03 20:30:43,077 p=447421 u=abalac n=ansible | send url 'https://a.b.c.d:443/restconf/data/openconfig-interfaces:interfaces' with data '{"openconfig-interfaces:interfaces": {"interface": [{"name": "Vlan10", "config": {"name": "Vlan10"}}]}}' and kwargs '{'timeout': 500, 'validate_certs': False, 'use_proxy': True, 'headers': {}, 'force_basic_auth': True, 'url_username': 'admin', 'url_password': 'admin'}'