Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ios_config fails when creating macro "timeout trying to send command:" #38420

Closed
yaplej opened this issue Apr 6, 2018 · 15 comments · Fixed by #44001
Closed

ios_config fails when creating macro "timeout trying to send command:" #38420

yaplej opened this issue Apr 6, 2018 · 15 comments · Fixed by #44001
Assignees
Labels
affects_2.4 This issue/PR affects Ansible v2.4 bug This issue/PR relates to a bug. cisco Cisco technologies feature This issue/PR relates to a feature request. ios Cisco IOS community module This issue/PR relates to a module. networking Network category support:network This issue/PR relates to code supported by the Ansible Network Team.
Milestone

Comments

@yaplej
Copy link
Contributor

yaplej commented Apr 6, 2018

ISSUE TYPE
  • Bug Report
COMPONENT NAME

ios_config

ANSIBLE VERSION
ansible 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]

CONFIGURATION

HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False
PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible.cfg) = 30

OS / ENVIRONMENT

CentOS Linux release 7.4.1708 (Core)
Cisco IOS XE Software, Version 16.06.03

SUMMARY

ios_config timeout when sending "macro name MYMACRO" command

STEPS TO REPRODUCE

Rerun playbook

---

- hosts: access_switches

  tasks:

  - name: MACRO PHONE
    ios_config:
      lines:
      - 'switchport voice vlan 60'
      - 'switchport port-security maximum 2'
      - 'spanning-tree portfast'
      - 'no cdp enable'
      - 'no macro desc'
      - '#macro keywords $DESCRIPTION'
      - '@'
      parents: macro name PHONE
      before: no macro name PHONE
      after: exit
      match: line
      replace: block
EXPECTED RESULTS

Creates the macro.

ACTUAL RESULTS
ansible-playbook 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible-playbook
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /etc/ansible/production inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: MACRO_PHONE.yml ********************************************************************************************************
1 plays in ./roles/network/switch/access/MACRO_PHONE.yml

PLAY [access_switches] ***********************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************
Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<10.X.X.X> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.X.X.X> EXEC /bin/sh -c 'echo ~ && sleep 0'
<10.X.X.X> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1523044289.76-194673473453695 `" && echo ansible-tmp-1523044289.76-194673473453695="` echo /root/.ansible/tmp/ansible-tmp-1523044289.76-194673473453695 `" ) && sleep 0'
<10.X.X.X> PUT /tmp/tmpZi78WP TO /root/.ansible/tmp/ansible-tmp-1523044289.76-194673473453695/setup.py
<10.X.X.X> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1523044289.76-194673473453695/ /root/.ansible/tmp/ansible-tmp-1523044289.76-194673473453695/setup.py && sleep 0'
<10.X.X.X> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1523044289.76-194673473453695/setup.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1523044289.76-194673473453695/" > /dev/null 2>&1 && sleep 0'
ok: [10.X.X.X]
META: ran handlers

TASK [MACRO PHONE] ***************************************************************************************************************
task path: /etc/ansible/roles/network/switch/access/MACRO_PHONE.yml:7
<10.X.X.X> using connection plugin network_cli
<10.X.X.X> socket_path: /root/.ansible/pc/70f3934ea5
Using module file /usr/lib/python2.7/site-packages/ansible/modules/network/ios/ios_config.py
<10.X.X.X> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.X.X.X> EXEC /bin/sh -c 'echo ~ && sleep 0'
<10.X.X.X> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1523044295.51-69820156660034 `" && echo ansible-tmp-1523044295.51-69820156660034="` echo /root/.ansible/tmp/ansible-tmp-1523044295.51-69820156660034 `" ) && sleep 0'
<10.X.X.X> PUT /tmp/tmpVVnmKX TO /root/.ansible/tmp/ansible-tmp-1523044295.51-69820156660034/ios_config.py
<10.X.X.X> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1523044295.51-69820156660034/ /root/.ansible/tmp/ansible-tmp-1523044295.51-69820156660034/ios_config.py && sleep 0'
<10.X.X.X> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1523044295.51-69820156660034/ios_config.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1523044295.51-69820156660034/" > /dev/null 2>&1 && sleep 0'
fatal: [10.X.X.X]: FAILED! => {
    "changed": false,
    "command": "macro name PHONE",
    "invocation": {
        "module_args": {
            "after": [
                "exit"
            ],
            "auth_pass": null,
            "authorize": null,
            "backup": false,
            "before": [
                "no macro name PHONE"
            ],
            "defaults": false,
            "diff_against": null,
            "diff_ignore_lines": null,
            "force": false,
            "host": null,
            "intended_config": null,
            "lines": [
                "switchport voice vlan 60",
                "switchport port-security maximum 5 ! was 2",
                "spanning-tree portfast",
                "no cdp enable",
                "no macro desc",
                "#macro keywords $DESCRIPTION",
                "@"
            ],
            "match": "line",
            "multiline_delimiter": "@",
            "parents": [
                "macro name PHONE"
            ],
            "password": null,
            "port": null,
            "provider": null,
            "replace": "block",
            "running_config": null,
            "save": false,
            "save_when": "never",
            "src": null,
            "ssh_keyfile": null,
            "timeout": null,
            "username": null
        }
    },
    "msg": "timeout trying to send command: macro name PHONE",
    "rc": 1
}
        to retry, use: --limit @/etc/ansible/roles/network/switch/access/MACRO_PHONE.retry

PLAY RECAP ***********************************************************************************************************************
10.X.X.X               : ok=1    changed=0    unreachable=0    failed=1
@ansibot
Copy link
Contributor

ansibot commented Apr 6, 2018

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

@ansibot
Copy link
Contributor

ansibot commented Apr 6, 2018

@ansibot ansibot added affects_2.4 This issue/PR affects Ansible v2.4 bug This issue/PR relates to a bug. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. networking Network category support:network This issue/PR relates to code supported by the Ansible Network Team. labels Apr 6, 2018
@yaplej
Copy link
Contributor Author

yaplej commented Apr 6, 2018

Just a guess but a macro is a multi-line command but does not allow you to choose the delimiter like "banner". So perhaps ansible its assuming the command should return a prompt or something instead of treating it as a multi-line command?

The output of "macro name MYMACRO" is:

Enter macro commands one per line. End with the character '@'.

Config on switch is like:

macro name PHONE
  switchport voice vlan 60
  switchport port-security maximum 2
  spanning-tree portfast
  no cdp enable
  no macro desc
#macro keywords $DESCRIPTION
@

@s-hertel s-hertel removed the needs_triage Needs a first human triage before being processed. label Apr 6, 2018
@calfonso calfonso added the feature This issue/PR relates to a feature request. label Apr 11, 2018
@calfonso
Copy link
Contributor

Adding this to our feature idea list. There is a workaround by using the ios_config module and enter the command as one long line.

@gundalow
Copy link
Contributor

I've raised #38591 to track giving an error message

@yaplej
Copy link
Contributor Author

yaplej commented Apr 11, 2018

@calfonso

Perhaps I misunderstand is there an example of this somewhere? I have tried some variations to enter the whole thing as a single command but nothing has worked so far.

---

- hosts: access_switches

  tasks:

  - name: MACRO PHONE
    ios_config:
      lines: |
       - 'macro name PHONE'
       - 'switchport voice vlan 60'
       - 'switchport port-security maximum 5 ! was 2'
       - 'spanning-tree portfast'
       - 'no cdp enable'
       - 'no macro desc'
       - '#macro keywords $DESCRIPTION'
       - '@'
      before: no macro name PHONE
      after: exit
      match: line
      replace: block
      timeout: 60

@bdowling
Copy link
Contributor

Had not known about this command, but this seems like another example of multi-line inputs such as with the banner commands. Instead of having another specially focused feature or module on macros, I'd propose to just have a detection for ^Enter.*End with the character '\(.\)' that puts the rest of the task into a sendonly mode for the duration of the commands in that task?

I did a quick test of this and was able to get this play to work (after fixing the structure issues in above),

Willing to create the PR if this sounds like an acceptable approach.

e.g.

  - name: MACRO PHONE
    ios_config:
      lines:
       - macro name PHONE
       - switchport voice vlan 60
       - switchport port-security maximum 5 ! was 2
       - spanning-tree portfast
       - no cdp enable
       - no macro desc
       - #macro keywords $DESCRIPTION
       - "@"
      before: no macro name PHONE
      match: line
      replace: block
      timeout: 60

@yaplej
Copy link
Contributor Author

yaplej commented Apr 13, 2018

@bdowling

I think a general method to handle multi-line inputs instead of only handling the "banner" command would be great.

Does your solution work if your using "parents:macro name PHONE"? If you end up making a PR I would certainly be willing to help test it.

@bdowling
Copy link
Contributor

@yaplej fwiw, on my 9500 it doesn't appear that a macro is indented, so parents would not apply.

Which, IMHO is really broken on Cisco's part not to indent that. It doesn't for banner either. But that's the way it is.

@calfonso calfonso added this to the 2.7.0 milestone Apr 16, 2018
@yaplej
Copy link
Contributor Author

yaplej commented Jun 17, 2018

@bdowling @calfonso
I never found a functional workaround for this. Did you ever create a PR?

Thanks.

@yaplej
Copy link
Contributor Author

yaplej commented Jul 26, 2018

@bdowling

I tried the structural changes you mentioned but I was unable to get it to work. Still getting "timeout trying to send command: macro name PHONE"

  - name: MACRO PHONE
    ios_config:
      lines:
      - macro name PHONE
      - switchport voice vlan 60
      - switchport port-security maximum 5
      - spanning-tree portfast
      - no cdp enable
      - no macro desc
      - #macro keywords $DESCRIPTION
      - "@"
      before: no macro name PHONE
      match: line
      replace: block
      timeout: 60

I also updated to ansible 2.6.1 but no luck.

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_hc4T4F/ansible_module_ios_config.py", line 561, in <module>
    main()
  File "/tmp/ansible_hc4T4F/ansible_module_ios_config.py", line 490, in main
    load_config(module, commands)
  File "/tmp/ansible_hc4T4F/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py", line 168, in load_config
  File "/tmp/ansible_hc4T4F/ansible_modlib.zip/ansible/module_utils/connection.py", line 149, in __rpc__
ansible.module_utils.connection.ConnectionError: timeout trying to send command: macro name PHONE

fatal: [10.1.100.11]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_hc4T4F/ansible_module_ios_config.py\", line 561, in <module>\n    main()\n  File \"/tmp/ansible_hc4T4F/ansible_module_ios_config.py\", line 490, in main\n    load_config(module, commands)\n  File \"/tmp/ansible_hc4T4F/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py\", line 168, in load_config\n  File \"/tmp/ansible_hc4T4F/ansible_modlib.zip/ansible/module_utils/connection.py\", line 149, in __rpc__\nansible.module_utils.connection.ConnectionError: timeout trying to send command: macro name PHONE\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}

@yaplej
Copy link
Contributor Author

yaplej commented Aug 6, 2018

I'm still trying to get it to work in 2.6.2. Its a total hack job but it seems like its close to running. There is probably a much better solution like detecting the line "Enter macro commands one per line. End with the character '@'." and going into sendonly mode as previously mentioned but I was not sure where that could be done.

In /modules/network/ios/ios_config.py created a modified function based on load_banners().

def load_config2(module, commands):
    delimiter = module.params['multiline_delimiter']
    commands.insert(0,'config terminal')
    commands.append(delimiter)
    commands.append('end')
    thecommands = []
    for line in thecommands:
        thecommands.appent(line + '\n')
    for cmd in [thecommands]:
        obj = {'command': cmd, 'sendonly': True}
        run_commands(module, [cmd])
    time.sleep(0.1)
    run_commands(module, ['\n'])

Temporarily send all commands through load_config2() function. Because I don't know what I am doing.

            if not module.check_mode:
                if commands:
                    load_config2(module, commands)
                if banners:
                    load_banners(module, banners)

Using a very simple playbook. I had to take the '@' out of the play and add it as the delimiter in load_config2() because it kept getting stripped out of the updates.

---

- hosts: access_switches

  tasks:

  - name: MACRO PHONE
    ios_config:
      provider:
        timeout: 60
      lines:
      - macro name PHONE
      - switchport voice vlan 60
      - switchport port-security maximum 5
      - spanning-tree portfast
      - no cdp enable
      - no macro desc
      - "#macro keywords $DESCRIPTION"
      before: no macro name PHONE
      match: line
      replace: block

It appears to run now but does not actually put the config on the device for some reason.

changed: [1.1.1.1] => {
    "banners": {},
    "changed": true,
    "commands": [
        "config terminal",
        "macro name PHONE",
        "switchport voice vlan 60",
        "switchport port-security maximum 5",
        "spanning-tree portfast",
        "no cdp enable",
        "no macro desc",
        "#macro keywords $DESCRIPTION",
        "@",
        "end"
    ],
    "invocation": {
        "module_args": {
            "after": null,
            "auth_pass": null,
            "authorize": null,
            "backup": false,
            "before": null,
            "defaults": false,
            "diff_against": null,
            "diff_ignore_lines": null,
            "force": false,
            "host": null,
            "intended_config": null,
            "lines": [
                "macro name PHONE",
                "switchport voice vlan 60",
                "switchport port-security maximum 5",
                "spanning-tree portfast",
                "no cdp enable",
                "no macro desc",
                "#macro keywords $DESCRIPTION"
            ],
            "match": "line",
            "multiline_delimiter": "@",
            "parents": null,
            "password": null,
            "port": null,
            "provider": {
                "auth_pass": null,
                "authorize": null,
                "host": null,
                "password": null,
                "port": null,
                "ssh_keyfile": null,
                "timeout": 60,
                "username": null
            },
            "replace": "block",
            "running_config": null,
            "save": false,
            "save_when": "never",
            "src": null,
            "ssh_keyfile": null,
            "timeout": null,
            "username": null
        }
    },
    "updates": [
        "config terminal",
        "macro name PHONE",
        "switchport voice vlan 60",
        "switchport port-security maximum 5",
        "spanning-tree portfast",
        "no cdp enable",
        "no macro desc",
        "#macro keywords $DESCRIPTION",
        "@",
        "end"
    ]
}

I am stumped why the updates are not actually getting executed on the device. I enable command logging on the device and can see nothing is getting executed.

Another thing I noticed with macro's in general is that they cannot be modified without first removing them. So the "before" statement of 'no macro name WORD' must be executed before the macro is re-created. If the macro partially exists or is modified the "updates" section is missing the commands that were formerly on the device.

@yaplej
Copy link
Contributor Author

yaplej commented Aug 6, 2018

@bdowling Your correct that the IOS does not indent anything under the macro command itself but you can manually indent them by prefixing the lines with a space or two when you enter them.

macro name PHONE
 switchport voice vlan 60
 switchport port-security maximum 5
 spanning-tree portfast
 no cdp enable
 no macro desc
 #macro keywords $DESCRIPTION
@

If you do that your able to treat each macro as a section in the config. With the exception of the terminating '@'.

show run | sec macro name PHONE
macro name PHONE
 switchport voice vlan 60
 switchport port-security maximum 5
 spanning-tree portfast
 no cdp enable
 no macro desc
 #macro keywords $DESCRIPTION

@yaplej
Copy link
Contributor Author

yaplej commented Aug 7, 2018

I actually got it to work. No its not pretty.

Created a new function to separate normal commands from macros.

def load_config_or_macro(module, commands):
    if "macro name" in commands[0]:
        load_macro(module, commands)
    else:
        load_config(module, commands)

Edit: Should be looking for 'macro name' not just 'macro'.

Send all commands through load_config_or_macro() to filter out macros to load_macro().

            if not module.check_mode:
                if commands:
                    load_config_or_macro(module, commands)
                if banners:
                    load_banners(module, banners)

Took all the lines from the macro and converted it into one big blob of commands similar to how the banner command is handled.

def load_macro(module, commands):
    thecommands = ''
    thecommands += 'config terminal\n'
    if "#macro keywords" in commands[2]:
        return
    for line in commands:
        if line != 'None':
             thecommands += (' ' + line + '\n')
    thecommands += 'end\n'
    for cmd in [thecommands]:
        obj = {'command': cmd, 'sendonly': True}
        run_commands(module, [cmd])
    time.sleep(0.1)
    run_commands(module, ['\n'])

Edit: Adding the delimiter is not needed after fixing the replace block functionality by adding the 'parent:' to the play.

The macro play had to get two lines quoted or else they were handled as 'None' or blank lines.

---

- hosts: access_switches

  tasks:

  - name: MACRO PHONE
    ios_config:
      provider:
        timeout: 60
      lines:
      - switchport voice vlan 60
      - switchport port-security maximum 5
      - spanning-tree portfast
      - no cdp enable
      - no macro desc
      - "#macro keywords $DESCRIPTION"
      - "@"
      before: 'no macro name PHONE'
      parents: macro name PHONE
      match: line
      replace: block

Edit: Replace block only works when used with 'parents:'. #29716

Edit: Tried using 'diff_ignore_lines:' for '@' and '#macro.*' does this work at all? #30562

Last thing is it needs improvement to be idempotent. I really do not want it to be running every time the play is executed but right now it seems to detect that the lines '#macro keyword.*' and '@' are different before each run. So I think the diff detection needs improved to handle macros better too.

@yaplej
Copy link
Contributor Author

yaplej commented Aug 8, 2018

I am not sure how to make it idempotent.

The difference is generated by candidate.difference() from the NetworkConfig class. In that it has DEFAULT_COMMENT_TOKENS and includes '#'. That is why the line '#macro keywords $DESCRIPTION' line is always detected as a difference/change.

I was able to resolve the issue with '@' being detected as a change by moving that to 'after:' in the play. Putting each line in single quotes keeps '#macro keywords' line from being treated as 'None' so I put them all in quotes for consistency.

---

- hosts: access_switches

  tasks:

  - name: MACRO PHONE
    ios_config:
      provider:
        timeout: 60
      lines:
      - 'switchport voice vlan 60'
      - 'switchport port-security maximum 5'
      - 'spanning-tree portfast'
      - 'no cdp enable'
      - 'no macro desc'
      - '#macro keywords $DESCRIPTION'
      before: no macro name PHONE
      after: '@'
      parents: macro name PHONE
      match: line
      replace: block

Edit: I confirmed by removing '#' from 'DEFAULT_COMMENT_TOKENS' in '/module_utils/network/common/config.py' and running my playbook again. This seems like a separate issue so I opened #43861.

ok: [1.1.1.1] => {
    "changed": false,
    "invocation": {
        "module_args": {
            "after": [
                "@"
            ],
            "auth_pass": null,
            "authorize": null,
            "backup": false,
            "before": [
                "no macro name PHONE"
            ],
            "defaults": false,
            "diff_against": null,
            "diff_ignore_lines": null,
            "force": false,
            "host": null,
            "intended_config": null,
            "lines": [
                "switchport voice vlan 60",
                "switchport port-security maximum 5",
                "spanning-tree portfast",
                "no cdp enable",
                "no macro desc",
                "#macro keywords $DESCRIPTION"
            ],
            "match": "line",
            "multiline_delimiter": "@",
            "parents": [
                "macro name PHONE"
            ],
            "password": null,
            "port": null,
            "provider": {
                "auth_pass": null,
                "authorize": null,
                "host": null,
                "password": null,
                "port": null,
                "ssh_keyfile": null,
                "timeout": 60,
                "username": null
            },
            "replace": "block",
            "running_config": null,
            "save": false,
            "save_when": "never",
            "src": null,
            "ssh_keyfile": null,
            "timeout": null,
            "username": null
        }
    }
}

@ganeshrn ganeshrn self-assigned this Aug 21, 2018
@dagwieers dagwieers added ios Cisco IOS community cisco Cisco technologies labels Feb 23, 2019
@ansible ansible locked and limited conversation to collaborators Jul 22, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.4 This issue/PR affects Ansible v2.4 bug This issue/PR relates to a bug. cisco Cisco technologies feature This issue/PR relates to a feature request. ios Cisco IOS community module This issue/PR relates to a module. networking Network category support:network This issue/PR relates to code supported by the Ansible Network Team.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants