Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

network.traceroute intermittently stacktraces #21927

Closed
jfindlay opened this issue Mar 24, 2015 · 6 comments

Comments

@jfindlay
Copy link
Contributor

commented Mar 24, 2015

versions

LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.0.1406 (Core)
Release:        7.0.1406
Codename:       Core
           Salt: 2014.7.2-450-gd3e14e5
         Python: 2.7.5 (default, Jun 17 2014, 18:11:42)
         Jinja2: 2.7.3
       M2Crypto: 0.21.1
 msgpack-python: 0.4.4
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 3.2.5
           Mako: Not Installed

execution

centos-7-main salt 2014.7 # salt centos-7-main network.traceroute gentoo.org
centos-7-main:
    The minion function caused an exception: Traceback (most recent call last):
      File "/root/salt/salt/minion.py", line 1019, in _thread_return
        return_data = func(*args, **kwargs)
      File "/root/salt/salt/modules/network.py", line 373, in traceroute
        'ms2': float(comps[3].split()[0]),
    IndexError: list index out of range

minion log

You can see the traceroute printout below that caused the error.

[INFO    ] Executing command 'traceroute gentoo.org' in directory '/root'
[INFO    ] User root Executing command saltutil.find_job with jid 20150323214403219816
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150323214403219816', 'tgt': 'centos-7-main', 'ret': '', 'user': 'root', 'arg': ['20150323214358198988'], 'fun': 'saltutil.find_job'}
[DEBUG   ] output: traceroute to gentoo.org (89.16.167.134), 30 hops max, 60 byte packets
 1  192.168.122.1 (192.168.122.1)  0.126 ms  0.092 ms  0.082 ms
 2  192.168.1.1 (192.168.1.1)  5.248 ms  5.187 ms  5.113 ms
 3  166-70-0-200.ip.xmission.com (166.70.0.200)  45.696 ms * *
 4  403.gi-3-13.dcr2.slc.xmission.net (166.70.5.82)  48.204 ms  48.524 ms  48.757 ms
 5  br2.core.xmission.net (166.70.1.3)  51.223 ms  52.776 ms  53.050 ms
 6  br1.core.xmission.net (166.70.1.2)  55.100 ms  56.522 ms  56.589 ms
 7  ip65-46-51-49.z51-46-65.customer.algx.net (65.46.51.49)  59.298 ms  58.270 ms  58.436 ms
 8  216.156.1.129.ptr.us.xo.net (216.156.1.129)  73.811 ms  62.515 ms  60.573 ms
 9  216.156.16.2.ptr.us.xo.net (216.156.16.2)  62.292 ms  64.693 ms  63.371 ms
10  ae0d0.cir1.denver2-co.us.xo.net (207.88.13.181)  58.043 ms  58.068 ms  59.538 ms
11  206.111.8.66.ptr.us.xo.net (206.111.8.66)  58.732 ms  58.602 ms  56.305 ms
12  ae-4-4.car1.Manchesteruk1.Level3.net (4.69.133.101)  317.893 ms  317.626 ms  302.095 ms
13  ae-4-4.car1.Manchesteruk1.Level3.net (4.69.133.101)  272.920 ms  272.596 ms  270.520 ms
14  te3-5.cr2.man.bytemark.co.uk (195.50.119.78)  269.640 ms  269.938 ms  267.758 ms
15  po12.cr1.man.bytemark.co.uk (91.223.58.132)  266.569 ms  266.317 ms  265.124 ms
16  te0-1.cr1.lds.bytemark.co.uk (91.223.58.69)  265.034 ms  264.846 ms  204.443 ms
17  te1-6.cr1.yrk.bytemark.co.uk (91.223.58.76)  204.474 ms  204.281 ms  204.111 ms
18  po1.ar1.dc1.yo26.yrk.bytemark.co.uk (91.223.58.29)  203.879 ms  203.704 ms  203.541 ms
19  www.gentoo.org (89.16.167.134)  203.359 ms  203.196 ms  203.039 ms
[INFO    ] Executing command 'traceroute --version' in directory '/root'
[INFO    ] Starting a new job with PID 13623
[DEBUG   ] output: Modern traceroute for Linux, version 2.0.19, Jun 10 2014
Copyright (c) 2008  Dmitry Butskoy,   License: GPL v2 or any later
[DEBUG   ] traceroute_version_raw: ('2', '0', '19')
[DEBUG   ] traceroute_version: [2, 0, 19]

This output below is the result of a print statement I put here, (from pprint import pprint ; pprint(comps)),

[' 1', '192.168.122.1 (192.168.122.1)', '0.126 ms', '0.092 ms', '0.082 ms']
[' 2', '192.168.1.1 (192.168.1.1)', '5.248 ms', '5.187 ms', '5.113 ms']
[' 3', '166-70-0-200.ip.xmission.com (166.70.0.200)', '45.696 ms * *']

The minion log continues,

[WARNING ] The minion function caused an exception
Traceback (most recent call last):
  File "/root/salt/salt/minion.py", line 1019, in _thread_return
    return_data = func(*args, **kwargs)
  File "/root/salt/salt/modules/network.py", line 373, in traceroute
    'ms2': float(comps[3].split()[0]),
IndexError: list index out of range

I would fix this myself, but I've fixed enough bugs today, and this function is somewhat involved, so I gave up.

@jfindlay jfindlay added this to the Approved milestone Mar 24, 2015
@jfindlay

This comment has been minimized.

Copy link
Contributor Author

commented Mar 25, 2015

Another example.

command

salt ubuntu-14 network.traceroute gentoo.org

minion log

[INFO    ] User root Executing command network.traceroute with jid 20150325122712207206
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150325122712207206', 'tgt': '*', 'ret': '', 'user': 'root', 'arg': ['gentoo.org'], 'fun': 'network.traceroute'}
[INFO    ] Starting a new job with PID 4251
[INFO    ] Executing command 'traceroute gentoo.org' in directory '/root'
[INFO    ] User root Executing command saltutil.find_job with jid 20150325122717220593
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150325122717220593', 'tgt': '*', 'ret': '', 'user': 'root', 'arg': ['20150325122712207206'], 'fun': 'saltutil.find_job'}
[INFO    ] Starting a new job with PID 4255
[INFO    ] Returning information for job: 20150325122717220593
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] output: traceroute to gentoo.org (89.16.167.134), 30 hops max, 60 byte packets
 1  192.168.122.1 (192.168.122.1)  0.144 ms  0.151 ms  0.158 ms
 2  192.168.1.1 (192.168.1.1)  3.162 ms  5.455 ms  11.998 ms
 3  166-70-0-200.ip.xmission.com (166.70.0.200)  44.782 ms  47.086 ms  48.541 ms
 4  403.gi-3-13.dcr2.slc.xmission.net (166.70.5.82)  48.069 ms  49.265 ms  50.781 ms
 5  br2.core.xmission.net (166.70.1.3)  52.832 ms  53.785 ms  55.260 ms
 6  br1.core.xmission.net (166.70.1.2)  57.250 ms  38.165 ms  39.619 ms
 7  ip65-46-51-49.z51-46-65.customer.algx.net (65.46.51.49)  41.889 ms  38.878 ms  39.941 ms
 8  216.156.1.129.ptr.us.xo.net (216.156.1.129)  55.978 ms  57.380 ms  62.529 ms
 9  216.156.16.2.ptr.us.xo.net (216.156.16.2)  63.201 ms  67.610 ms  68.012 ms
10  ae0d0.cir1.denver2-co.us.xo.net (207.88.13.181)  68.022 ms  68.020 ms  68.018 ms
11  206.111.8.66.ptr.us.xo.net (206.111.8.66)  123.867 ms  124.625 ms  51.173 ms
12  ae-4-4.car1.Manchesteruk1.Level3.net (4.69.133.101)  172.689 ms  173.118 ms  174.250 ms
13  ae-4-4.car1.Manchesteruk1.Level3.net (4.69.133.101)  175.722 ms  177.157 ms  177.541 ms
14  te3-5.cr2.man.bytemark.co.uk (195.50.119.78)  169.884 ms  172.161 ms  173.243 ms
15  po12.cr1.man.bytemark.co.uk (91.223.58.132)  180.825 ms  180.858 ms  180.887 ms
16  te0-1.cr1.lds.bytemark.co.uk (91.223.58.69)  182.093 ms  178.097 ms  179.155 ms
17  te1-6.cr1.yrk.bytemark.co.uk (91.223.58.76)  179.674 ms  176.203 ms  178.130 ms
18  po1.ar1.dc1.yo26.yrk.bytemark.co.uk (91.223.58.29)  179.275 ms  180.694 ms  181.078 ms
19  www.gentoo.org (89.16.167.134)  182.066 ms  256.507 ms *
[INFO    ] Executing command 'traceroute --version' in directory '/root'
[DEBUG   ] output: Modern traceroute for Linux, version 2.0.20, Aug 19 2014
Copyright (c) 2008  Dmitry Butskoy,   License: GPL v2 or any later
[DEBUG   ] traceroute_version_raw: ('2', '0', '20')
[DEBUG   ] traceroute_version: [2, 0, 20]
[WARNING ] The minion function caused an exception
Traceback (most recent call last):
  File "/root/salt-checkout/salt/minion.py", line 1020, in _thread_return
    return_data = func(*args, **kwargs)
  File "/root/salt-checkout/salt/modules/network.py", line 372, in traceroute
    'ms3': float(comps[4].split()[0])}
IndexError: list index out of range
[INFO    ] Returning information for job: 20150325122712207206
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
@tjstansell

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2015

The problem looks like it's the single * for one of the three icmp responses. Parsing traceroute output can get pretty tricky (even for a "successful" run) ... take this example, for instance, from my mac:

traceroute to e10088.dspb.akamaiedge.net (23.73.127.162), 64 hops max, 52 byte packets
 1  192.168.1.254 (192.168.1.254)  1.986 ms  1.311 ms  1.262 ms
 2  * * *
 3  75.29.64.146 (75.29.64.146)  23.313 ms  22.631 ms  21.859 ms
 4  12.83.77.137 (12.83.77.137)  24.599 ms
    12.83.77.145 (12.83.77.145)  23.323 ms  22.179 ms
 5  12.122.114.53 (12.122.114.53)  24.742 ms  26.020 ms  27.172 ms
 6  att-gw.sj.alter.net (192.205.36.62)  26.105 ms  26.298 ms  25.301 ms
 7  0.xe-6-0-1.xl3.sjc7.alter.net (152.63.51.241)  26.519 ms  26.668 ms
    0.xe-6-0-2.xl3.sjc7.alter.net (152.63.51.245)  26.003 ms
 8  tengige0-5-0-4.gw6.sjc7.alter.net (152.63.51.34)  33.524 ms
    tengige0-4-0-3.gw6.sjc7.alter.net (152.63.52.54)  37.091 ms
    tengige0-7-0-0.gw6.sjc7.alter.net (152.63.49.194)  28.939 ms
 9  23.73.127.162 (23.73.127.162)  27.373 ms  27.545 ms  27.559 ms
@arnisoph

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2015

# salt-call -l debug network.traceroute arnoldbechtoldt.com
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: master1.saltbox.local.inovex.de
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] The `lspci` binary is not available on the system. GPU grains will not be available.
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'master1.saltbox.local.inovex.de', 'tcp://127.0.0.1:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (3982)
[DEBUG   ] Setting zmq_reconnect_ivl to '3982ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'master1.saltbox.local.inovex.de', 'tcp://127.0.0.1:4506', 'aes')
[DEBUG   ] Initializing new SAuth for ('/etc/salt/pki/minion', 'master1.saltbox.local.inovex.de', 'tcp://127.0.0.1:4506')
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'master1.saltbox.local.inovex.de', 'tcp://127.0.0.1:4506', 'clear')
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded network.traceroute
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command 'traceroute arnoldbechtoldt.com' in directory '/root'
[DEBUG   ] output: traceroute to arnoldbechtoldt.com (185.26.156.41), 30 hops max, 60 byte packets
 1  10.0.2.2 (10.0.2.2)  0.406 ms  0.315 ms  0.235 ms
 2  192.168.2.1 (192.168.2.1)  1.064 ms  1.590 ms  1.499 ms
 3  * * *
 4  172.30.21.93 (172.30.21.93)  19.760 ms  19.687 ms  20.020 ms
 5  84.116.191.53 (84.116.191.53)  18.403 ms  19.125 ms  19.062 ms
 6  * de-kae01a-rd1-ae0.kae.unity-media.net (78.42.40.0)  10.509 ms  17.072 ms
 7  84.116.191.2 (84.116.191.2)  17.990 ms  18.607 ms  14.695 ms
 8  193.159.164.161 (193.159.164.161)  20.404 ms 80.157.131.181 (80.157.131.181)  18.435 ms 193.159.164.161 (193.159.164.161)  20.279 ms
 9  f-ee3-i.F.DE.NET.DTAG.DE (62.154.14.250)  21.263 ms  21.175 ms  21.281 ms
10  193.159.165.118 (193.159.165.118)  19.969 ms  19.917 ms *
11  37.208.104.113 (37.208.104.113)  23.973 ms *  11.977 ms
12  37.208.110.58 (37.208.110.58)  22.672 ms  27.386 ms  27.263 ms
13  schedar.uberspace.de (185.26.156.41)  12.519 ms !X  19.313 ms !X  20.426 ms !X
[INFO    ] Executing command 'traceroute --version' in directory '/root'
[DEBUG   ] output: Modern traceroute for Linux, version 2.0.19, Jun 10 2014
Copyright (c) 2008  Dmitry Butskoy,   License: GPL v2 or any later
[DEBUG   ] traceroute_version_raw: ('2', '0', '19')
[DEBUG   ] traceroute_version: [2, 0, 19]
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
IndexError: list index out of range
Traceback (most recent call last):
  File "/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 331, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 51, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 133, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 196, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/network.py", line 639, in traceroute
    'ms3': float(comps[4].split()[0])}
IndexError: list index out of range
Traceback (most recent call last):
  File "/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 331, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 51, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 133, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 196, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/network.py", line 639, in traceroute
    'ms3': float(comps[4].split()[0])}
IndexError: list index out of range
[root@master1 ~]# salt-call --versions
Salt Version:
           Salt: 2015.8.0-27-gcf59a03

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)
           RAET: Not Installed
        Tornado: 4.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.0.1406 Core
        machine: x86_64
        release: 3.10.0-123.el7.x86_64
         system: CentOS Linux 7.0.1406 Core
@stale

This comment has been minimized.

Copy link

commented Dec 30, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Dec 30, 2017
@stale stale bot closed this Jan 6, 2018
@akrus

This comment has been minimized.

Copy link

commented Mar 28, 2019

This bug is still present with 2019.2.0.

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1660, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/salt/modules/network.py", line 940, in traceroute
        'ms2': float(comps[3].split()[0]),
    IndexError: list index out of range
@jfindlay

This comment has been minimized.

Copy link
Contributor Author

commented Apr 1, 2019

I have created a proposed fix: skip lines that cause an IndexError and log them at warning level to the minion log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.