Skip to content

[BUG] Salt Deltaproxy NAPALM Issues - PR #60090 #60177

Closed
@ggiesen

Description

@ggiesen

Description
A number of issues related to the open-sourcing of deltaproxy in PR #60090 when using NAPALM as the proxy type.

  1. Slow startup. Running in debug mode, the deltaproxy appears to connect to each device sequentially and issue a number of commands. When testing a 21 node setup on a mix of (admittedly slow) IOS and NX-OS devices, it took 13 minutes to connect to all of the devices
  2. Only the last device in the list stays connected after the intial connection setup:

router1:

router1#show users
    Line       User       Host(s)              Idle       Location
*  1 vty 0     real_user  idle                 00:00:00 198.51.100.254

router2:

router2#show users
    Line       User       Host(s)              Idle       Location
*  1 vty 0     real_user  idle                 00:00:00 198.51.100.254

router3:

router3#show users
    Line       User       Host(s)              Idle       Location
   1 vty 0     salt_user  idle                 00:00:27 198.51.100.1
*  2 vty 1     real_user  idle                 00:00:00 198.51.100.254
  1. All commands that target any device in the list of ids are responded to by the last device only, regardless of whether or not that device is included in the list:
$ salt router1 net.facts
router1:
    ----------
    comment:
    out:
        ----------
        fqdn:
            router3.example.com
        hostname:
            router3
        interface_list:
            - TenGigabitEthernet5/2
            - GigabitEthernet5/3
            - TenGigabitEthernet6/1
            - TenGigabitEthernet6/2
            - GigabitEthernet6/3
            - Loopback0
        model:
            WS-C6509
        os_version:
            s3223_rp Software (s3223_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXJ10, RELEASE SOFTWARE (fc3)
        serial_number:
            ABC00000003
        uptime:
            48431880
        vendor:
            Cisco
    result:
        True
$ salt router2 net.facts
router2:
    ----------
    comment:
    out:
        ----------
        fqdn:
            router3.example.com
        hostname:
            router3
        interface_list:
            - TenGigabitEthernet5/2
            - GigabitEthernet5/3
            - TenGigabitEthernet6/1
            - TenGigabitEthernet6/2
            - GigabitEthernet6/3
            - Loopback0
        model:
            WS-C6509
        os_version:
            s3223_rp Software (s3223_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXJ10, RELEASE SOFTWARE (fc3)
        serial_number:
            ABC00000003
        uptime:
            48432420
        vendor:
            Cisco
    result:
        True
$ salt router3 net.facts
router3:
    ----------
    comment:
    out:
        ----------
        fqdn:
            router3.example.com
        hostname:
            router3
        interface_list:
            - TenGigabitEthernet5/2
            - GigabitEthernet5/3
            - TenGigabitEthernet6/1
            - TenGigabitEthernet6/2
            - GigabitEthernet6/3
            - Loopback0
        model:
            WS-C6509
        os_version:
            s3223_rp Software (s3223_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXJ10, RELEASE SOFTWARE (fc3)
        serial_number:
            ABC00000003
        uptime:
            48432480
        vendor:
            Cisco
    result:
        True

Setup
Salt master and proxy are running 3003 with patches from #60090

/etc/salt/proxy:

master: 198.51.100.1
...
metaproxy: deltaproxy

/srv/pillar/top.sls:

  deltaproxy:
    - deltaproxy
  router1:
    - router1
  router2:
    - router2
  router3:
    - router3

/srv/pillar/deltaproxy.sls:

proxy:
  proxytype: deltaproxy
  ids:
    - router1
    - router2
    - router3

/srv/pillar/router1.sls:

proxy:
  driver: ios
  host: 192.0.2.1
  proxytype: napalm
  username: salt_user
  passwd: salt_pass
  multiprocessing: False
  optional_args:
    global_delay_factor: 20

/srv/pillar/router2.sls:

proxy:
  driver: ios
  host: 192.0.2.2
  proxytype: napalm
  username: salt_user
  passwd: salt_pass
  multiprocessing: False
  optional_args:
    global_delay_factor: 20

/srv/pillar/router3.sls:

proxy:
  driver: ios
  host: 192.0.2.3
  proxytype: napalm
  username: salt_user
  passwd: salt_pass
  multiprocessing: False
  optional_args:
    global_delay_factor: 20

Steps to Reproduce the behavior
$ salt-proxy --proxyid=deltaproxy -l debug

Expected behavior

  1. Ideally, deltaproxies should setup/connect in parallel to speed up startup time
  2. All devices should stay connected
  3. Commands should execute on the correct device

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3003

Dependency Versions:
          cffi: 1.14.0
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.2
       libgit2: 1.0.1
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 1.3.0
        Python: 3.6.8 (default, Aug 24 2020, 17:57:11)
  python-gnupg: 0.4.6
        PyYAML: 5.3.1
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: centos 8
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-240.15.1.el8_3.x86_64
        system: Linux
       version: CentOS Linux 8

Additional context
Using NAPALM 3.2.0

Metadata

Metadata

Labels

Bugbroken, incorrect, or confusing behaviorDelta-ProxySilicon v3004.0Release code nameseverity-high2nd top severity, seen by most users, causes major problems

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions