junos.install_config and netconfig.managed output showing KeyError: u'junos.conn', but set/conf file is successfully applied to router via netconf

when running sudo salt bjn-eu1-e2 state.apply DrainEU1SG1secondary i am getting an error, see below. But while saltstack is showing an error, on the router the commits for the Drain and UnDrainSJ1SG1 sls are succesful. i have confirmation below via rollback and i have confirmation via oxidizer that the state.apply above was successful in both attempts even though the output shown for saltstack cli is failed. you will also see the minion output below that shows napalm is installed on the minion host

[2019-02-03 08:32:40-0800] saltmaster% sudo salt bjn-eu1-e2 state.apply DrainEU1SG1secondary
bjn-eu1-e2:
----------
          ID: DrainEU1SG1secondary
    Function: netconfig.managed
      Result: False
     Comment: State 'netconfig.managed' was not found in SLS 'DrainEU1SG1secondary'
              Reason: 'netconfig' __virtual__ returned False: "netconfig"" (/usr/lib/python2.7/dist-packages/salt/states/netconfig.pyc) cannot be loaded: NAPALM is not installed or not running in a (proxy) minion
     Changes:

Summary for bjn-eu1-e2
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms

[2019-02-03 09:17:50-0800] saltmaster% sudo salt bjn-eu1-e2 state.apply UnDrainEU1SG1secondary
bjn-eu1-e2:
----------
          ID: UnDrainEU1SG1secondary
    Function: netconfig.managed
      Result: False
     Comment: State 'netconfig.managed' was not found in SLS 'UnDrainEU1SG1secondary'
              Reason: 'netconfig' __virtual__ returned False: "netconfig"" (/usr/lib/python2.7/dist-packages/salt/states/netconfig.pyc) cannot be loaded: NAPALM is not installed or not running in a (proxy) minion
     Changes:

Summary for bjn-eu1-e2
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
[DEBUG   ] jid 20190203091802152685 found all minions set([u'bjn-eu1-e2'])
ERROR: Minions returned with non-zero exit code

that is salt draining and undraining a circuit, confirmed via CLI

Possible completions:
  <[Enter]>            Execute this command
  0                    2019-02-03 09:18:08 PST by SALT via netconf
  1                    2019-02-03 08:32:56 PST by SALT via netconf

i also have confirmation via oxidizer in our slack channel that salt did commit the disable and passive to the rsvp and isis interface

-## Last commit: 2019-02-02 13:01:37 PST by ****
+## Last commit: 2019-02-03 08:32:56 PST by SALT
 version 17.4R1-S4.2;
 groups {
     global;
@@ -502,6 +502,7 @@ routing-options {
 protocols {
     rsvp {
         interface xe-0/1/1:1.0 {
+            disable;
             link-protection;
         }
         interface xe-0/1/0:2.0 {
@@ -711,6 +712,7 @@ protocols {
             level 2 metric 81;
         }
         interface xe-0/1/1:1.0 {
+            passive;
             level 2 metric 156;
         }
         interface lo0.0;

proxy:
 proxytype: napalm
 driver: junos
 host: ****
 username: SALT
 password: ****
 port: 22

removed lines so it didnt clutter the channel

minion1-40-201 ~ % sudo pip install napalm
The directory '/home/julius/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/julius/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: napalm in ./.local/lib/python2.7/site-packages (2.3.3)
Requirement already satisfied: junos-eznc>=2.1.5 in ./.local/lib/python2.7/site-packages (from napalm) (2.2.0)
Requirement already satisfied: enum34 in ./.local/lib/python2.7/site-packages (from netmiko>=2.1.1->napalm) (1.1.6)
Requirement already satisfied: ipaddress in ./.local/lib/python2.7/site-packages (from netmiko>=2.1.1->napalm) (1.0.22)
Requirement already satisfied: pyasn1>=0.1.7 in ./.local/lib/python2.7/site-packages (from paramiko>=1.15.2->junos-eznc>=2.1.5->napalm) (0.4.5)
Requirement already satisfied: bcrypt>=3.1.3 in ./.local/lib/python2.7/site-packages (from paramiko>=1.15.2->junos-eznc>=2.1.5->napalm) (3.1.6)

salt-call --versions-report
Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: ISO-8859-1
        machine: x86_64
        release: 4.4.0-124-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[Ch3LL]

hmmm that is odd that it still ran the state even though it could not load it.

In the code it runs this: if ((HAS_NAPALM and NAPALM_MAJOR >= 2) or HAS_NAPALM_BASE) and (is_proxy(opts) or is_minion(opts)): to verify if napalm is installed or not:

can you try the following:

python -c "import napalm"
python -c "import napalm.base as napalm_base"
python -c "import napalm; int(napalm.__version__.split('.')[0])"

it also verifies if its a napalm proxy minion or not by running: return salt.utils.platform.is_proxy() and opts.get('proxy', {}).get('proxytype') == 'napalm'

when i ran the commands you mentioned there was no output, but also no errors so i assume that was the intended outcome? this weekend i will run the state again as this is being done on a production network @Ch3LL

@Ch3LL still the same. i ran the state.apply twice and saltstack showed errors but again it committed to the router even with those errors. you will also find the router logs confirming saltstack did commit config via proxy set up in /srv/pillar/

Possible completions:
  <[Enter]>            Execute this command
  0                    2019-02-07 13:30:11 PST by SALT via netconf commit synchronize
  1                    2019-02-07 12:50:34 PST by SALT via netconf commit synchronize

# rollback 1
load complete

{master}[edit]
# show | compare
[edit protocols rsvp interface xe-0/0/7.0]
+    disable;
[edit protocols isis interface xe-0/0/7.0]
+     passive;

 sudo salt bjn-sj1-e2 state.apply DrainSJ1SG1secondary
bjn-sj1-e2:
----------
          ID: DrainSJ1SG1secondary
    Function: netconfig.managed
      Result: False
     Comment: State 'netconfig.managed' was not found in SLS 'DrainSJ1SG1secondary'
              Reason: 'netconfig' __virtual__ returned False: "netconfig"" (/usr/lib/python2.7/dist-packages/salt/states/netconfig.pyc) cannot be loaded: NAPALM is not installed or not running in a (proxy) minion
     Changes:

Summary for bjn-sj1-e2
------------
Succeeded: 0
Failed:    1

show rsvp interface
RSVP interface: 3 active
                          Active  Subscr- Static      Available   Reserved    Highwater
Interface          State  resv    iption  BW          BW          BW          mark
et-0/3/0.0             Up       6   100%  40Gbps      37.5Gbps    2.5Gbps     29Gbps
xe-0/0/0.0             Up       4   100%  10Gbps      9.5Gbps     500Mbps     10Gbps
xe-0/0/7.0             Up       0   100%  10Gbps      10Gbps      0bps        0bps

sudo salt bjn-sj1-e2 state.apply UnDrainSJ1SG1secondary
bjn-sj1-e2:
----------
          ID: UnDrainSJ1SG1secondary
    Function: junos.install_config
        Name: salt://UnDrainSJ1SG1secondary.conf
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1913, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1898, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/junos.py", line 330, in install_config
                  ret['changes'] = __salt__['junos.install_config'](name, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/modules/junos.py", line 805, in install_config
                  conn = __proxy__['junos.conn']()
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1155, in __getitem__
                  func = super(LazyLoader, self).__getitem__(item)
                File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 106, in __getitem__
                  return self._dict[key]
              KeyError: u'junos.conn'
     Started: 13:30:03.293856
    Duration: 2.232 ms
     Changes:

Summary for bjn-sj1-e2
------------
Succeeded: 0
Failed:    1
------------

show rsvp interface
RSVP interface: 3 active
                          Active  Subscr- Static      Available   Reserved    Highwater
Interface          State  resv    iption  BW          BW          BW          mark
et-0/3/0.0             Up       3   100%  40Gbps      39.5Gbps    500Mbps     29Gbps
xe-0/0/0.0             Up       5   100%  10Gbps      9.25Gbps    750Mbps     10Gbps
xe-0/0/7.0             Up       2   100%  10Gbps      8.25Gbps    1.75Gbps    1.75Gbps

anything on this @mirceaulinic @wasabi222 @Ch3LL

[Ch3LL]

ping @saltstack/team-network any ideas here?

[wasabi222]

What does the state file and template look like? Have you tried this in a test environment with a vSRX or vMX?

@wasabi222 ill paste them here. and no im not using vMX or vSRX. so the results you get from that would be irrelevant when im using a physical box on a specific Junos, unless you are able to load Junos 17.1R2S7. just my 2 cents it has nothing to do with the state file or conf file because it applies successfully to the router. the issue is the error that saltstack output shows which isnt relevant to what the actual device is saying. as there isnt an error on the device (because the conf applies successfully)

here is the state file and conf file

DrainSJ1SG1primary:
 netconfig.managed:
 - template_name: salt://DrainSJ1SG1primary.conf

        protocols {
            rsvp {
                interface xe-0/0/10 {
                    disable;
                }
            }
            isis {
                interface xe-0/0/10 {
                    passive;
                }
            }
        }
    }

@wasabi222 @Ch3LL junos.install_config does the same as netconfig.managed ....shows an error but when i check the router it was successfully applied please see below

Router Output confirming junos.install_config was applied successfully

sj1-e2-re0# rollback ?
Possible completions:
  <[Enter]>            Execute this command
  0                    2019-03-02 13:41:29 PST by SALT via netconf commit synchronize
  1                    2019-03-02 13:40:42 PST by SALT via netconf commit synchronize

sj1-e2-re0# rollback 1
load complete

sj1-e2-re0# show | compare
[edit protocols rsvp interface xe-0/0/7.0]
+    disable;
[edit protocols isis interface xe-0/0/7.0]
+     passive;

{master}[edit]
sj1-e2-re0# rollback 0

{master}[edit]
sj1-e2-re0# show | compare

{master}[edit]
sj1-e2-re0#



sj1-e2-re0> show rsvp interface
RSVP interface: 2 active
                          Active  Subscr- Static      Available   Reserved    Hi                                                                                        ghwater
Interface          State  resv    iption  BW          BW          BW          ma                                                                                        rk
et-0/3/0.0             Up       8   100%  40Gbps      37.25Gbps   2.75Gbps    29                                                                                        Gbps
xe-0/0/0.0             Up       4   100%  10Gbps      9.75Gbps    250Mbps     10                                                                                        Gbps


{master}
sj1-e2-re0> show rsvp interface
RSVP interface: 3 active
                          Active  Subscr- Static      Available   Reserved    Hi                                                                                        ghwater
Interface          State  resv    iption  BW          BW          BW          ma                                                                                        rk
et-0/3/0.0             Up       8   100%  40Gbps      37.25Gbps   2.75Gbps    29                                                                                        Gbps
xe-0/0/0.0             Up       4   100%  10Gbps      9.75Gbps    250Mbps     10                                                                                        Gbps
xe-0/0/7.0             Up       0   100%  10Gbps      10Gbps      0bps        0b                                                                                        ps
sj1-e2-re0> show rsvp interface
RSVP interface: 3 active
                          Active  Subscr- Static      Available   Reserved    Hi                                                                                        ghwater
Interface          State  resv    iption  BW          BW          BW          ma                                                                                        rk
et-0/3/0.0             Up       3   100%  40Gbps      39.75Gbps   250Mbps     29                                                                                        Gbps
xe-0/0/0.0             Up       7   100%  10Gbps      9Gbps       1000Mbps    10                                                                                        Gbps
xe-0/0/7.0             Up       2   100%  10Gbps      8.25Gbps    1.75Gbps    1.                                                                                        75Gbps

SaltStack CLI junos.install_config output

DrainSJ1SG1secondary.set and UnDrainSJ1SG1secondary.set configs 

set protocols rsvp interface xe-0/0/7 disable
set protocols isis interface xe-0/0/7 passive

delete protocols rsvp interface xe-0/0/7 disable
delete protocols isis interface xe-0/0/7 passive
_________________________________________________________


[2019-03-02 13:40:11-0800]saltmaster % sudo salt sj1-e2 junos.install_config 'salt://DrainSJ1SG1secondary.set'
sj1-e2:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1607, in _thread_return
        fp_.write(minion_instance.serial.dumps(sdata))
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/junos.py", line 805, in install_config
        conn = __proxy__['junos.conn']()
      File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1155, in __getitem__
        self.loaded_files = set()  # TODO: just remove them from file_mapping?
      File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 106, in __getitem__
        else:
    KeyError: u'junos.conn'



saltmaster % sudo salt sj1-e2 junos.install_config 'salt://UnDrainSJ1SG1secondary.set'
sj1-e2:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1607, in _thread_return
        fp_.write(minion_instance.serial.dumps(sdata))
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/junos.py", line 805, in install_config
        conn = __proxy__['junos.conn']()
      File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1155, in __getitem__
        self.loaded_files = set()  # TODO: just remove them from file_mapping?
      File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 106, in __getitem__
        else:
    KeyError: u'junos.conn'

Logs from the Master

2019-03-02 13:40:34,495 [salt.master      :1577][ERROR   ][9572] Received minion error from [sj1-e2]: The minion function caused an exception
2019-03-02 13:40:34,505 [salt.master      :1577][ERROR   ][9571] Received minion error from [sj1-e2]: The minion function caused an exception
2019-03-02 13:40:34,510 [salt.loaded.int.returner.local_cache:153 ][ERROR   ][9565] An extra return was detected from minion sj1-e2, please verify the minion, this could be a replay attack
2019-03-02 13:40:34,513 [salt.master      :1577][ERROR   ][9565] Received minion error from [sj1-e2]: The minion function caused an exception
2019-03-02 13:40:34,529 [salt.master      :1577][ERROR   ][9573] Received minion error from [sj1-e2]: The minion function caused an exception
2019-03-02 13:40:34,533 [salt.loaded.int.returner.local_cache:153 ][ERROR   ][9562] An extra return was detected from minion sj1-e2, please verify the minion, this could be a replay attack

i have a new salt version as well

 salt --versions-report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.12.2
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: ISO-8859-1
        machine: x86_64
        release: 4.4.0-124-generic
         system: Linux
        version: Ubuntu 16.04 xenial