Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] cannot get full output using Salt CLi net.cli function napalm_network module is large #59167

Open
saifk-20786 opened this issue Dec 18, 2020 · 4 comments
Assignees
Labels
Bug broken, incorrect, or confusing behavior Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged
Milestone

Comments

@saifk-20786
Copy link

saifk-20786 commented Dec 18, 2020

Description
When running commands on Salt CLI like net.cli function from NAPALM_NETWORK MODULE of Salt ,If the output/response of the command is large it simply trims the output completely.
In other words Salt is not able to to fetch switch’s routing table, other than a very small VRF table with 100 - 200 routes. If at all the data is managed to be received with 100 - 200, it is taking more than 5 mins. Otherwise, it simply trims the data completely and display "VALUE_TRIMMED" as a result returned.
And trying to run the same command 2 or more time will result in
""
Minion did not return. [Not connected]
"""
I have tried changing the the master and minions configuration with max_event_size to 5242880 and kept increasing this to check if this would work. But no luck in getting the full output for the command
Also tried passing the flag to increase the timing to wait for the response from the minion using --timeout flag, This also doesn't seem to help.

Whereas, When using a python script or calling pyeapi I can get the complete output of the command with in 2 -3 seconds.
For example:
from napalm import get_network_driver
driver = get_network_driver('eos')
device = driver(hostname='arista-sw-d2', usename='username', password='password')
device.open()
device.cli(['show ip route vrf all'])
---output snipped ----

Setup
VM1 configured as salt-master

salt-master service running on Ubuntu 16.04
salt-master version 3001.1
memory 48 GB
cpus 20

VM2 configured as salt-minion for running the proxy services for network devices using napalm

salt-minion service running on Ubuntu 16.04
salt-minion version 3001.1
memory 64 GB
cpus 24
napalm version 2.5.0

https://docs.saltstack.com/en/3000/ref/proxy/all/salt.proxy.napalm.html#module-salt.proxy.napalm

Steps to Reproduce the behavior
tried many ways of executing the command show ip route vrf all from Salt on the device.
Using napalm.call

root@dc4-net-salt-prd-01:~# salt arista-sw-d2 napalm.call cli commands=["show ip route vrf all"]
arista-sw-d2:
    ----------
    comment:
    out:
        ----------
        show ip route vrf all:
            VALUE_TRIMMED
    result:
        True
Using pyeapi
salt arista-sw-d2.dc4-in.nvidia.com pyeapi.run_commands "show ip route vrf all" host='arista-sw-d2' username='username' password='********' transport='https' -t 300
arista-sw-d2:
    ----------
    comment:
    out:
        ----------
        show ip route vrf all:
            VALUE_TRIMMED
    result:
        True
Using net.cli
salt arista-sw-d2 net.cli "show ip route vrf all" -t 300
arista-sw-d2:
    ----------
    comment:
    out:
        ----------
        show ip route vrf all:
            VALUE_TRIMMED
    result:
        True

Expected behavior
Running command "show ip route vrf all" on a network device using net.cli should display the complete output, not the "VALUE_TRIMMED"

Screenshots
Added screenshots to help explain the problem.
image

Versions Report

salt --versions-report
root:~# salt --versions-report
Salt Version:
           Salt: 3001.1

Dependency Versions:
           cffi: 1.14.0
       cherrypy: 3.2.3
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
         Jinja2: 2.11.2
        libgit2: 0.24.0
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: 0.24.0
         Python: 3.5.2 (default, Oct  7 2020, 17:19:02)
   python-gnupg: 0.3.8
         PyYAML: 5.3.1
          PyZMQ: 19.0.1
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: ubuntu 16.04 Xenial Xerus
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-193-generic
         system: Linux
        version: Ubuntu 16.04 Xenial Xerus

There is no differences in master/minion versions.
(Provided by running salt --versions-report.)

Additional context
Add any other context about the problem here.

@saifk-20786 saifk-20786 added the Bug broken, incorrect, or confusing behavior label Dec 18, 2020
@ksator
Copy link

ksator commented Jan 3, 2021

Hello, just curious, are you facing the same issue if you output to a file?
salt 'arista-sw-d2' net.cli "show ip route vrf all" --output-file=routes.txt

@saifk-20786
Copy link
Author

Hi Khelil,

Yes, I am facing the same issue if I output to a file.

@sagetherage sagetherage changed the title [BUG] [BUG] cannot get full output using Salt CLi net.cli function napalm_network module is large Jan 4, 2021
@sagetherage sagetherage assigned garethgreenaway and unassigned Ch3LL Jan 4, 2021
@garethgreenaway
Copy link
Contributor

@saifk-20786 Thanks for the report. It looks like you are definitely hitting a limit somewhere, increasing the event limit should allow it to work, so it must be somewhere else. Do you have a sense of how much data is being returned from that command?

@garethgreenaway garethgreenaway added Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged and removed needs-triage labels Jan 13, 2021
@garethgreenaway garethgreenaway added this to the Blocked milestone Jan 13, 2021
@dmulyalin
Copy link

Having same issue - salt returns VALUE_TRIMED after certain size - in my case it is about about 3000 lines of text - before that limit can see full output, after that limit VALUE_TRIMMED returned instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged
Projects
None yet
Development

No branches or pull requests

6 participants