Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dell S6100: snmpwalk for IF-MIB and IP-MIB fails #118

Open
chitra-raghavan opened this issue Oct 23, 2017 · 17 comments
Open

Dell S6100: snmpwalk for IF-MIB and IP-MIB fails #118

chitra-raghavan opened this issue Oct 23, 2017 · 17 comments
Labels
bug question

Comments

@chitra-raghavan
Copy link
Contributor

Snmpwalk for IF-MIB and IP-MIB provided in library files are failing in S6100,whereas System mibs are passing .
In S6000 Snmpwalk for all the provided MIBs are passing.

Working mibs in s6100:

Snip from library file : /var/johnar/sonic-mgmt/ansible/library/snmp_facts.py

    # From SNMPv2-MIB
    self.sysDescr    = dp + "1.3.6.1.2.1.1.1.0"
    self.sysObjectId = dp + "1.3.6.1.2.1.1.2.0"
    self.sysUpTime   = dp + "1.3.6.1.2.1.1.3.0"
    self.sysContact  = dp + "1.3.6.1.2.1.1.4.0"
    self.sysName     = dp + "1.3.6.1.2.1.1.5.0"
    self.sysLocation = dp + "1.3.6.1.2.1.1.6.0"

Sample o/p in S6100:

rchitra@login-sjc-01[3:28am] : SONIC >  snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.1.1.0
SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.HEAD.361-f7cdca4 - HwSku: 		Force10-S6100 - Distribution: Debian 8.9 - Kernel: 3.16.0-4-amd64

rchitra@login-sjc-01[3:28am] : SONIC >   snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.1.2.0
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10

Failing mibs in s6100 :

The below oid’s are failing in 6100 .All these OIDs are running fine in S6000

Snip from library file :

   # From IF-MIB
    self.ifIndex       = dp + "1.3.6.1.2.1.2.2.1.1"
    self.ifDescr       = dp + "1.3.6.1.2.1.2.2.1.2"
    self.ifMtu         = dp + "1.3.6.1.2.1.2.2.1.4"
    self.ifSpeed       = dp + "1.3.6.1.2.1.2.2.1.5"
    self.ifPhysAddress = dp + "1.3.6.1.2.1.2.2.1.6"
    self.ifAdminStatus = dp + "1.3.6.1.2.1.2.2.1.7"
    self.ifOperStatus  = dp + "1.3.6.1.2.1.2.2.1.8"
    self.ifAlias       = dp + "1.3.6.1.2.1.31.1.1.1.18"
    self.ifInDiscards  = dp + "1.3.6.1.2.1.2.2.1.13"
    self.ifOutDiscards = dp + "1.3.6.1.2.1.2.2.1.19"
    self.ifInErrors    = dp + "1.3.6.1.2.1.2.2.1.14"
    self.ifOutErrors   = dp + "1.3.6.1.2.1.2.2.1.20"
    self.ifHCInOctets  = dp + "1.3.6.1.2.1.31.1.1.1.6"
    self.ifHCOutOctets = dp + "1.3.6.1.2.1.31.1.1.1.10"
    self.ifInUcastPkts = dp + "1.3.6.1.2.1.2.2.1.11"
    self.ifOutUcastPkts= dp + "1.3.6.1.2.1.2.2.1.17"

    # From IP-MIB
    self.ipAdEntAddr    = dp + "1.3.6.1.2.1.4.20.1.1"
    self.ipAdEntIfIndex = dp + "1.3.6.1.2.1.4.20.1.2"
    self.ipAdEntNetMask = dp + "1.3.6.1.2.1.4.20.1.3"

    # From Dell Private MIB
    self.ChStackUnitCpuUtil5sec = dp + "1.3.6.1.4.1.6027.3.10.1.2.9.1.2.1"

Sample o/p inS6100 :

rchitra@login-sjc-01[3:30am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2.2.1.1
IF-MIB::ifIndex = No Such Object available on this agent at this OID
1.3.6.1.2.1.2.2 (ifTable)
rchitra@login-sjc-01[3:30am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2
IF-MIB::interfaces = No Such Object available on this agent at this OID
1.3.6.1.2.1.31 (ifMIB)
rchitra@login-sjc-01[3:32am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.31
IF-MIB::ifMIB = No Such Object available on this agent at this OID

1.3.6.1.2.1.4.20 (ipAddrTable)
rchitra@login-sjc-01[3:34am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.4.20
IP-MIB::ipAddrTable = No Such Object available on this agent at this OID
1.3.6.1.4.1.6027.3 (f10Mgmt)
rchitra@login-sjc-01[3:34am] : SONIC > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.4.1.6027.3
SNMPv2-SMI::enterprises.6027.3 = No Such Object available on this agent at this OID

S6000 sample o/p :

rchitra@login-sjc-01[4:08am] : SONIC > snmpwalk -v 2c -c public 10.11.150.18 1.3.6.1.2.1.2.2.1.2
IF-MIB::ifDescr.1 = STRING: fortyGigE0/0
IF-MIB::ifDescr.5 = STRING: fortyGigE0/4
IF-MIB::ifDescr.9 = STRING: fortyGigE0/8
IF-MIB::ifDescr.13 = STRING: fortyGigE0/12
IF-MIB::ifDescr.17 = STRING: fortyGigE0/16
IF-MIB::ifDescr.21 = STRING: fortyGigE0/20
IF-MIB::ifDescr.25 = STRING: fortyGigE0/24
IF-MIB::ifDescr.29 = STRING: fortyGigE0/28
IF-MIB::ifDescr.33 = STRING: fortyGigE0/32
IF-MIB::ifDescr.37 = STRING: fortyGigE0/36
IF-MIB::ifDescr.41 = STRING: fortyGigE0/40
IF-MIB::ifDescr.45 = STRING: fortyGigE0/44
IF-MIB::ifDescr.49 = STRING: fortyGigE0/48
IF-MIB::ifDescr.53 = STRING: fortyGigE0/52
IF-MIB::ifDescr.57 = STRING: fortyGigE0/56
IF-MIB::ifDescr.61 = STRING: fortyGigE0/60
IF-MIB::ifDescr.65 = STRING: fortyGigE0/64
IF-MIB::ifDescr.69 = STRING: fortyGigE0/68
IF-MIB::ifDescr.73 = STRING: fortyGigE0/72
IF-MIB::ifDescr.77 = STRING: fortyGigE0/76
IF-MIB::ifDescr.81 = STRING: fortyGigE0/80
IF-MIB::ifDescr.85 = STRING: fortyGigE0/84
IF-MIB::ifDescr.89 = STRING: fortyGigE0/88
IF-MIB::ifDescr.93 = STRING: fortyGigE0/92
IF-MIB::ifDescr.97 = STRING: fortyGigE0/96
IF-MIB::ifDescr.101 = STRING: fortyGigE0/100
IF-MIB::ifDescr.105 = STRING: fortyGigE0/104
IF-MIB::ifDescr.109 = STRING: fortyGigE0/108
IF-MIB::ifDescr.113 = STRING: fortyGigE0/112
IF-MIB::ifDescr.117 = STRING: fortyGigE0/116
IF-MIB::ifDescr.121 = STRING: fortyGigE0/120
IF-MIB::ifDescr.125 = STRING: fortyGigE0/124
@prsunny
Copy link
Contributor

prsunny commented Oct 23, 2017 •
edited

Did you check it on the latest image? There were few fixes made in the context of SNMP agent.
One is sonic-net/sonic-py-swsssdk#17

You can check the logs to see if there are any failures/crashes in SNMP agent initialization.

@stcheng
Copy link
Contributor

stcheng commented Oct 23, 2017

Hello, @chitra-raghavan we've run the SNMP test against the S6100 platform, and it passes fine.
Please check if you're on the latest version of the image. Thanks.

root@s6100:/home/admin# show version
SONiC Software Version: SONiC.HEAD.294-3417e78 
Distribution: Debian 8.9                       
Kernel: 3.16.0-4-amd64                         
Build commit: 3417e78

@chitra-raghavan
Copy link
Contributor Author

@prsunny , @stcheng I am using the latest version of the label and seeing the same issue.

root@sonic:/var/log# show ver
SONiC Software Version: SONiC.HEAD.368-3417e78
Distribution: Debian 8.9
Kernel: 3.16.0-4-amd64
Build commit: 3417e78
Build date: Mon Oct 23 12:58:23 UTC 2017
Built by: johnar@jenkins-worker-3

Docker images:
REPOSITORY                TAG                 IMAGE ID            SIZE
docker-syncd-brcm         HEAD.368-3417e78    fbe33656cfd3        318.3 MB
docker-syncd-brcm         latest              fbe33656cfd3        318.3 MB
docker-orchagent-brcm     HEAD.368-3417e78    06b3e355a5b9        259 MB
docker-orchagent-brcm     latest              06b3e355a5b9        259 MB
docker-lldp-sv2           HEAD.368-3417e78    e532679f9de4        256.5 MB
docker-lldp-sv2           latest              e532679f9de4        256.5 MB
docker-dhcp-relay         HEAD.368-3417e78    75c8ace8196a        253.5 MB
docker-dhcp-relay         latest              75c8ace8196a        253.5 MB
docker-database           HEAD.368-3417e78    14320c14aef1        251.7 MB
docker-database           latest              14320c14aef1        251.7 MB
docker-snmp-sv2           HEAD.368-3417e78    9ac3dd770004        291.3 MB
docker-snmp-sv2           latest              9ac3dd770004        291.3 MB
docker-teamd              HEAD.368-3417e78    c8a66e1bb150        256.1 MB
docker-teamd              latest              c8a66e1bb150        256.1 MB
docker-platform-monitor   HEAD.368-3417e78    7b130159ec57        271 MB
docker-platform-monitor   latest              7b130159ec57        271 MB
docker-fpm-quagga         HEAD.368-3417e78    80774276af0d        262.6 MB
docker-fpm-quagga         latest              80774276af0d        262.6 MB

I see the below messages in syslog file.

Oct 24 06:35:20.505886 sonic-s6100-01 INFO supervisord: snmp-subagent Traceback (most recent call last):
Oct 24 06:35:20.505970 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
Oct 24 06:35:20.506004 sonic-s6100-01 INFO supervisord: snmp-subagent     "__main__", mod_spec)
Oct 24 06:35:20.506038 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
Oct 24 06:35:20.506073 sonic-s6100-01 INFO supervisord: snmp-subagent     exec(code, run_globals)
Oct 24 06:35:20.506107 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/__main__.py", line 74, in <module>
Oct 24 06:35:20.506157 sonic-s6100-01 INFO supervisord: snmp-subagent     from .main import main
Oct 24 06:35:20.506190 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/main.py", line 12, in <module>
Oct 24 06:35:20.506223 sonic-s6100-01 INFO supervisord: snmp-subagent     from sonic_ax_impl.mibs import ieee802_1ab
Oct 24 06:35:20.506257 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 122, in <module>
Oct 24 06:35:20.506290 sonic-s6100-01 INFO supervisord: snmp-subagent     _lldp_updater = LLDPUpdater()
Oct 24 06:35:20.506323 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 34, in __init__
Oct 24 06:35:20.506374 sonic-s6100-01 INFO supervisord: snmp-subagent     self.reinit_data()
Oct 24 06:35:20.506406 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 50, in reinit_data
Oct 24 06:35:20.506440 sonic-s6100-01 INFO supervisord: snmp-subagent     self.oid_name_map = mibs.init_sync_d_interface_tables(self.db_conn)
Oct 24 06:35:20.506473 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/sonic_ax_impl/mibs/__init__.py", line 77, in init_sync_d_interface_tables
Oct 24 06:35:20.506524 sonic-s6100-01 INFO supervisord: snmp-subagent     if_name_map, if_id_map = port_util.get_interface_oid_map(db_conn)
Oct 24 06:35:20.506557 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/port_util.py", line 40, in get_interface_oid_map
Oct 24 06:35:20.506609 sonic-s6100-01 INFO supervisord: snmp-subagent     if_name_map = db.get_all('COUNTERS_DB', 'COUNTERS_PORT_NAME_MAP', blocking=True)
Oct 24 06:35:20.506643 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 38, in wrapped
Oct 24 06:35:20.506676 sonic-s6100-01 INFO supervisord: snmp-subagent     ret_data = f(inst, db_name, *args, **kwargs)
Oct 24 06:35:20.506709 sonic-s6100-01 INFO supervisord: snmp-subagent   File "/usr/local/lib/python3.6/dist-packages/swsssdk/interface.py", line 303, in get_all
Oct 24 06:35:20.506742 sonic-s6100-01 INFO supervisord: snmp-subagent     raise UnavailableDataError(message, _hash)
Oct 24 06:35:20.506775 sonic-s6100-01 INFO supervisord: snmp-subagent swsssdk.exceptions.UnavailableDataError: Key 'COUNTERS_PORT_NAME_MAP' unavailable in database 'COUNTERS_DB'
Oct 24 06:35:20.552338 sonic-s6100-01 INFO snmp.sh[9294]: 2017-10-24 06:35:20,551 INFO exited: snmp-subagent (exit status 1; not expected)
Oct 24 06:35:29.123978 sonic-s6100-01 INFO supervisord 2017-10-24 06:35:20,551 INFO exited: snmp-subagent (exit status 1; not expected)

@prsunny
Copy link
Contributor

prsunny commented Oct 24, 2017 •
edited by stcheng

Looks like it is due the following pull-requests.

sonic-net/sonic-sairedis#234
sonic-net/sonic-swss#331

We are tracking this!

@stcheng
Copy link
Contributor

stcheng commented Nov 2, 2017

we've fixed this. please confirm.

@chitra-raghavan
Copy link
Contributor Author

seeing same issue in build 376. snmpwalk doesnt return any output.
version :

root@sonic-s6100-01:~# show ver
SONiC Software Version: SONiC.HEAD.376-d7d6c41
Distribution: Debian 8.9
Kernel: 3.16.0-4-amd64
Build commit: d7d6c41
Build date: Tue Oct 31 13:13:47 UTC 2017
Built by: johnar@jenkins-worker-3

Docker images:
REPOSITORY                TAG                 IMAGE ID            SIZE
docker-syncd-brcm         HEAD.376-d7d6c41    c9c0917e1f09        318.3 MB
docker-syncd-brcm         latest              c9c0917e1f09        318.3 MB
docker-orchagent-brcm     HEAD.376-d7d6c41    d1040046a4c5        259.1 MB
docker-orchagent-brcm     latest              d1040046a4c5        259.1 MB
docker-lldp-sv2           HEAD.376-d7d6c41    4bf6be6d4f13        256.5 MB
docker-lldp-sv2           latest              4bf6be6d4f13        256.5 MB
docker-dhcp-relay         HEAD.376-d7d6c41    99e930f82d84        253.5 MB
docker-dhcp-relay         latest              99e930f82d84        253.5 MB
docker-database           HEAD.376-d7d6c41    72344548f653        251.7 MB
docker-database           latest              72344548f653        251.7 MB
docker-snmp-sv2           HEAD.376-d7d6c41    b0f9c9a36e9a        291.3 MB
docker-snmp-sv2           latest              b0f9c9a36e9a        291.3 MB
docker-teamd              HEAD.376-d7d6c41    c74659900c4d        256.1 MB
docker-teamd              latest              c74659900c4d        256.1 MB
docker-platform-monitor   HEAD.376-d7d6c41    58881d29a46e        271 MB
docker-platform-monitor   latest              58881d29a46e        271 MB
docker-fpm-quagga         HEAD.376-d7d6c41    19b478542bc7        262.7 MB
docker-fpm-quagga         latest              19b478542bc7        262.7 MB

snmp o/p :
version:

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.1.1.0
SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.HEAD.376-d7d6c41 - HwSku: Force10-S6100 - Distribution: Debian 8.9 - Kernel: 3.16.0-4-amd64

IF-MIB :

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2.2.1.1
IF-MIB::ifIndex = No Such Object available on this agent at this OID

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.2
IF-MIB::interfaces = No Such Object available on this agent at this OID

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.31
IF-MIB::ifMIB = No Such Object available on this agent at this OID

IP-MIB:

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.4.20
IP-MIB::ipAddrTable = No Such Object available on this agent at this OID

rchitra@login-sjc-01[1:58am] : ~ >

@qiluo-msft
Copy link
Contributor

qiluo-msft commented Nov 18, 2017 •
edited

@chitra-raghavan The syslog you pasted on Oct 23 is saying the 'COUNTERS_PORT_NAME_MAP' is not available in redis-db. You may double check by command.

redis-cli -n 2 -c hgetall COUNTERS_PORT_NAME_MAP

Normally it is due to very slow initialization on this platform. We are trying to solve this problem (@yxieca ). Please confirm if it is the case. If yes, mitigating solution is to restart snmp docker:

sudo systemctl restart snmp

@chitra-raghavan
Copy link
Contributor Author

After restarting the snmp docker , snmpwalk for IF-MIB works fine. (sudo systemctl restart snmp)

@chitra-raghavan
Copy link
Contributor Author

snmpwalk for IP-MIB still fails.

rchitra@login-sjc-01[1:57am] : ~ > snmpwalk -v 2c -c public 10.11.150.100 1.3.6.1.2.1.4.20
IP-MIB::ipAddrTable = No Such Object available on this agent at this OID

@qiluo-msft
Copy link
Contributor

My original comment still applies if you met the same error in syslog.

@chitra-raghavan
Copy link
Contributor Author

Even after restarting the snmp , IP-MIB doesnt work.
IF_MIB alone works fine after restarting the snmp

@qiluo-msft
Copy link
Contributor

qiluo-msft commented Nov 24, 2017 •
edited

@chitra-raghavan Sorry, I thought you were talking about IF-MIB.
I double checked the code and lab device, that current snmpagent implemented partial IP-MIB.
If you query "1.3.6.1.2.1.4", you will have some idea.

If you want to check code, you may read: class IpMib, class IpCidrRouteTable.

What do you really need in IP-MIB? Feel free to submit a pull request if you would like to contribute.

@chitra-raghavan
Copy link
Contributor Author

it is fine if the oid polled in the sonic-mgmt script is enough for now

@rraghav-cisco
Copy link

Any resolution on this ? The latest sonic builds are still seeing this problem. On 2 different vendor platforms.
Thanks,
rraghav

@scottlaird
Copy link

I'm also seeing this, on a fresh install of build 306 from Jenkins (a day or two old), on an as7326_56x (Broadcom). snmpwalk shows IP-MIB but not IF-MIB.

@scottlaird
Copy link

Found in syslog:

./syslog.1:Jun 10 17:57:29.089708 sonic ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012 File "/usr/local/lib/python3.7/dist-packages/ax_interface/mib.py", line 43, in start#012 self.update_data()#012 File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ietf/rfc2863.py", line 98, in update_data#012 for sai_id in self.if_id_map}#012 File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ietf/rfc2863.py", line 98, in <dictcomp>#012 for sai_id in self.if_id_map}#012 File "/usr/local/lib/python3.7/dist-packages/swsssdk/dbconnector.py", line 308, in get_all#012 return super(SonicV2Connector, self).get_all(db_id, _hash, *args, **kwargs)#012 File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 38, in wrapped#012 ret_data = f(inst, db_id, *args, **kwargs)#012 File "/usr/local/lib/python3.7/dist-packages/swsssdk/interface.py", line 303, in get_all#012 raise UnavailableDataError(message, _hash)#012swsssdk.exceptions.UnavailableDataError: Key 'b'COUNTERS:oid:0x1000000000009'' unavailable in database '2'

I'm not that familiar with Redis, but it doesn't look like that key exists:

$ redis-cli -n 2 -c keys COUNTERS:oid:0x10*
1) "COUNTERS:oid:0x1000000000002"
2) "COUNTERS:oid:0x1000000000006"
3) "COUNTERS:oid:0x1000000000007"
4) "COUNTERS:oid:0x1000000000008"
5) "COUNTERS:oid:0x1000000000003"
6) "COUNTERS:oid:0x1000000000005"
7) "COUNTERS:oid:0x1000000000004"```

@scottlaird
Copy link

FWIW, after upgrading to build 310 and rebooting, IF-MIB entries now appear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@scottlaird @stcheng @qiluo-msft @prsunny @chitra-raghavan @rraghav-cisco