Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pmon] xcvrd failed to start on Arista-7260 #6341

Closed
bingwang-ms opened this issue Jan 4, 2021 · 2 comments · Fixed by sonic-net/sonic-platform-common#157 or #6352
Closed

[pmon] xcvrd failed to start on Arista-7260 #6341

bingwang-ms opened this issue Jan 4, 2021 · 2 comments · Fixed by sonic-net/sonic-platform-common#157 or #6352

Comments

@bingwang-ms
Copy link
Contributor

Description
xcvrd failed to start on Arista-7260, which results in pmon service failed to start.

Jan  4 02:29:34.281759 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd Traceback (most recent call last):
Jan  4 02:29:34.282026 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/bin/xcvrd", line 8, in <module>
Jan  4 02:29:34.282078 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     sys.exit(main())
Jan  4 02:29:34.282078 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 1376, in main
Jan  4 02:29:34.282091 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     xcvrd.run()
Jan  4 02:29:34.282091 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 1324, in run
Jan  4 02:29:34.282104 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     self.init()
Jan  4 02:29:34.282157 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 1289, in init
Jan  4 02:29:34.282157 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     post_port_sfp_dom_info_to_db(is_warm_start, self.stop_event)
Jan  4 02:29:34.282196 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 486, in post_port_sfp_dom_info_to_db
Jan  4 02:29:34.282196 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     post_port_sfp_info_to_db(logical_port_name, int_tbl[asic_index], transceiver_dict, stop_event)
Jan  4 02:29:34.282196 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 280, in post_port_sfp_info_to_db
Jan  4 02:29:34.282217 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     port_info_dict = _wrapper_get_transceiver_info(physical_port)
Jan  4 02:29:34.282275 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python3.7/dist-packages/xcvrd/xcvrd.py", line 164, in _wrapper_get_transceiver_info
Jan  4 02:29:34.282289 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     return platform_chassis.get_sfp(physical_port).get_transceiver_info()
Jan  4 02:29:34.282363 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/lib/python3/dist-packages/arista/utils/sonic_platform/sfp.py", line 110, in get_transceiver_info
Jan  4 02:29:34.282363 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     return self._get_sfputil().get_transceiver_info_dict(self._index)
Jan  4 02:29:34.282424 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd   File "/usr/local/lib/python3.7/dist-packages/sonic_sfp/sfputilbase.py", line 912, in get_transceiver_info_dict
Jan  4 02:29:34.282424 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd     byte0 = self._read_eeprom_specific_bytes(sysfsfile_eeprom, 0, 1)[0]
Jan  4 02:29:34.282485 str-7260cx3-acs-2 INFO pmon#/supervisord: xcvrd TypeError: 'NoneType' object is not subscriptable
Jan  4 02:29:35.327279 str-7260cx3-acs-2 INFO pmon#/supervisor-proc-exit-listener: Process xcvrd exited unxepectedly. Terminating supervisor...

Steps to reproduce the issue:
1.
2.
3.

Describe the results you received:
pmon service is down because xcvrd failed to start.

Describe the results you expected:
All service should be healthy.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**
SONiC Software Version: SONiC.HEAD.203-ba92a081
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: ba92a081
Build date: Thu Dec 31 02:13:42 UTC 2020
Built by: johnar@jenkins-worker-22

Platform: x86_64-arista_7260cx3_64
HwSKU: Arista-7260CX3-D108C8
ASIC: broadcom
ASIC Count: 1
Serial Number: SSJ17432414
Uptime: 02:37:04 up 14:27,  2 users,  load average: 4.34, 3.13, 2.63

Docker images:
REPOSITORY                    TAG                 IMAGE ID            SIZE
docker-platform-monitor       HEAD.203-ba92a081   9122a354b3f7        588MB
docker-platform-monitor       latest              9122a354b3f7        588MB
docker-syncd-brcm             HEAD.203-ba92a081   5c2940ba09c6        637MB
docker-syncd-brcm             latest              5c2940ba09c6        637MB
docker-snmp                   HEAD.203-ba92a081   58ef68401d0a        431MB
docker-snmp                   latest              58ef68401d0a        431MB
docker-teamd                  HEAD.203-ba92a081   ff8c4d7914d5        449MB
docker-teamd                  latest              ff8c4d7914d5        449MB
docker-sonic-mgmt-framework   HEAD.203-ba92a081   3c7fae0581dd        604MB
docker-sonic-mgmt-framework   latest              3c7fae0581dd        604MB
docker-router-advertiser      HEAD.203-ba92a081   83e30d5fae91        390MB
docker-router-advertiser      latest              83e30d5fae91        390MB
docker-lldp                   HEAD.203-ba92a081   c5346ab419e0        430MB
docker-lldp                   latest              c5346ab419e0        430MB
docker-dhcp-relay             HEAD.203-ba92a081   8d526a0511ab        397MB
docker-dhcp-relay             latest              8d526a0511ab        397MB
docker-database               HEAD.203-ba92a081   e39c71a1a2ac        390MB
docker-database               latest              e39c71a1a2ac        390MB
docker-orchagent              HEAD.203-ba92a081   3a2f6b7c84c7        468MB
docker-orchagent              latest              3a2f6b7c84c7        468MB
docker-nat                    HEAD.203-ba92a081   2b8555260901        452MB
docker-nat                    latest              2b8555260901        452MB
docker-sonic-telemetry        HEAD.203-ba92a081   29211dac9791        464MB
docker-sonic-telemetry        latest              29211dac9791        464MB
docker-sflow                  HEAD.203-ba92a081   4ccfc8890955        450MB
docker-sflow                  latest              4ccfc8890955        450MB
docker-fpm-frr                HEAD.203-ba92a081   cf06956fb441        467MB
docker-fpm-frr                latest              cf06956fb441        467MB
**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
@theasianpianist
Copy link
Contributor

Also seen on 7050cx3

@jleveque
Copy link
Contributor

jleveque commented Jan 5, 2021

This issue will be fixed once sonic-net/sonic-platform-common#157 is merged and the submodule is updated.

@theasianpianist theasianpianist added this to Needs triage in Master Image Stabilization via automation Jan 5, 2021
@yxieca yxieca moved this from Needs triage to In progress in Master Image Stabilization Jan 6, 2021
Master Image Stabilization automation moved this from In progress to Done Jan 6, 2021
yxieca pushed a commit that referenced this issue Jan 6, 2021
)

src/sonic-platform-common 9935fca...8664efc (2):

Make sonic_sfp Python2 and Python3 compatible (#157)
[sffbase.py] Fix to make Python 3-compatible (#156)

src/sonic-platform-daemons e6c786b...81318f7 (1):

[psud] Fix issue where PSU Fan info is not updated in State DB (#137)

Fixes #6341
lguohan pushed a commit that referenced this issue Jan 9, 2021
)

src/sonic-platform-common 9935fca...8664efc (2):

Make sonic_sfp Python2 and Python3 compatible (#157)
[sffbase.py] Fix to make Python 3-compatible (#156)

src/sonic-platform-daemons e6c786b...81318f7 (1):

[psud] Fix issue where PSU Fan info is not updated in State DB (#137)

Fixes #6341
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment