Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[master][chassis][mutil-asic] db_migrate.py show error and back trace while loading configuration on Linecard #18389

Closed
mlok-nokia opened this issue Mar 19, 2024 · 5 comments
Closed

[master][chassis][mutil-asic] db_migrate.py show error and back trace while loading configuration on Linecard #18389

mlok-nokia opened this issue Mar 19, 2024 · 5 comments
Assignees
@qiluo-msft @liuh-80
Labels
MSFT Triaged this issue has been triaged

Comments

@mlok-nokia
Copy link
Contributor

mlok-nokia commented Mar 19, 2024 •
edited
Loading

Description

On master branch, during multi-asic platfom linecard is booting up, db_migrate,py show the following back trace. This could be caused by PR sonic-net/sonic-utilities#3100. database@#.Service dependency "Requires=database.service
After=database.service". While db_migrate.py is run on the local database, ASIC instance database0 and database1 have not created the /var/run/redis0/sonic-db/database-config.json yet. It looks like PR sonic-net/sonic-utilities#3100 use "load_db_config()" and cause this backtarce.

Mar 19 04:20:01.443336 ixre-egl-board40 ERR db_migrator: :- parseDatabaseConfig: Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.443474 ixre-egl-board40 ERR db_migrator: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.443585 ixre-egl-board40 ERR db_migrator: Caught exception: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.449596 ixre-egl-board40 INFO database.sh[1907]: Traceback (most recent call last):
Mar 19 04:20:01.449806 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/local/bin/db_migrator.py", line 1280, in main
Mar 19 04:20:01.449855 ixre-egl-board40 INFO database.sh[1907]:     load_db_config()
Mar 19 04:20:01.449900 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/local/lib/python3.11/dist-packages/utilities_common/general.py", line 30, in load_db_config
Mar 19 04:20:01.449950 ixre-egl-board40 INFO database.sh[1907]:     swsscommon.SonicDBConfig.load_sonic_global_db_config()
Mar 19 04:20:01.450010 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1661, in load_sonic_global_db_config
Mar 19 04:20:01.450056 ixre-egl-board40 INFO database.sh[1907]:     SonicDBConfig.initializeGlobalConfig(global_db_file_path)
Mar 19 04:20:01.450097 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1656, in initializeGlobalConfig
Mar 19 04:20:01.450136 ixre-egl-board40 INFO database.sh[1907]:     return _swsscommon.SonicDBConfig_initializeGlobalConfig(*args, **kwargs)
Mar 19 04:20:01.450182 ixre-egl-board40 INFO database.sh[1907]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 19 04:20:01.450226 ixre-egl-board40 INFO database.sh[1907]: RuntimeError: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.451439 ixre-egl-board40 INFO database.sh[1907]: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.451596 ixre-egl-board40 INFO database.sh[1907]: usage: db_migrator.py [-h] [-o operation migrate, set_version, get_version]
Mar 19 04:20:01.452231 ixre-egl-board40 INFO database.sh[1907]:                       [-s unix socket] [-n asic namespace]
Mar 19 04:20:01.452775 ixre-egl-board40 INFO database.sh[1907]: options:
Mar 19 04:20:01.453413 ixre-egl-board40 INFO database.sh[1907]:   -h, --help            show this help message and exit
Mar 19 04:20:01.453724 ixre-egl-board40 INFO database.sh[1907]:   -o operation (migrate, set_version, get_version)
Mar 19 04:20:01.453773 ixre-egl-board40 INFO database.sh[1907]:                         operation to perform [default: get_version]
Mar 19 04:20:01.453821 ixre-egl-board40 INFO database.sh[1907]:   -s unix socket        the unix socket that the desired database listens on
Mar 19 04:20:01.453864 ixre-egl-board40 INFO database.sh[1907]:   -n asic namespace     The asic namespace whose DB instance we need to
Mar 19 04:20:01.453903 ixre-egl-board40 INFO database.sh[1907]:                         connect
Mar 19 04:20:01.530003 ixre-egl-board40 INFO database.sh[1911]: True

Steps to reproduce the issue:

  1. On a chassis, reboot multi-asic linecard.
  2. the following error be shown in syslog
Mar 19 04:20:01.443336 ixre-egl-board40 ERR db_migrator: :- parseDatabaseConfig: Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.443474 ixre-egl-board40 ERR db_migrator: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.443585 ixre-egl-board40 ERR db_migrator: Caught exception: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.449596 ixre-egl-board40 INFO database.sh[1907]: Traceback (most recent call last):
Mar 19 04:20:01.449806 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/local/bin/db_migrator.py", line 1280, in main
Mar 19 04:20:01.449855 ixre-egl-board40 INFO database.sh[1907]:     load_db_config()
Mar 19 04:20:01.449900 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/local/lib/python3.11/dist-packages/utilities_common/general.py", line 30, in load_db_config
Mar 19 04:20:01.449950 ixre-egl-board40 INFO database.sh[1907]:     swsscommon.SonicDBConfig.load_sonic_global_db_config()
Mar 19 04:20:01.450010 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1661, in load_sonic_global_db_config
Mar 19 04:20:01.450056 ixre-egl-board40 INFO database.sh[1907]:     SonicDBConfig.initializeGlobalConfig(global_db_file_path)
Mar 19 04:20:01.450097 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1656, in initializeGlobalConfig
Mar 19 04:20:01.450136 ixre-egl-board40 INFO database.sh[1907]:     return _swsscommon.SonicDBConfig_initializeGlobalConfig(*args, **kwargs)
Mar 19 04:20:01.450182 ixre-egl-board40 INFO database.sh[1907]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 19 04:20:01.450226 ixre-egl-board40 INFO database.sh[1907]: RuntimeError: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.451439 ixre-egl-board40 INFO database.sh[1907]: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.451596 ixre-egl-board40 INFO database.sh[1907]: usage: db_migrator.py [-h] [-o operation migrate, set_version, get_version]
Mar 19 04:20:01.452231 ixre-egl-board40 INFO database.sh[1907]:                       [-s unix socket] [-n asic namespace]
Mar 19 04:20:01.452775 ixre-egl-board40 INFO database.sh[1907]: options:
Mar 19 04:20:01.453413 ixre-egl-board40 INFO database.sh[1907]:   -h, --help            show this help message and exit
Mar 19 04:20:01.453724 ixre-egl-board40 INFO database.sh[1907]:   -o operation (migrate, set_version, get_version)
Mar 19 04:20:01.453773 ixre-egl-board40 INFO database.sh[1907]:                         operation to perform [default: get_version]
Mar 19 04:20:01.453821 ixre-egl-board40 INFO database.sh[1907]:   -s unix socket        the unix socket that the desired database listens on
Mar 19 04:20:01.453864 ixre-egl-board40 INFO database.sh[1907]:   -n asic namespace     The asic namespace whose DB instance we need to
Mar 19 04:20:01.453903 ixre-egl-board40 INFO database.sh[1907]:                         connect
Mar 19 04:20:01.530003 ixre-egl-board40 INFO database.sh[1911]: True

Describe the results you received:

Error show on syslog

Mar 19 04:20:01.443336 ixre-egl-board40 ERR db_migrator: :- parseDatabaseConfig: Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.443474 ixre-egl-board40 ERR db_migrator: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.443585 ixre-egl-board40 ERR db_migrator: Caught exception: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.449596 ixre-egl-board40 INFO database.sh[1907]: Traceback (most recent call last):
Mar 19 04:20:01.449806 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/local/bin/db_migrator.py", line 1280, in main
Mar 19 04:20:01.449855 ixre-egl-board40 INFO database.sh[1907]:     load_db_config()
Mar 19 04:20:01.449900 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/local/lib/python3.11/dist-packages/utilities_common/general.py", line 30, in load_db_config
Mar 19 04:20:01.449950 ixre-egl-board40 INFO database.sh[1907]:     swsscommon.SonicDBConfig.load_sonic_global_db_config()
Mar 19 04:20:01.450010 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1661, in load_sonic_global_db_config
Mar 19 04:20:01.450056 ixre-egl-board40 INFO database.sh[1907]:     SonicDBConfig.initializeGlobalConfig(global_db_file_path)
Mar 19 04:20:01.450097 ixre-egl-board40 INFO database.sh[1907]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1656, in initializeGlobalConfig
Mar 19 04:20:01.450136 ixre-egl-board40 INFO database.sh[1907]:     return _swsscommon.SonicDBConfig_initializeGlobalConfig(*args, **kwargs)
Mar 19 04:20:01.450182 ixre-egl-board40 INFO database.sh[1907]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 19 04:20:01.450226 ixre-egl-board40 INFO database.sh[1907]: RuntimeError: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.451439 ixre-egl-board40 INFO database.sh[1907]: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
Mar 19 04:20:01.451596 ixre-egl-board40 INFO database.sh[1907]: usage: db_migrator.py [-h] [-o operation migrate, set_version, get_version]
Mar 19 04:20:01.452231 ixre-egl-board40 INFO database.sh[1907]:                       [-s unix socket] [-n asic namespace]
Mar 19 04:20:01.452775 ixre-egl-board40 INFO database.sh[1907]: options:
Mar 19 04:20:01.453413 ixre-egl-board40 INFO database.sh[1907]:   -h, --help            show this help message and exit
Mar 19 04:20:01.453724 ixre-egl-board40 INFO database.sh[1907]:   -o operation (migrate, set_version, get_version)
Mar 19 04:20:01.453773 ixre-egl-board40 INFO database.sh[1907]:                         operation to perform [default: get_version]
Mar 19 04:20:01.453821 ixre-egl-board40 INFO database.sh[1907]:   -s unix socket        the unix socket that the desired database listens on
Mar 19 04:20:01.453864 ixre-egl-board40 INFO database.sh[1907]:   -n asic namespace     The asic namespace whose DB instance we need to
Mar 19 04:20:01.453903 ixre-egl-board40 INFO database.sh[1907]:                         connect
Mar 19 04:20:01.530003 ixre-egl-board40 INFO database.sh[1911]: True

Describe the results you expected:

Output of show version:

Master branch

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):
@prgeor prgeor added MSFT Triaged this issue has been triaged labels Mar 27, 2024
@liuh-80
Copy link
Contributor

liuh-80 commented Mar 29, 2024

Compare of current code and original code:

Current code:
if args.namespace is not None:
SonicDBConfig.initializeGlobalConfig()
else:
SonicDBConfig.initialize()

Original code:
if is_multi_asic():
if not swsscommon.SonicDBConfig.isGlobalInit():
SonicDBConfig.initializeGlobalConfig() <== crash here
else:
if not swsscommon.SonicDBConfig.isInit():
SonicDBConfig.initialize()

So, what happen is, on multiasic device, during bootup, the namespace parameter is None, then in old code db_migrator will initialize with local config. in new code, db_migrator will initialize with global config.

Will verify fix on hardware.

@liuh-80
Copy link
Contributor

liuh-80 commented Apr 3, 2024 •
edited
Loading

Created fix PR: sonic-net/sonic-utilities#3257

@liuh-80 liuh-80 mentioned this issue Apr 7, 2024
Merged
@liuh-80
Copy link
Contributor

liuh-80 commented Apr 7, 2024 •
edited
Loading

@mlok-nokia , according to Judy's comments in this PR: sonic-net/sonic-utilities#3257

sonic-net/sonic-utilities#3257 (comment):
I feel it is more like a timing issue as you pointed out, Can we call db_migrator, after /var/run/redis0/sonic-db/database-config.json is created ?

Can you check if the issue can fix by call this script after config file generated?

@mlok-nokia
Copy link
Contributor Author

mlok-nokia commented Apr 16, 2024 •
edited
Loading

@mlok-nokia , according to Judy's comments in this PR: sonic-net/sonic-utilities#3257

sonic-net/sonic-utilities#3257 (comment): I feel it is more like a timing issue as you pointed out, Can we call db_migrator, after /var/run/redis0/sonic-db/database-config.json is created ?

Can you check if the issue can fix by call this script after config file generated?

@liuh-80 @judyjoseph all instance database@.service have dependency "Requires=database.service
After=database.service". db_mihgrator script in local host database will not be able to wait for all other instance ready.

There are two options for multi-asic platform:

  1. Change the databse@.service dependency. Since the migrate in database.sh just does the migrate for localhost database. Need modify to migrate for all ASIC instance databases.
    or
  2. Modify the config-setup script to do the migrate for reboot case since config-setup.service is after all database@.service. This is very straight forward. Also, the existing database.sh just does the migrate for localhost database, not for ASIC instance databases.

liuh-80 added a commit to sonic-net/sonic-utilities that referenced this issue Apr 25, 2024
@liuh-80
Fix db_migrate.py show error and back trace while loading configurati…
cd5c058 
…on on Linecard (#3257)

Fix db_migrate.py show error and back trace while loading configuration on Linecard

#### Why I did it
    Fix [issue @](sonic-net/sonic-buildimage#18389)

#### How I did it
    Revert code change by #3100
    Check DB config initialize state and ignore when initialized.

#### How to verify it
    Pass all UT.
    Manually test.

##### Work item tracking
- Microsoft ADO **(number only)**: 27384235

#### Which release branch to backport (provide reason below if selected)
    N/A

#### Description for the changelog
    Fix db_migrate.py show error and back trace while loading configuration on Linecard

#### A picture of a cute animal (not mandatory but encouraged)
@mlok-nokia
Copy link
Contributor Author

Fixed

arfeigin pushed a commit to arfeigin/sonic-utilities that referenced this issue Jun 16, 2024
@liuh-80 @arfeigin
Fix db_migrate.py show error and back trace while loading configurati…
dccf863 
…on on Linecard (sonic-net#3257)

Fix db_migrate.py show error and back trace while loading configuration on Linecard

#### Why I did it
    Fix [issue @](sonic-net/sonic-buildimage#18389)

#### How I did it
    Revert code change by sonic-net#3100
    Check DB config initialize state and ignore when initialized.

#### How to verify it
    Pass all UT.
    Manually test.

##### Work item tracking
- Microsoft ADO **(number only)**: 27384235

#### Which release branch to backport (provide reason below if selected)
    N/A

#### Description for the changelog
    Fix db_migrate.py show error and back trace while loading configuration on Linecard

#### A picture of a cute animal (not mandatory but encouraged)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@qiluo-msft qiluo-msft

@liuh-80 liuh-80

Labels
MSFT Triaged this issue has been triaged
Projects
SONiC Chassis
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@qiluo-msft @prgeor @liuh-80 @mlok-nokia