Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[multi-asic] Fix for sonic-cfggen exception during platform string read #8229

Merged
merged 2 commits into from Aug 18, 2021

Conversation

minionatwork
Copy link
Contributor

during fresh install and start of sonic in multi asic, /var/run/redisX/ is created after database docker is
started. Currently sonic-cfggen is throwing exception while reading platform string from /usr/bin/database.sh

sonic python3[2259]: :- parseDatabaseConfig: Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
sonic python3[2259]: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
sonic database.sh[2259]: Traceback (most recent call last):
sonic database.sh[2259]: File "/usr/local/bin/sonic-cfggen", line 452, in
sonic database.sh[2259]: main()
sonic database.sh[2259]: File "/usr/local/bin/sonic-cfggen", line 310, in main
sonic database.sh[2259]: SonicDBConfig.load_sonic_global_db_config(namespace=args.namespace)
sonic database.sh[2259]: File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1342, in load_sonic_global_db_config
sonic database.sh[2259]: SonicDBConfig.initializeGlobalConfig(global_db_file_path)
sonic database.sh[2259]: RuntimeError: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
sonic database.sh[2246]: Creating new database0 container
sonic database.sh[2516]: 4be09b19659f7db6f8399455f32b9f7c7a47ee66f67c4d0dddf1c54fb0f9d59c
sonic database.sh[3296]: database0

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

during fresh install and start of sonic in multi asic, /var/run/redisX/ is created after database docker is
started. Currently sonic-cfggen is throwing exception while reading platform string from /usr/bin/database.sh

 sonic python3[2259]: :- parseDatabaseConfig: Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
 sonic python3[2259]: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
 sonic database.sh[2259]: Traceback (most recent call last):
 sonic database.sh[2259]:   File "/usr/local/bin/sonic-cfggen", line 452, in <module>
 sonic database.sh[2259]:     main()
 sonic database.sh[2259]:   File "/usr/local/bin/sonic-cfggen", line 310, in main
 sonic database.sh[2259]:     SonicDBConfig.load_sonic_global_db_config(namespace=args.namespace)
 sonic database.sh[2259]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1342, in load_sonic_global_db_config
 sonic database.sh[2259]:     SonicDBConfig.initializeGlobalConfig(global_db_file_path)
 sonic database.sh[2259]: RuntimeError: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
 sonic database.sh[2246]: Creating new database0 container
 sonic database.sh[2516]: 4be09b19659f7db6f8399455f32b9f7c7a47ee66f67c4d0dddf1c54fb0f9d59c
 sonic database.sh[3296]: database0
during fresh install and start of sonic in multi asic, /var/run/redisX/ is created after database docker is
started. Currently sonic-cfggen is throwing exception while reading platform string from /usr/bin/database.sh

 sonic python3[2259]: :- parseDatabaseConfig: Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
 sonic python3[2259]: :- initializeGlobalConfig: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
 sonic database.sh[2259]: Traceback (most recent call last):
 sonic database.sh[2259]:   File "/usr/local/bin/sonic-cfggen", line 452, in <module>
 sonic database.sh[2259]:     main()
 sonic database.sh[2259]:   File "/usr/local/bin/sonic-cfggen", line 310, in main
 sonic database.sh[2259]:     SonicDBConfig.load_sonic_global_db_config(namespace=args.namespace)
 sonic database.sh[2259]:   File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1342, in load_sonic_global_db_config
 sonic database.sh[2259]:     SonicDBConfig.initializeGlobalConfig(global_db_file_path)
 sonic database.sh[2259]: RuntimeError: Sonic database config file syntax error >> Sonic database config file doesn't exist at /var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json
 sonic database.sh[2246]: Creating new database0 container
 sonic database.sh[2516]: 4be09b19659f7db6f8399455f32b9f7c7a47ee66f67c4d0dddf1c54fb0f9d59c
 sonic database.sh[3296]: database0
@arlakshm
Copy link
Contributor

@minionatwork. We are not hitting this issue when we bring-up a multi asic VS. Is there specific sequence of steps done to repro this issue ?

adding @SuvarnaMeenakshi for viz.

@minionatwork
Copy link
Contributor Author

It was straightforward happening after first install and boot of sonic.

This file is created inside docker (database docker) but to start database docker, this file is was checked and exception happens. So, I dont know how multi asic VS is working. Discussed with Judy offline. He might have more picture on this how multi asic VS is working.

/var/run/redis/sonic-db/../../redis0/sonic-db/database_config.json

@judyjoseph
Copy link
Contributor

judyjoseph commented Aug 3, 2021

@minionatwork. We are not hitting this issue when we bring-up a multi asic VS. Is there specific sequence of steps done to repro this issue ?

adding @SuvarnaMeenakshi for viz.

Thanks to @suvarna for the multi-vs image, did a check on this error, so I see it happens there too on the first boot directly as a multi-asic VS ( earlier we didn't observe as we were booting in single ASIC mode and then converting to multi-asic VS).

This is happening as we started to use swsscommon dbconnector classes and there the checks are strict and we throw runtime_error. Ref: https://github.com/Azure/sonic-swss-common/blob/f89b2ac1049c61f77b14fefc2fb0f252e1bd7aa7/common/dbconnector.cpp#L66

We will take in this fix --as this will make sure that in sonic-cfggen we load_global_config only when we pass a namespace parameter.

Copy link
Contributor

@judyjoseph judyjoseph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks ok in multi-asic device too where global database config json is used.

@judyjoseph judyjoseph changed the title Fix for sonic-cfggen exception during platform string read [multi-asic] Fix for sonic-cfggen exception during platform string read Aug 18, 2021
@judyjoseph judyjoseph merged commit 3f095f1 into sonic-net:master Aug 18, 2021
judyjoseph pushed a commit that referenced this pull request Aug 25, 2021
…ad (#8229)

Fix for sonic-cfggen exception during platform string read during fresh install and start of sonic in multi asic, /var/run/redisX/ is created after database docker is started.
@anamehra
Copy link
Contributor

Hi Judy (@judyjoseph ), I am seeing this issue on a multi asic system.
This is happening for database container itself:
Jul 21 19:42:56 sonic database.sh[58861]: + DOCKERMOUNT=
Jul 21 19:42:56 sonic database.sh[58861]: + '[' x == x ']'
Jul 21 19:42:56 sonic database.sh[58861]: + preStartAction
Jul 21 19:42:56 sonic database.sh[58861]: + WARM_DIR=/host/warmboot
Jul 21 19:42:56 sonic database.sh[58861]: + '[' '' '!=' chassisdb ']'
Jul 21 19:42:56 sonic database.sh[58861]: + [[ cold == \w\a\r\m ]]
Jul 21 19:42:56 sonic database.sh[58861]: + [[ cold == \f\a\s\t\f\a\s\t ]]
Jul 21 19:42:56 sonic database.sh[58861]: + echo -n
Jul 21 19:42:56 sonic database.sh[58861]: + docker cp /tmp/dump.rdb database:/var/lib/redis/
Jul 21 19:42:57 sonic database.sh[58861]: + updateSyslogConf
Jul 21 19:42:57 sonic database.sh[58861]: + [[ 2 -gt 1 ]]
Jul 21 19:42:57 sonic database.sh[58905]: ++ docker network inspect bridge '--format={{(index .IPAM.Config 0).Gateway}}'
Jul 21 19:42:57 sonic database.sh[58861]: + TARGET_IP=240.127.1.1
Jul 21 19:42:57 sonic database.sh[58861]: + CONTAINER_NAME=database
Jul 21 19:42:57 sonic database.sh[58861]: + TMP_FILE=/tmp/rsyslog.database.conf
Jul 21 19:42:57 sonic database.sh[58861]: + sonic-cfggen -t /usr/share/sonic/templates/rsyslog-container.conf.j2 -a '{"target_ip": "240.127.1.1", "container_name": "database" }'
Jul 21 19:42:57 sonic python3[58918]: :- parseDatabaseConfig: Sonic database config file doesn't exist at /var/run/redis/sonic-db/database_config.json
Jul 21 19:42:57 sonic database.sh[58918]: Traceback (most recent call last):
Jul 21 19:42:57 sonic database.sh[58918]: File "/usr/local/bin/sonic-cfggen", line 441, in
Jul 21 19:42:57 sonic database.sh[58918]: main()
Jul 21 19:42:57 sonic database.sh[58918]: File "/usr/local/bin/sonic-cfggen", line 402, in main
Jul 21 19:42:57 sonic database.sh[58918]: env = _get_jinja2_env(paths)
Jul 21 19:42:57 sonic database.sh[58918]: File "/usr/local/bin/sonic-cfggen", line 237, in _get_jinja2_env
Jul 21 19:42:57 sonic database.sh[58918]: redis_bcc = RedisBytecodeCache(SonicV2Connector(host='127.0.0.1'))
Jul 21 19:42:57 sonic database.sh[58918]: File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1704, in init
Jul 21 19:42:57 sonic database.sh[58918]: for db_name in self.get_db_list():
Jul 21 19:42:57 sonic database.sh[58918]: File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1640, in get_db_list
Jul 21 19:42:57 sonic database.sh[58918]: return _swsscommon.SonicV2Connector_Native_get_db_list(self)
Jul 21 19:42:57 sonic database.sh[58918]: RuntimeError: Sonic database config file doesn't exist at /var/run/redis/sonic-db/database_config.json
Jul 21 19:42:57 sonic database.sh[58861]: + docker cp /tmp/rsyslog.database.conf database:/etc/rsyslog.conf
Jul 21 19:42:58 sonic database.sh[58861]: + rm -rf /tmp/rsyslog.database.conf
Jul 21 19:42:58 sonic database.sh[58861]: + echo 'Starting existing database container'
Jul 21 19:42:58 sonic database.sh[58861]: Starting existing database container

My worspace is based of following commit id:
commit 63ba489
Date: Wed Sep 8 19:59:50 2021 +0300

@@ -435,6 +432,7 @@ def main():
if args.namespace is None:
configdb = ConfigDBPipeConnector(use_unix_socket_path=True, **db_kwargs)
else:
SonicDBConfig.load_sonic_global_db_config(namespace=args.namespace)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SonicDBConfig.load_sonic_global_db_config(namespace=args.namespace)

An extreme case:
if both args.from_db and args.write_to_db have valid stirng, will the second load_sonic_global_db_config throw exception?

qiluo-msft pushed a commit that referenced this pull request Sep 29, 2021
…ad (#8229)

Fix for sonic-cfggen exception during platform string read during fresh install and start of sonic in multi asic, /var/run/redisX/ is created after database docker is started.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants