[VOQ][saidump] Install rdbtools into the docker base related containers. #16466

JunhongMao · 2023-09-06T16:47:58Z

Why did I do it?

To fix the issue: #13561
The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access.

This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table.

Related PRs:
sonic-net/sonic-utilities#2972
#16466
sonic-net/sonic-sairedis#1288
sonic-net/sonic-sairedis#1298

Work item tracking

Microsoft ADO (number only):

How did I do it?

To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it.

1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers.

2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format.

3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps:

  For each ASIC, such as ASIC0,

  3.1. Config Redis consistency directory. 
  redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null

  3.2. Save the Redis data.
  redis-cli -h $hostname -p $port SAVE > /dev/null

  3.3. Run rdb command to convert the dump files into JSON files
    rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null

  3.4.  Run saidump -r to update the JSON files' format as same as the saidump before. 
       Then we can get the saidump's result in standard output."
       saidump -r $redis_dir/dump.json -m 100

  3.5. Clear the temporary files.
   rm -f $redis_dir/dump.rdb
   rm -f $redis_dir/dump.json

4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.

How to verify it

On T2 setup with more than 96K routes, execute CLI command -- generate_dump
No error should be shown
Download the generate_dump result and verify the saidump file after unpacking it.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

202205

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

• Saidump for DNX-SAI sonic-net#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh

lguohan · 2023-09-07T08:40:30Z

files/scripts/saidump.sh

@@ -0,0 +1,39 @@
+#!/bin/bash


please move this file to syncd/scripts

Hi Lguohan, there is an issue that's hard for putting saidump.sh in the syncd docker. Because the below command is not available inside the syncd docker due to the isolation between docker containers. I.e., the dump.rdb generated by the Redis save is invisible in the syncd container. So, one reasonable solution is to be leaving it in the host. The host has access to operate the dump.rdb.

debug "saidump.sh: [2] Move dump.rdb to /var/run/redis$1/ in container database$1." docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/"

@lguohan , please help to review it again. Thanks.

thanks for the explanation. I feel we should still move this saidump.sh to syncd docker, to address the problem you listed, it is better to mount a shared folder between database docker and syncd docker, so that you can see the rdb file in syncd docker once dumped.

Thanks for your comments! I will fix it.

i should have be more clear. when i say syncd docker, i really means sairedis repo, so that saidump.sh will be part of the sairedis package.

build_debian.sh

abdosi · 2023-09-11T19:30:28Z

@anamehra for viz. Please check

arlakshm · 2023-09-06T20:05:49Z

files/scripts/saidump.sh

+
+save_saidump_by_rdb() {
+  debug "saidump.sh: [1] sonic-db-cli -n asic$1 SAVE."
+  sudo sonic-db-cli -n asic$1 SAVE > /dev/null


this command sonic-db-cli -n asic$1 may not work for single asic linecard?

arlakshm · 2023-09-06T20:07:18Z

files/scripts/saidump.sh

+NUM_ASICS=`python -c 'from sonic_py_common.multi_asic import get_num_asics; print(get_num_asics())'`
+
+
+if (( $# == 0 &&  $NUM_ASICS == 1 )); then


there are sudo statements in save_saidump_by_rdb. When a user with only Read-only privileges executes show tech-support these will fail?

The command "show techsupport" calls "sudo generate_dump -v -t 5".
By checking the below related commands' mode bits, we can see they are all have "other user executable bit".
And I run the command "show techsupport" in a user with only Read-only privileges, then get correct saidump results.

$ ls `which generate_dump` -l -rwxr-xr-x 1 root root 66028 Sep 19 14:32 /usr/local/bin/generate_dump $ ls `which saidump.sh` -l -rwxr-xr-x 1 root root 1752 Sep 21 15:28 /usr/local/bin/saidump.sh $ ls `which rm` -l -rwxr-xr-x 1 root root 72704 Sep 24 2020 /usr/bin/rm

So, finally, remove all sudo for concise.

@arlakshm , please review, thanks.

approved by mistake

Install rdbtools into syncd docker and add saidump.sh into host. * [saidump] • Saidump for DNX-SAI sonic-net#13561 Solution and modification: To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2, install Python library rdbtools into the syncd containter. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Updated sonic-buildimage/build_debian.sh, to add a new script file: files/scripts/saidump.sh into the host. This shell file does the below steps: For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files docker exec syncd$1 sh -c "rdb --command json /var/run/redis$1/dump.rdb | tee /var/run/redis$1/dump.json > /dev/null" 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json -m 100" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, to check the asic db size and if it is larger than xxx entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.

platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2

…file and displays/format the right output (#1288) Why I did it Fix issue: sonic-net/sonic-buildimage#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-buildimage#16466

To add the rdbtools into base docker. Remove sudo in saidump.sh

To add the rdbtools into base docker. Move saidump.sh from host to syncd docker container.

lguohan · 2023-09-30T08:13:26Z

platform/broadcom/docker-syncd-brcm-dnx/saidump.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+set -e


can you move this file to sairedis repo?

Yes. But the sairedis is using C++. And there are no shell scripts in it. Based on concerns about the source code consistency and conciseness, I think I need to re-write saidump.sh from shell script to C++ language, and extra time for development and review is needed. At the same time, the alternative method is to just move saidump.sh from the main repo into repo sairedis, and no extra time is needed. What's your opinion? Thanks!

there are script in syncd https://github.com/sonic-net/sonic-sairedis/tree/master/syncd/scripts

Thanks. I will move saidump.sh into this folder.

@lguohan , @mlok-nokia , please review again. And a new PR was created as below.
sonic-net/sonic-sairedis#1298

@judyjoseph , are you available to process this PR and sonic-net/sonic-sairedis#1298? Thanks.

To add the rdbtools into base docker. Move saidump.sh from host to syncd docker container.

…file and displays/format the right output (#1288) Why I did it Fix issue: sonic-net/sonic-buildimage#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-buildimage#16466

…rs. (#16466) (#17222)

…rs. (sonic-net#16466) Fix sonic-net#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

mssonicbld · 2023-11-21T01:02:19Z

Cherry-pick PR to 202211: #17244

mssonicbld · 2023-11-21T01:02:20Z

Cherry-pick PR to 202205: #17245

…rs. (#16466) Fix #13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

sonic-net#2972 added two below functions into scripts/generate_dump. get_route_table_size_by_asic_id_and_ipver save_saidump_by_route_size The unittest scripts need to be added. Related PRs: sonic-net#2972 sonic-net/sonic-buildimage#16466 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 Microsoft ADO (25892277): Add two scripts: tests/saidump_test.py tests/saidump_test.sh To use below 6 test cases to verify the functionality of get_route_table_size_by_asic_id_and_ipver and save_saidump_by_route_size behave correctly. ``` saidump test list format: [ACIS number, ipv4 and ipv6 route table size, expected function save_cmd arguments] saidump_test_list = [ [1, 10000, "docker exec syncd saidump saidump"], [1, 12000, "docker exec syncd saidump saidump"], [1, 12001, "docker exec syncd saidump.sh saidump"], [1, 20000, "docker exec syncd saidump.sh saidump"], [2, 10000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12001, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"], [2, 20000, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"] ] ``` During the compiling stage, run the below command to check if it's PASSED. jumao@1b1ffba5949a:/sonic/src/sonic-utilities$ time python3 setup.py test tests/saidump_test.py::test_saidump PASSED

JunhongMao added 2 commits September 1, 2023 17:37

JunhongMao requested a review from lguohan as a code owner September 6, 2023 16:47

lguohan reviewed Sep 7, 2023

View reviewed changes

build_debian.sh Outdated Show resolved Hide resolved

JunhongMao force-pushed the master branch from 0fb27c2 to ef503b7 Compare September 19, 2023 03:59

JunhongMao changed the title ~~[VOQ][saidump] Install rdbtools into system and add saidump.sh~~ [VOQ][saidump] Install rdbtools into syncd docker container and add saidump.sh into host Sep 19, 2023

JunhongMao force-pushed the master branch from ef503b7 to 0fa7499 Compare September 19, 2023 20:12

arlakshm previously approved these changes Sep 20, 2023

View reviewed changes

JunhongMao force-pushed the master branch from 05fdc95 to 1a414d5 Compare September 21, 2023 15:53

lguohan reviewed Sep 23, 2023

View reviewed changes

platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2 Outdated Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master'

f2215ec

JunhongMao requested a review from qiluo-msft as a code owner September 27, 2023 15:51

sonic-net#16466 fixing based on review comments:

65fc011

To add the rdbtools into base docker. Remove sudo in saidump.sh

JunhongMao force-pushed the master branch from bd73b7b to 65fc011 Compare September 27, 2023 15:58

JunhongMao added a commit to JunhongMao/sonic-utilities that referenced this pull request Sep 30, 2023

sonic-net/sonic-buildimage#16466 fixing based on review comments:

cd08c2e

To add the rdbtools into base docker. Move saidump.sh from host to syncd docker container.

lguohan reviewed Sep 30, 2023

View reviewed changes

sonic-net#16466 fixing based on review comments:

aaac734

To add the rdbtools into base docker. Move saidump.sh from host to syncd docker container.

JunhongMao force-pushed the master branch from 033795d to aaac734 Compare September 30, 2023 13:29

mssonicbld added a commit that referenced this pull request Nov 19, 2023

[VOQ][saidump] Install rdbtools into the docker base related containe…

f4df761

…rs. (#16466) (#17222)

mssonicbld added Included in 202305 Branch and removed Created PR to 202305 Branch labels Nov 19, 2023

yxieca added Approved for 202205 Branch Request for 202211 Branch Approved for 202211 Branch labels Nov 21, 2023

mssonicbld mentioned this pull request Nov 21, 2023

[action] [PR:16466] [VOQ][saidump] Install rdbtools into the docker base related containers. #17244

Merged

10 tasks

mssonicbld mentioned this pull request Nov 21, 2023

[action] [PR:16466] [VOQ][saidump] Install rdbtools into the docker base related containers. #17245

Merged

10 tasks

mssonicbld added the Created PR to 202211 Branch label Nov 21, 2023

mssonicbld added the Created PR to 202205 Branch label Nov 21, 2023

mssonicbld added Included in 202205 Branch Included in 202211 Branch and removed Approved for 202205 Branch Created PR to 202205 Branch Approved for 202211 Branch Created PR to 202211 Branch labels Nov 21, 2023

judyjoseph mentioned this pull request Nov 21, 2023

Start to enable save_saidump in generate_dump for SpineRouter sonic-net/sonic-utilities#3050

Closed

JunhongMao mentioned this pull request Dec 12, 2023

[VOQ][saidump] Add saidump unit test scripts sonic-net/sonic-utilities#3079

Open

JunhongMao mentioned this pull request Dec 27, 2023

[VOQ][saidump] Enhance saidump with new option -r to parser the JSON file and displays/format the right output, fix compiling error sonic-net/sonic-sairedis#1335

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VOQ][saidump] Install rdbtools into the docker base related containers. #16466

[VOQ][saidump] Install rdbtools into the docker base related containers. #16466

JunhongMao commented Sep 6, 2023 •

edited

Loading

lguohan Sep 7, 2023

JunhongMao Sep 13, 2023 •

edited

Loading

JunhongMao Sep 19, 2023

lguohan Sep 28, 2023

JunhongMao Sep 28, 2023

lguohan Sep 30, 2023

abdosi commented Sep 11, 2023

arlakshm Sep 6, 2023

JunhongMao Sep 20, 2023 •

edited

Loading

arlakshm Sep 6, 2023

JunhongMao Sep 20, 2023 •

edited

Loading

JunhongMao Sep 21, 2023

lguohan Sep 30, 2023

JunhongMao Sep 30, 2023 •

edited

Loading

lguohan Oct 2, 2023

JunhongMao Oct 2, 2023

JunhongMao Oct 3, 2023

JunhongMao Nov 8, 2023

mssonicbld commented Nov 21, 2023

mssonicbld commented Nov 21, 2023

		NUM_ASICS=`python -c 'from sonic_py_common.multi_asic import get_num_asics; print(get_num_asics())'`


		if (( $# == 0 && $NUM_ASICS == 1 )); then

		@@ -0,0 +1,43 @@
		#!/bin/bash
		set -e

[VOQ][saidump] Install rdbtools into the docker base related containers. #16466

[VOQ][saidump] Install rdbtools into the docker base related containers. #16466

Conversation

JunhongMao commented Sep 6, 2023 • edited Loading

Why did I do it?

Work item tracking

How did I do it?

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Choose a reason for hiding this comment

JunhongMao Sep 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abdosi commented Sep 11, 2023

Choose a reason for hiding this comment

JunhongMao Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JunhongMao Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JunhongMao Sep 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mssonicbld commented Nov 21, 2023

mssonicbld commented Nov 21, 2023

JunhongMao commented Sep 6, 2023 •

edited

Loading

JunhongMao Sep 13, 2023 •

edited

Loading

JunhongMao Sep 20, 2023 •

edited

Loading

JunhongMao Sep 20, 2023 •

edited

Loading

JunhongMao Sep 30, 2023 •

edited

Loading