[VOQ][saidump] Enhance saidump with new option -r to parser the JSON file and displays/format the right output #1288

JunhongMao · 2023-09-06T16:54:47Z

Why I did it

Fix issue: sonic-net/sonic-buildimage#13561
The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access.

This solution uses the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table.
Related PRs:
sonic-net/sonic-utilities#2972
sonic-net/sonic-buildimage#16466

MSFT ADO: 25892357

How I did it

To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it.

    (1) Updated platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2, install Python library rdbtools into the syncd containter.
    (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format.
    (3) Updated sonic-buildimage/build_debian.sh, to add a new script file: files/scripts/saidump.sh into the host. This shell file does the below steps:
      For each ASIC0, such as ASIC0,

      1. Save the Redis data.
      sudo sonic-db-cli -n asic$1 SAVE > /dev/null

      2. Move dump files to /var/run/redisX/
      docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/"

      3. Run rdb command to convert the dump files into JSON files
      docker exec syncd$1 sh -c "rdb --command json /var/run/redis$1/dump.rdb | tee /var/run/redis$1/dump.json > /dev/null"

      4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output.
      docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json -m 100"

      5. clear
      sudo rm -f /var/run/redis$1/dump.rdb
      sudo rm -f /var/run/redis$1/dump.json

    (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, to check the asic db size and if it is larger than xxx entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.

How to verify it

On T2 setup with more than 96K routes, execute CLI command -- generate_dump
No error should be shown
Download the generate_dump result and verify the saidump file after unpacking it.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

202205

• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, #1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null #2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" sonic-net#3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null sonic-net#4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" sonic-net#5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh

• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh

saidump/saidump.cpp

• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh

saidump/saidump.cpp

kcudnik · 2023-09-07T12:24:18Z

change all \r\n to std::endl

…rs. (#16466) Fix #13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

mlok-nokia · 2023-11-17T14:45:39Z

@judyjoseph Need to cherry-pick all related to Ps to 202205 branch

gechiang · 2023-11-18T00:23:56Z

@StormLiangMS , @yxieca , Please review/approve for 202205 and 202305.
MSFT ADO: 25892357

…rs. (sonic-net#16466) Fix sonic-net#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

…file and displays/format the right output (#1288) Why I did it Fix issue: sonic-net/sonic-buildimage#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-buildimage#16466

…rs. (sonic-net#16466) Fix sonic-net#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

…rs. (#16466) Fix #13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

sonic-net#2972 added two below functions into scripts/generate_dump. get_route_table_size_by_asic_id_and_ipver save_saidump_by_route_size The unittest scripts need to be added. Related PRs: sonic-net#2972 sonic-net/sonic-buildimage#16466 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 Microsoft ADO (25892277): Add two scripts: tests/saidump_test.py tests/saidump_test.sh To use below 6 test cases to verify the functionality of get_route_table_size_by_asic_id_and_ipver and save_saidump_by_route_size behave correctly. ``` saidump test list format: [ACIS number, ipv4 and ipv6 route table size, expected function save_cmd arguments] saidump_test_list = [ [1, 10000, "docker exec syncd saidump saidump"], [1, 12000, "docker exec syncd saidump saidump"], [1, 12001, "docker exec syncd saidump.sh saidump"], [1, 20000, "docker exec syncd saidump.sh saidump"], [2, 10000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12001, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"], [2, 20000, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"] ] ``` During the compiling stage, run the below command to check if it's PASSED. jumao@1b1ffba5949a:/sonic/src/sonic-utilities$ time python3 setup.py test tests/saidump_test.py::test_saidump PASSED

StormLiangMS · 2023-12-15T09:18:40Z

hi @JunhongMao @gechiang in 202305, we should use "#include
"
swss/json.hpp
"
" other than #include <nlohmann/json.hpp>,
otherwise it will cause compile issue like below,
make[3]: Entering directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump'
g++ -DHAVE_CONFIG_H -I. -I.. -Wdate-time -D_FORTIFY_SOURCE=2 -I/home/runner/work/sonic-swss/usr/include -g -I../SAI/inc -I../SAI/experimental -I../SAI/meta -I../lib -ansi -fPIC -std=c++14 -Wall -Wcast-align -Wcast-qual -Wconversion -Wdisabled-optimization -Werror -Wextra -Wfloat-equal -Wformat=2 -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wimport -Winit-self -Wno-inline -Winvalid-pch -Wmissing-field-initializers -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wno-aggregate-return -Wno-padded -Wno-switch-enum -Wno-unused-parameter -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstack-protector -Wstrict-aliasing=3 -Wswitch -Wswitch-default -Wunreachable-code -Wunused -Wvariadic-macros -Wwrite-strings -Wno-switch-default -Wconversion -Wno-psabi -Wcast-align=strict -Wno-error -c -o saidump-saidump.o test -f 'saidump.cpp' || echo './'saidump.cpp
saidump.cpp:17:10: fatal error: nlohmann/json.hpp: No such file or directory
17 | #include <nlohmann/json.hpp>
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [Makefile:477: saidump-saidump.o] Error 1
make[3]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump'
make[2]: *** [Makefile:444: all-recursive] Error 1
make[2]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis'
make[1]: *** [Makefile:376: all] Error 2
make[1]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis'
dh_auto_build: error: make -j1 returned exit code 2
make: *** [debian/rules:69: binary-syncd-vs] Error 25
Error: Process completed with exit code 2.

I will revert this PR from 202305, pls file separate PR to 202305 for this one.

…he JSON file and displays/format the right output (#1288)" This reverts commit 01f17ba.

gechiang · 2023-12-21T02:01:45Z

hi @JunhongMao @gechiang in 202305, we should use "#include " swss/json.hpp " " other than #include <nlohmann/json.hpp>, otherwise it will cause compile issue like below, make[3]: Entering directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump' g++ -DHAVE_CONFIG_H -I. -I.. -Wdate-time -D_FORTIFY_SOURCE=2 -I/home/runner/work/sonic-swss/usr/include -g -I../SAI/inc -I../SAI/experimental -I../SAI/meta -I../lib -ansi -fPIC -std=c++14 -Wall -Wcast-align -Wcast-qual -Wconversion -Wdisabled-optimization -Werror -Wextra -Wfloat-equal -Wformat=2 -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wimport -Winit-self -Wno-inline -Winvalid-pch -Wmissing-field-initializers -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wno-aggregate-return -Wno-padded -Wno-switch-enum -Wno-unused-parameter -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstack-protector -Wstrict-aliasing=3 -Wswitch -Wswitch-default -Wunreachable-code -Wunused -Wvariadic-macros -Wwrite-strings -Wno-switch-default -Wconversion -Wno-psabi -Wcast-align=strict -Wno-error -c -o saidump-saidump.o test -f 'saidump.cpp' || echo './'saidump.cpp saidump.cpp:17:10: fatal error: nlohmann/json.hpp: No such file or directory 17 | #include <nlohmann/json.hpp> | ^~~~~~~~~~~~~~~~~~~ compilation terminated. make[3]: *** [Makefile:477: saidump-saidump.o] Error 1 make[3]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump' make[2]: *** [Makefile:444: all-recursive] Error 1 make[2]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis' make[1]: *** [Makefile:376: all] Error 2 make[1]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis' dh_auto_build: error: make -j1 returned exit code 2 make: *** [debian/rules:69: binary-syncd-vs] Error 25 Error: Process completed with exit code 2.

I will revert this PR from 202305, pls file separate PR to 202305 for this one.

@JunhongMao , can you please address the commnets made by @StormLiangMS so we can make progreess with this PR which is blocking your other PR #1298 ...
Thanks!

sonic-net#1288 (comment)

JunhongMao · 2023-12-27T00:41:48Z

@gechiang, @StormLiangMS , I have created a new PR: #1335, please help to review. Thanks.

JunhongMao added 5 commits September 1, 2023 17:25

Merge branch 'master' of github.com:JunhongMao/sonic-sairedis

898421a

This was referenced Sep 6, 2023

[VOQ][saidump] Install rdbtools into the docker base related containers. sonic-net/sonic-buildimage#16466

Merged

[VOQ][saidump] Modify generate_dump: replace save_saidump with save_saidump_by_route_size sonic-net/sonic-utilities#2972

Merged

kcudnik requested changes Sep 6, 2023

View reviewed changes

JunhongMao added 2 commits September 6, 2023 16:54