[VOQ][saidump] Modify generate_dump: replace save_saidump with save_saidump_by_route_size #2972

JunhongMao · 2023-09-06T16:27:10Z

Why did I do it?

To fix the issue: sonic-net/sonic-buildimage#13561
The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access.

This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table.

Related PRs:
#2972
sonic-net/sonic-buildimage#16466
sonic-net/sonic-sairedis#1288
sonic-net/sonic-sairedis#1298

Work item tracking

Microsoft ADO (25892277):

How did I do it?

To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it.

1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers.

2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format.

3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps:

  For each ASIC, such as ASIC0,

  3.1. Config Redis consistency directory. 
  redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null

  3.2. Save the Redis data.
  redis-cli -h $hostname -p $port SAVE > /dev/null

  3.3. Run rdb command to convert the dump files into JSON files
    rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null

  3.4.  Run saidump -r to update the JSON files' format as same as the saidump before. 
       Then we can get the saidump's result in standard output."
       saidump -r $redis_dir/dump.json -m 100

  3.5. Clear the temporary files.
   rm -f $redis_dir/dump.rdb
   rm -f $redis_dir/dump.json

4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.

How to verify it

On T2 setup with more than 96K routes, execute CLI command -- generate_dump
No error should be shown
Download the generate_dump result and verify the saidump file after unpacking it.
The test results
Execute show techsupport ccomand with more than 24k routes we see the system cpu higher than 3.8%. The following result

Oct 25 20:14:01.188269 ixre-egl-board5 NOTICE syncd0#root: saidump.sh: [4] Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output.
Oct 25 20:14:02.686092 ixre-egl-board5 NOTICE syncd0#root: saidump.sh: [5] Clear the temporary files.
Oct 25 20:14:03.278677 ixre-egl-board5 NOTICE syncd1#root: saidump.sh: hostname:240.127.1.2, port:6379, redis_dir:/var/run/redis1
Oct 25 20:14:03.279898 ixre-egl-board5 NOTICE syncd1#root: saidump.sh: [1] Config Redis consistency directory.
Oct 25 20:14:03.286410 ixre-egl-board5 NOTICE syncd1#root: saidump.sh: [2] SAVE.
Oct 25 20:14:03.459810 ixre-egl-board5 NOTICE syncd1#root: saidump.sh: [3] Run rdb command to convert the dump files into JSON files.
Oct 25 20:14:07.546016 ixre-egl-board5 NOTICE syncd1#root: saidump.sh: [4] Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output.
Oct 25 20:14:09.028892 ixre-egl-board5 NOTICE syncd1#root: saidump.sh: [5] Clear the temporary files.


==================================================================
top - 20:14:01 up  3:05,  3 users,  load average: 2.79, 2.35, 2.13
Tasks: 488 total,   3 running, 481 sleeping,   0 stopped,   4 zombie
%Cpu(s): 18.8 us,  1.8 sy,  0.0 ni, 79.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31978.1 total,  21475.0 free,   7383.4 used,   3119.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  24088.1 avail Mem 

--
 536840 root      20   0    2480    576    508 S   0.0   0.0   0:00.00 sh
 536842 root      20   0    2480    568    504 S   0.0   0.0   0:00.00 sh
 536843 root      20   0   11144   5444   4880 S   0.0   0.0   0:00.00 bcmcmd
Wed 25 Oct 2023 08:14:01 PM UTC
==================================================================
top - 20:14:04 up  3:06,  3 users,  load average: 2.80, 2.36, 2.14
Tasks: 484 total,   2 running, 478 sleeping,   0 stopped,   4 zombie
%Cpu(s): 12.4 us,  0.4 sy,  0.0 ni, 87.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31978.1 total,  21549.6 free,   7319.1 used,   3109.4 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  24169.8 avail Mem 

--
 537057 root      20   0   62328  43688  14900 S   0.0   0.1   0:00.32 show
 537110 root      20   0    2480    512    444 S   0.0   0.0   0:00.00 sh
 537111 root      20   0   11048   4076   3128 R   0.0   0.0   0:00.01 top
Wed 25 Oct 2023 08:14:04 PM UTC
==================================================================
top - 20:14:06 up  3:06,  3 users,  load average: 2.80, 2.36, 2.14
Tasks: 484 total,   2 running, 478 sleeping,   0 stopped,   4 zombie
%Cpu(s):  8.1 us,  0.7 sy,  0.0 ni, 91.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31978.1 total,  21535.2 free,   7318.7 used,   3124.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  24155.4 avail Mem 

--
 537033 root      20   0    3896   2992   2684 S   0.0   0.0   0:00.00 saidump.sh
 537130 root      20   0   62332  43828  15040 S   0.0   0.1   0:00.30 show
 537178 root      20   0    2480    572    504 S   0.0   0.0   0:00.00 sh
Wed 25 Oct 2023 08:14:06 PM UTC
==================================================================
top - 20:14:09 up  3:06,  3 users,  load average: 2.74, 2.35, 2.14
Tasks: 479 total,   2 running, 473 sleeping,   0 stopped,   4 zombie
%Cpu(s):  3.4 us,  3.0 sy,  3.0 ni, 90.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31978.1 total,  21555.6 free,   7323.9 used,   3098.7 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  24182.9 avail Mem 

--
 537211 root      25   5   12652   3756   1596 S   0.0   0.0   0:00.00 generate_dump
 537213 root      25   5   16740   9284   5392 S   0.0   0.0   0:00.03 python
 537260 root      20   0    2480    568    504 S   0.0   0.0   0:00.00 sh
Wed 25 Oct 2023 08:14:09 PM UTC
==================================================================
top - 20:14:12 up  3:06,  3 users,  load average: 2.74, 2.35, 2.14
Tasks: 482 total,   3 running, 475 sleeping,   0 stopped,   4 zombie
%Cpu(s):  8.2 us,  3.4 sy,  4.1 ni, 84.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  31978.1 total,  21569.8 free,   7309.5 used,   3098.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  24197.5 avail Mem 

--
 537776 root      25   5   11144   5432   4868 S   0.0   0.0   0:00.00 bcmcmd
 537778 root      20   0    2480    568    504 S   0.0   0.0   0:00.00 sh
 537779 root      20   0   11144   5408   4844 S   0.0   0.0   0:00.00 bcmcmd
Wed 25 Oct 2023 08:14:12 PM UTC
==================================================================


dmin@ixre-egl-board5:~$ show ip route summ
asic0:
Route Source         Routes               FIB  (vrf default)
kernel               42                   42                   
connected            17                   17                   
ebgp                 109                  109                  
ibgp                 22785                22785                
------
Totals               22953                22953                


asic1:
Route Source         Routes               FIB  (vrf default)
kernel               42                   42                   
connected            17                   17                   
ebgp                 10075                10075                
ibgp                 12819                12819                
------
Totals               22953                22953                


admin@ixre-egl-board5:~$ show ipv6 route summ
asic0:
Route Source         Routes               FIB  (vrf default)
kernel               43                   43                   
connected            46                   46                   
static               1                    1                    
ebgp                 109                  109                  
ibgp                 14786                14786                
------
Totals               14985                14985                


asic1:
Route Source         Routes               FIB  (vrf default)
kernel               43                   43                   
connected            46                   46                   
static               1                    1                    
ebgp                 2075                 2075                 
ibgp                 12820                12820                
------
Totals               14985                14985

Also for syncd crash we do not see any swss timeouts or high cpu, also the syncd is able to recover without affecting other syncd.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

output data of the new saidump is identical to the previous.

• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, sonic-net#1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null sonic-net#2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" sonic-net#3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null sonic-net#4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" sonic-net#5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh

• Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh

linux-foundation-easycla · 2023-09-06T16:27:14Z

The committers listed above are authorized under a signed CLA.

✅ login: JunhongMao (2b43852, 62aab2d, 5ecb3c1, 3504fdc, 2450c48, 9fa769e, 8a3f93b, 32f5607, 64ad586, 867a6d1, 185bfef, 86d3efa)

lguohan · 2023-09-07T08:36:49Z

scripts/generate_dump

@@ -1791,6 +1812,8 @@ main() {

    if [[ "$device_type" != "SpineRouter" ]]; then
        save_saidump
+    else
+        save_saidump_by_redis_save_cmd


i do not feel we should check device_type, and doing different saidump based on the device type. Can you check the asic db size and if it is larger than xxx entries, then we do the new way?

@lguohan, please help to review my updates. The variable ROUTE_TAB_LIMIT_DIRECT_ITERATION with default value 24000.

sonic-net#2972 SAI DUMP based on the route table size

sonic-net#2972 SAI DUMP based on the route table size * [saidump] • Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2, install Python library rdbtools into the syncd containter. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Updated sonic-buildimage/build_debian.sh, to add a new script file: files/scripts/saidump.sh into the host. This shell file does the below steps: For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files docker exec syncd$1 sh -c "rdb --command json /var/run/redis$1/dump.rdb | tee /var/run/redis$1/dump.json > /dev/null" 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json -m 100" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scrip

sonic-net#2972 SAI DUMP based on the route table size * [saidump] • Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2, install Python library rdbtools into the syncd containter. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Updated sonic-buildimage/build_debian.sh, to add a new script file: files/scripts/saidump.sh into the host. This shell file does the below steps: For each ASIC0, such as ASIC0, 1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null 2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" 3. Run rdb command to convert the dump files into JSON files docker exec syncd$1 sh -c "rdb --command json /var/run/redis$1/dump.rdb | tee /var/run/redis$1/dump.json > /dev/null" 4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json -m 100" 5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, to check the asic db size and if it is larger than xxx entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.

…_LIMIT_DIRECT_ITERATION to 24000

…file and displays/format the right output (#1288) Why I did it Fix issue: sonic-net/sonic-buildimage#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-buildimage#16466

To add the rdbtools into base docker. Move saidump.sh from host to syncd docker container.

scripts/generate_dump

JunhongMao · 2023-10-25T20:40:34Z

Lgtm. Can you add some test results to PR summary that with routes eg: 23k per asic on a multi-asic platform we don't see CPU high and swss timeouts in scenarios viz.

show tech support

one syncd crashed.

We have tested these cases and provided the result above. Please see the "How To Verify It" section of this PR:
#2972 (comment)

…rs. (#16466) Fix #13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

JunhongMao · 2023-11-14T15:47:52Z

@lguohan , please help to review this PR. Thanks.

judyjoseph · 2023-11-14T21:00:39Z

@JunhongMao please add a testcase for covering this case where "$route_size > $ROUTE_TAB_LIMIT_DIRECT_ITERATION" and the new saidump flow will be taken. You could add as part of this PR .. or a separate one

JunhongMao · 2023-11-15T15:48:58Z

@JunhongMao please add a testcase for covering this case where "$route_size > $ROUTE_TAB_LIMIT_DIRECT_ITERATION" and the new saidump flow will be taken. You could add as part of this PR .. or a separate one

@judyjoseph I would suggest to merge this PR first since it has been opened for a long time. I will create a test PR for that separately.

mlok-nokia · 2023-11-17T14:30:16Z

@judyjoseph Please merge this to 202205 branch also.

…rs. (sonic-net#16466) Fix sonic-net#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

…aidump_by_route_size (#2972) * * [saidump] • Saidump for DNX-SAI sonic-net/sonic-buildimage#13561 Solution and modification: To use the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. (1) Updated sonic-buildimage/build_debian.sh, to install Python library rdbtools into the host. (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. (3) Add a new script file: files/scripts/saidump.sh, to do the below steps For each ASIC0, such as ASIC0, #1. Save the Redis data. sudo sonic-db-cli -n asic$1 SAVE > /dev/null #2. Move dump files to /var/run/redisX/ docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/" #3. Run rdb command to convert the dump files into JSON files sudo python /usr/local/bin/rdb --command json /var/run/redis$1/dump.rdb | sudo tee /var/run/redis$1/dump.json > /dev/null #4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output. docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json" #5. clear sudo rm -f /var/run/redis$1/dump.rdb sudo rm -f /var/run/redis$1/dump.json (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, replace saidump with saidump.sh * * [saidump] • Saidump for DNX-SAI sonic-net/sonic-buildimage#13561

…file and displays/format the right output (#1288) Why I did it Fix issue: sonic-net/sonic-buildimage#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-buildimage#16466

…rs. (sonic-net#16466) Fix sonic-net#13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

…rs. (#16466) Fix #13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access. This solution uses the Redis SAVE command to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: sonic-net/sonic-utilities#2972 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 How did I do it? To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it. 1. Updated dockers/docker-base-bullseye/Dockerfile.j2, install Python library rdbtools into the all the docker-base-bullseye containers. 2. Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format. 3. To add a new script file: syncd/scripts/saidump.sh into the sairedis repo. This shell script does the following steps: For each ASIC, such as ASIC0, 3.1. Config Redis consistency directory. redis-cli -h $hostname -p $port CONFIG SET dir $redis_dir > /dev/null 3.2. Save the Redis data. redis-cli -h $hostname -p $port SAVE > /dev/null 3.3. Run rdb command to convert the dump files into JSON files rdb --command json $redis_dir/dump.rdb | tee $redis_dir/dump.json > /dev/null 3.4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump's result in standard output." saidump -r $redis_dir/dump.json -m 100 3.5. Clear the temporary files. rm -f $redis_dir/dump.rdb rm -f $redis_dir/dump.json 4. Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump. To check the asic db size and if it is larger than ROUTE_TAB_LIMIT_DIRECT_ITERATION (with default value 24000) entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB. How to verify it On T2 setup with more than 96K routes, execute CLI command -- generate_dump No error should be shown Download the generate_dump result and verify the saidump file after unpacking it.

sonic-net#2972 added two below functions into scripts/generate_dump. get_route_table_size_by_asic_id_and_ipver save_saidump_by_route_size The unittest scripts need to be added. Related PRs: sonic-net#2972 sonic-net/sonic-buildimage#16466 sonic-net/sonic-sairedis#1288 sonic-net/sonic-sairedis#1298 Microsoft ADO (25892277): Add two scripts: tests/saidump_test.py tests/saidump_test.sh To use below 6 test cases to verify the functionality of get_route_table_size_by_asic_id_and_ipver and save_saidump_by_route_size behave correctly. ``` saidump test list format: [ACIS number, ipv4 and ipv6 route table size, expected function save_cmd arguments] saidump_test_list = [ [1, 10000, "docker exec syncd saidump saidump"], [1, 12000, "docker exec syncd saidump saidump"], [1, 12001, "docker exec syncd saidump.sh saidump"], [1, 20000, "docker exec syncd saidump.sh saidump"], [2, 10000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12000, "docker exec syncd0 saidump saidump0\ndocker exec syncd1 saidump saidump1"], [2, 12001, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"], [2, 20000, "docker exec syncd0 saidump.sh saidump0\ndocker exec syncd1 saidump.sh saidump1"] ] ``` During the compiling stage, run the below command to check if it's PASSED. jumao@1b1ffba5949a:/sonic/src/sonic-utilities$ time python3 setup.py test tests/saidump_test.py::test_saidump PASSED

JunhongMao added 5 commits September 1, 2023 17:27

Merge branch 'master' of github.com:JunhongMao/sonic-utilities

5ecb3c1

This was referenced Sep 6, 2023

[VOQ][saidump] Install rdbtools into the docker base related containers. sonic-net/sonic-buildimage#16466

Merged

[VOQ][saidump] Enhance saidump with new option -r to parser the JSON file and displays/format the right output sonic-net/sonic-sairedis#1288

Merged

lguohan reviewed Sep 7, 2023

View reviewed changes

JunhongMao added a commit to JunhongMao/sonic-utilities that referenced this pull request Sep 19, 2023

Fixup based on the below PR comments.

9c02cd4

sonic-net#2972 SAI DUMP based on the route table size

JunhongMao added a commit to JunhongMao/sonic-utilities that referenced this pull request Sep 19, 2023

Fixup based on the below PR comments.

5948055

sonic-net#2972 SAI DUMP based on the route table size

JunhongMao force-pushed the master branch from ce38d0b to 5948055 Compare September 19, 2023 19:05

JunhongMao added a commit to JunhongMao/sonic-utilities that referenced this pull request Sep 19, 2023

Fixup based on the below PR comments.

7c4412c

sonic-net#2972 SAI DUMP based on the route table size

JunhongMao force-pushed the master branch from fd61a2e to 7c4412c Compare September 19, 2023 19:12

JunhongMao force-pushed the master branch from 6033574 to d7ae34d Compare September 19, 2023 20:08

JunhongMao force-pushed the master branch 3 times, most recently from 9621283 to d0598d9 Compare September 21, 2023 20:47

According to the testing group's advice, change the default ROUTE_TAB…

8a3f93b

…_LIMIT_DIRECT_ITERATION to 24000

JunhongMao force-pushed the master branch from d0598d9 to cd08c2e Compare September 30, 2023 05:20

JunhongMao mentioned this pull request Oct 3, 2023

[VOQ][saidump] To move saidump.sh from the sonic-buildimage repo to the sairedis repo sonic-net/sonic-sairedis#1298

Merged

10 tasks

JunhongMao changed the title ~~[VOQ][saidump] Modify generate_dump to save_saidump_by_redis_save_cmd~~ [VOQ][saidump] Modify generate_dump: replace save_saidump with save_saidump_by_route_size Oct 3, 2023

sonic-net/sonic-buildimage#16466 fixing based on review comments:

32f5607

To add the rdbtools into base docker. Move saidump.sh from host to syncd docker container.

JunhongMao force-pushed the master branch from 59efa78 to 32f5607 Compare October 3, 2023 18:47

Merge remote-tracking branch 'upstream/master'

64ad586

judyjoseph reviewed Oct 4, 2023

View reviewed changes

scripts/generate_dump Outdated Show resolved Hide resolved

JunhongMao force-pushed the master branch from 73fc380 to 1304eba Compare October 4, 2023 15:06

Merge remote-tracking branch 'upstream/master'

86d3efa

judyjoseph approved these changes Nov 8, 2023

View reviewed changes

judyjoseph requested a review from lguohan November 8, 2023 21:17

judyjoseph merged commit cd85569 into sonic-net:master Nov 15, 2023
5 checks passed

gechiang added Request for 202205 Branch Request for 202305 Branch labels Nov 18, 2023

mssonicbld mentioned this pull request Nov 19, 2023

[action] [PR:16466] [VOQ][saidump] Install rdbtools into the docker base related containers. sonic-net/sonic-buildimage#17222

Merged

10 tasks

StormLiangMS added the Included in 202305 Branch label Nov 19, 2023

mssonicbld mentioned this pull request Nov 21, 2023

[action] [PR:16466] [VOQ][saidump] Install rdbtools into the docker base related containers. sonic-net/sonic-buildimage#17244

Merged

10 tasks

mssonicbld mentioned this pull request Nov 21, 2023

[action] [PR:16466] [VOQ][saidump] Install rdbtools into the docker base related containers. sonic-net/sonic-buildimage#17245

Merged

10 tasks

judyjoseph mentioned this pull request Nov 21, 2023

Start to enable save_saidump in generate_dump for SpineRouter #3050

Closed

JunhongMao mentioned this pull request Dec 12, 2023

[VOQ][saidump] Add saidump unit test scripts #3079

Open

JunhongMao mentioned this pull request Dec 27, 2023

[VOQ][saidump] Enhance saidump with new option -r to parser the JSON file and displays/format the right output, fix compiling error sonic-net/sonic-sairedis#1335

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VOQ][saidump] Modify generate_dump: replace save_saidump with save_saidump_by_route_size #2972

[VOQ][saidump] Modify generate_dump: replace save_saidump with save_saidump_by_route_size #2972

JunhongMao commented Sep 6, 2023 •

edited by gechiang

Loading

linux-foundation-easycla bot commented Sep 6, 2023 •

edited

Loading

lguohan Sep 7, 2023

JunhongMao Sep 19, 2023 •

edited

Loading

JunhongMao commented Oct 25, 2023

JunhongMao commented Nov 14, 2023

judyjoseph commented Nov 14, 2023

JunhongMao commented Nov 15, 2023

mlok-nokia commented Nov 17, 2023

[VOQ][saidump] Modify generate_dump: replace save_saidump with save_saidump_by_route_size #2972

[VOQ][saidump] Modify generate_dump: replace save_saidump with save_saidump_by_route_size #2972

Conversation

JunhongMao commented Sep 6, 2023 • edited by gechiang Loading

Why did I do it?

Work item tracking

How did I do it?

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

linux-foundation-easycla bot commented Sep 6, 2023 • edited Loading

lguohan Sep 7, 2023

Choose a reason for hiding this comment

JunhongMao Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

JunhongMao commented Oct 25, 2023

JunhongMao commented Nov 14, 2023

judyjoseph commented Nov 14, 2023

JunhongMao commented Nov 15, 2023

mlok-nokia commented Nov 17, 2023

JunhongMao commented Sep 6, 2023 •

edited by gechiang

Loading

linux-foundation-easycla bot commented Sep 6, 2023 •

edited

Loading

JunhongMao Sep 19, 2023 •

edited

Loading