Add write socket buffer to fix CoPP rx performance issue

[cytsai0409]

Add write socket buffer to fix CoPP rx performance issue

- What I did

Add write socket buffer to fix packet drop issue in ptf_nn_agent.py

- How I did it

Add set command in the file build_debian.sh

- How to verify it

Run the CoPP tests in the testbed and verify the test results are passed

- Description for the changelog

Add write socket buffer to fix CoPP rx performance issue

[cytsai0409]

More detailed discussion:
sonic-mgmt#308

[pavel-shirshov]

Hi Jason,

Thank you for your PR.

I'd like to understand the reason of increasing the write buffer.

I increased the read buffer previously because ingress packets comes with the ASIC line rate which is impossible to reach by python read. So I increased socket memory buffer to ask Linux kernel preserve unread packets for ptf_nn_agent.

In your case contrary increasing write buffer means that your application writes packets with the rate higher than your NIC/ASIC can reach. But in your case a writer is a python and consumer is Linux kernel with ASIC. But as I understand python is much slower than Linux kernel + ASIC.

Can you please explain how your change can help you? I think increasing read buffer is enough.

Thanks

[pavel-shirshov]

I don't request any change. I'd like to understand why the changes were offered.

[cytsai0409]

Hi, Pavel:

Thanks for your review.

In the CoPP tests, the ptf_nn_agent will send packets back from DUT to test server to count matched packets and insufficient write socket buffer will cause ingress packet drop on DUT.

If the ptf_nn_agent does not send packets back from DUT to test server, I think increasing read socket buffer on DUT is enough. However this action is required by ptf_nn_agent on the test server.

When sending packet back to testbed server, I think the bottleneck is the between python and linux kernel, not on the ASIC.

For example, the CoPP test for DHCP is to send 100000 DHCP packets from test server to DUT without CoPP policy applied and expected low packet loss rate (<10%) on DUT.

Before increasing write socket buffer, the CoPP test for DHCP is failed due to more packet loss than expected.

• Ex: 2017-10-20 01:28:20 : total_rcv_pkt_cnt (83072) > pkt_rx_limit (90000): False

After increasing socket buffer, the CoPP test for DHCP is passed.

• Ex: 2017-10-20 01:46:07 : total_rcv_pkt_cnt (98733) > pkt_rx_limit (90000): True

Below is how we increase write buffer on Linux kernel and python.

To add write buffer on Linux kernel, we add the following line in the file /etc/sysctl.conf on DUT

• net.core.wmem_max = 2097152

To add write buffer on Python, we add the options below in the ptf_nn_agent command on DUT

• python ptf_nn_agent.py --device-socket 1@tcp://[DUT_MGMT_IP]:10900 -i 1-3@Ethernet12 --set-nn-rcv-buffer=10000000 --set-iface-rcv-buffer=10000000 --set-nn-snd-buffer=10000000 --set-iface-snd-buffer=10000000

As we tested the CoPP test will pass only when write buffer on Linux kernel and Python are both increased.

[pavel-shirshov]

Honestly I don't understand how Linux kernel could be slower then python?
How write buffer helps in your case?

[cytsai0409]

Fix this issue by increasing read socket buffer instead of write buffer:

1. sysctl.conf: net.core.rmem_max=109430400
2. ptf_nn_agent.conf: --set-nn-rcv-buffer=109430400 --set-iface-rcv-buffer=109430400

See more discussion: sonic-mgmt#308