Performance of nng_stream_send?

[alawn-wang]

Hi,
I have a confusution about the performance, I have create a webserve, when it received a request of connection it would create a nng_stream，and a nng_stream will have two nng_aio object , one for send, another for receive. When send a msg, I have to use 'nng_stream_send' and then use 'nng_aio_wait' to ensure the 'aio' object can be used for send next time.
I have try to send 8K/s msgs for a connction(msg size betwwen 200~500 bytes), the nng_stream_send call need 10 milliseconds, and the nng_aio_wait will be blocked almost 40 milliseconds. it seems a little low-effective.
If the connections up to a dozen, the process will increased more 400 milliseconds. I want to send msg as far as quickly, is there any way can solve this problem? use more aio objects for a stream(connection)? Or have any other can send msg in a really asynchronous mode.
I know send msg in low-latency and high-throughput seems a little difficult. But i sincerely want to known how to slove this problem

[JaylinYu]

Recommend using nng_send_aio and managing cb on your own if you are looking for extreme performance of NNG.

[alawn-wang]

thanks for your advice, I have look through the function of nng_send_aio, it seems just send return immediatly. But nng_stream_send seems do the same thing? I think that the nng may be wait implictly when schedule task. I will try to managing cb without any wait.
But the thing that worried me mostly is the performan decreasing caused by hardware limit. Frankly speaking, the throughput is not very high(no more than 300Mbps/s), but send will be call high-frequency( more than 8000/s). To get lower latency, I disabled the Nagle, every 'send' may be caused a system-call( not ensure yet)

[JaylinYu]

You can allocate more sending aio to achieve parallesim, but the latency indeed is limited by hardware interupption -- which is your network interface driver. Software is here to extract the last drop of its power.

[gdamore]

Disabling Nagle may be very important -- in fact I recommend most web applications disable it. NNG does that by default on SP sockets, not sure if we did that on stream sockets by default. If you block, waiting for that send to complete, and Nagle (and also btw delayed ack can be fairly tragic -- that's usually not a socket option but an OS tunable) are enabled, you'll block for milliseconds waiting before it even tries to flush the buffer. (Nagle's algorithm was designed and optimized for human typing speeds with low baud -- say 300 bps or so -- connections. It fares poorly in modern systems.)

Anyway, try not to block. If you can send and receive asynchronously, then use two separate aios. If you have multiple connections, use one or two aios for each socket. This will get quite good performance.

[alawn-wang]

Disabling Nagle may be very important -- in fact I recommend most web applications disable it. NNG does that by default on SP sockets, not sure if we did that on stream sockets by default. If you block, waiting for that send to complete, and Nagle (and also btw delayed ack can be fairly tragic -- that's usually not a socket option but an OS tunable) are enabled, you'll block for milliseconds waiting before it even tries to flush the buffer. (Nagle's algorithm was designed and optimized for human typing speeds with low baud -- say 300 bps or so -- connections. It fares poorly in modern systems.)

Anyway, try not to block. If you can send and receive asynchronously, then use two separate aios. If you have multiple connections, use one or two aios for each socket. This will get quite good performance.

I have notice that almost transport‘s’(In directory transport) send_cb() in nng，just call 'nni_aio_finish_error()' if send unsuccessfully. All this 'pipe_trans' have exactly one aio for send, so message will send one by one. If send msg error the user would know which msg sent failed. If a connection have more than one 'aio' object, and we send msg in parallel use multi aios, the user will get error by send_cb(), but can't get which msg cause the error？

Would you mind tell me the consideration when you design of pipe_trans? one 'aio' for send let the msg-send become serial.