tcp 自动关闭。查不到原因(libhv1.3.0)

[LightSun]

你好。libhv是一个非常棒的项目。但是我在处理大文件上传时，遇到一个麻烦的问题。就是发送完成后会自动关闭tcp。 我不知道为什么。

客户端是c++, 服务器是java。java服务端我查过，没有报任何异常。
现象是这样的。

c++ 这边采用的是多线程写文件片段。java这边读的时候是 一个线程（一个客户端）。我发现java还没有读取完。c++ 就自动关闭tcp了, 不知道为什么。我这边客户端和服务端都没有主动调用close. 下图是我debug的断点。

disconnected to 127.0.0.1:12390! connfd=13, all task will be remove

你们可以帮助下么？

这个是libhv的日志

2023-01-12 09:22:32.929 INFO  http server listening on 0.0.0.0:21353 [HttpServer.cpp:336:http_server_run]
2023-01-12 09:22:32.930 INFO  EventLoop started, pid=26041 tid=26045 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:22:32.930 INFO  EventLoop started, pid=26041 tid=26041 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:22:32.930 INFO  EventLoop started, pid=26041 tid=26046 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:23:19.495 INFO  [127.0.0.1:39660] Upgrade: websocket [HttpServer.cpp:111:on_recv]
2023-01-12 09:23:19.495 INFO  [26041-26046][127.0.0.1:39660][GET /]=>[101 Switching Protocols] [HttpServer.cpp:172:on_recv]
2023-01-12 09:23:21.468 INFO  EventLoopThread started, tid=26147 [EventLoopThread.h:86:loop_thread]
2023-01-12 09:25:19.798 WARN  write len=1048668 enqueue 1048668, bufsize=8979536 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:25:19.799 WARN  write len=1048668 enqueue 1048668, bufsize=10028204 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:25:19.800 WARN  write len=1048668 enqueue 1048668, bufsize=11076872 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:25:19.801 WARN  write len=1048668 enqueue 1048668, bufsize=12125540 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:28:32.729 INFO  http server listening on 0.0.0.0:21353 [HttpServer.cpp:336:http_server_run]
2023-01-12 09:28:32.729 INFO  EventLoop started, pid=26541 tid=26541 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:28:32.729 INFO  EventLoop started, pid=26541 tid=26543 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:28:32.729 INFO  EventLoop started, pid=26541 tid=26542 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:28:39.403 INFO  http server listening on 0.0.0.0:21353 [HttpServer.cpp:336:http_server_run]
2023-01-12 09:28:39.403 INFO  EventLoop started, pid=26727 tid=26731 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:28:39.403 INFO  EventLoop started, pid=26727 tid=26727 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:28:39.403 INFO  EventLoop started, pid=26727 tid=26732 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:28:46.647 INFO  [127.0.0.1:39704] Upgrade: websocket [HttpServer.cpp:111:on_recv]
2023-01-12 09:28:46.647 INFO  [26727-26731][127.0.0.1:39704][GET /]=>[101 Switching Protocols] [HttpServer.cpp:172:on_recv]
2023-01-12 09:28:48.405 INFO  EventLoopThread started, tid=26736 [EventLoopThread.h:86:loop_thread]
2023-01-12 09:29:48.991 INFO  http server listening on 0.0.0.0:21353 [HttpServer.cpp:336:http_server_run]
2023-01-12 09:29:48.992 INFO  EventLoop started, pid=26949 tid=26953 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:29:48.992 INFO  EventLoop started, pid=26949 tid=26949 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:29:48.992 INFO  EventLoop started, pid=26949 tid=26954 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:30:30.565 INFO  [127.0.0.1:39732] Upgrade: websocket [HttpServer.cpp:111:on_recv]
2023-01-12 09:30:30.565 INFO  [26949-26953][127.0.0.1:39732][GET /]=>[101 Switching Protocols] [HttpServer.cpp:172:on_recv]
2023-01-12 09:30:32.512 INFO  EventLoopThread started, tid=27047 [EventLoopThread.h:86:loop_thread]
2023-01-12 09:34:48.231 INFO  http server listening on 0.0.0.0:21353 [HttpServer.cpp:336:http_server_run]
2023-01-12 09:34:48.231 INFO  EventLoop started, pid=27664 tid=27664 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:34:48.231 INFO  EventLoop started, pid=27664 tid=27665 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:34:48.231 INFO  EventLoop started, pid=27664 tid=27666 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:35:36.869 INFO  [127.0.0.1:39770] Upgrade: websocket [HttpServer.cpp:111:on_recv]
2023-01-12 09:35:36.869 INFO  [27664-27664][127.0.0.1:39770][GET /]=>[101 Switching Protocols] [HttpServer.cpp:172:on_recv]
2023-01-12 09:35:38.262 INFO  EventLoopThread started, tid=27753 [EventLoopThread.h:86:loop_thread]
2023-01-12 09:35:40.815 WARN  write len=1048668 enqueue 1048668, bufsize=9012304 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.816 WARN  write len=1048668 enqueue 1048668, bufsize=10060972 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.817 WARN  write len=1048668 enqueue 1048668, bufsize=11109640 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.817 WARN  write len=1048668 enqueue 1048668, bufsize=12158308 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.818 WARN  write len=1048668 enqueue 1048668, bufsize=13206976 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.819 WARN  write len=1048668 enqueue 1048668, bufsize=14255644 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.820 WARN  write len=1048668 enqueue 1048668, bufsize=15304312 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.820 WARN  write len=1048668 enqueue 1048668, bufsize=16352980 over high water 8388608 [nio.c:540:hio_write]
2023-01-12 09:35:40.820 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.820 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.820 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.821 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.821 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.822 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.822 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.823 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.823 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.824 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.824 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.824 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.824 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.825 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.825 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.825 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.825 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.826 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.826 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.826 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.826 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.826 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.826 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.827 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.827 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.828 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.828 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.828 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.828 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.829 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.829 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.829 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.829 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.830 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_write]
2023-01-12 09:35:40.830 ERROR write bufsize > 16777216, close it! [nio.c:521:hio_2023-01-12 09:38:13.401 INFO  http server listening on 0.0.0.0:21353 [HttpServer.cpp:336:http_server_run]
2023-01-12 09:38:13.402 INFO  EventLoop started, pid=28808 tid=28812 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:38:13.402 INFO  EventLoop started, pid=28808 tid=28813 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:38:13.402 INFO  EventLoop started, pid=28808 tid=28808 [HttpServer.cpp:316:loop_thread]
2023-01-12 09:38:56.078 INFO  [127.0.0.1:39804] Upgrade: websocket [HttpServer.cpp:111:on_recv]
2023-01-12 09:38:56.078 INFO  [28808-28813][127.0.0.1:39804][GET /]=>[101 Switching Protocols] [HttpServer.cpp:172:on_recv]
2023-01-12 09:39:00.128 INFO  EventLoopThread started, tid=28915 [EventLoopThread.h:86:loop_thread]

[ithewei]

https://github.com/ithewei/libhv/wiki/%E4%B8%AD%E6%96%87FAQ#q%E6%95%B0%E6%8D%AE%E9%87%8F%E8%BF%87%E5%A4%A7%E5%8F%91%E9%80%81%E8%BF%87%E5%BF%AB%E5%AF%BC%E8%87%B4%E6%96%AD%E9%93%BE%E6%83%85%E5%86%B5%E6%8E%92%E6%9F%A5

Q：数据量过大，发送过快，导致断链情况排查
A：当日志里出现ERROR write bufsize > xxx, close it!的字样时，说明写缓存满了，hv内部自动断开了连接。

对于大数据的发送，我们应该做分片和流控，不然带宽不够或者对端接收过慢都会导致内部写缓存堆积，如果不加约束就会导致内存占用过大，所以hv内部对写缓存有个最大限制是16M，可以通过hio_set_max_write_bufsize或者channel->setMaxWriteBufsize修改该值，当然最正确的方法还是做流控，在channel->onwrite回调里，通过channel->isWriteComplete判断写完成时再发送下一帧数据，也可以通过channel->writeBufsize()获取当前写缓冲数据积压情况做决定。 具体代码示例可参考 examples/tinyhttpd.c 里的 http_serve_file。

[LightSun]

好的。我当时也是看源代码才知道问题。