Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

一直在复用已关闭的连接 #21

Closed
HanadaLee opened this issue Oct 12, 2023 · 6 comments
Closed

一直在复用已关闭的连接 #21

HanadaLee opened this issue Oct 12, 2023 · 6 comments

Comments

@HanadaLee
Copy link

在日志里面发现进程一直在复用关闭的连接，导致一直连接不上，除非重启进程

Oct 12 11:32:16 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:16 0c731 ws write err: use of closed network connection
Oct 12 11:32:16 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:16 0c731 ws -> tcp Boom!
Oct 12 11:32:16 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:17 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:17 0c731 ws -> tcp Boom!
Oct 12 11:32:17 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:17 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:17 0c731 ws -> tcp Boom!
Oct 12 11:32:17 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:17 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:17 0c731 ws -> tcp Boom!
Oct 12 11:32:17 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:17 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:17 0c731 ws write err: use of closed network connection
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:18 0c731 ws -> tcp Boom!
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:18 0c731 ws -> tcp Boom!
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:18 0c731 ws -> tcp Boom!
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:18 0c731 ws -> tcp Boom!
Oct 12 11:32:18 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:19 0c731 ws -> tcp Boom!
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:19 0c731 ws write err: use of closed network connection
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:19 0c731 ws -> tcp Boom!
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:19 0c731 ws -> tcp Boom!
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:19 0c731 ws -> tcp Boom!
Oct 12 11:32:19 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:20 0c731 ws -> tcp Boom!
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:20 0c731 ws write err: use of closed network connection
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:20 0c731 ws -> tcp Boom!
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:20 0c731 ws -> tcp Boom!
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:20 0c731 ws -> tcp Boom!
Oct 12 11:32:20 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:21 0c731 ws -> tcp Boom!
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:21 0c731 ws write err: use of closed network connection
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:21 0c731 ws -> tcp Boom!
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:21 0c731 ws -> tcp Boom!
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:21 0c731 ws -> tcp Boom!
Oct 12 11:32:21 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:22 0c731 ws -> tcp Boom!
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:22 0c731 ws -> tcp Boom!
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:22 0c731 ws write err: use of closed network connection
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:22 0c731 ws -> tcp Boom!
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:22 0c731 ws -> tcp Boom!
Oct 12 11:32:22 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:23 0c731 ws -> tcp Boom!
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:23 0c731 ws -> tcp Boom!
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:23 0c731 ws write err: use of closed network connection
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:23 0c731 ws -> tcp Boom!
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:23 0c731 ws -> tcp Boom!
Oct 12 11:32:23 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:24 0c731 ws -> tcp Boom!
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:24 0c731 ws -> tcp Boom!
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:24 0c731 ws -> tcp Boom!
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:24 0c731 ws write err: use of closed network connection
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:24 0c731 ws -> tcp Boom!
Oct 12 11:32:24 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:25 0c731 ws -> tcp Boom!
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:25 0c731 ws -> tcp Boom!
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:25 0c731 ws -> tcp Boom!
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:25 0c731 ws write err: use of closed network connection
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:25 0c731 ws -> tcp Boom!
Oct 12 11:32:25 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:26 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:26 0c731 ws -> tcp Boom!
Oct 12 11:32:26 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:26 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:26 0c731 ws -> tcp Boom!
Oct 12 11:32:26 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:26 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:26 0c731 ws -> tcp Boom!
Oct 12 11:32:26 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:26 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:26 0c731 ws write err: use of closed network connection
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:27 0c731 ws -> tcp Boom!
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:27 0c731 ws -> tcp Boom!
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:27 0c731 ws -> tcp Boom!
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:27 0c731 ws -> tcp Boom!
Oct 12 11:32:27 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:28 0c731 ws write err: use of closed network connection
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:28 0c731 ws -> tcp Boom!
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:28 0c731 ws -> tcp Boom!
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:28 0c731 ws -> tcp Boom!
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:28 0c731 ws -> tcp Boom!
Oct 12 11:32:28 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:29 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:29 0c731 ws write err: use of closed network connection
Oct 12 11:32:29 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:29 0c731 ws -> tcp Boom!
Oct 12 11:32:29 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:29 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:29 0c731 ws -> tcp Boom!
Oct 12 11:32:29 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:32:29 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:32:29 0c731 ws -> tcp Boom!
Oct 12 11:32:29 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
@zanjie1999
Copy link
Owner

哇你这是做了什么操作呢

@HanadaLee
Copy link
Author

HanadaLee commented Oct 12, 2023 •
edited
Loading

没啥操作啊，/usr/local/tcp2ws/bin/tcp2ws wss://xxxxxxx.com/report 127.0.0.1:35601，接入测的是cloudflare，我这个是用来上报机器状态信息用的

重启进程后就正常了

Oct 12 11:41:31 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:41:31 0c731 ws -> tcp Boom!
Oct 12 11:41:31 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:41:32 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:41:32 0c731 ws -> tcp Boom!
Oct 12 11:41:32 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:41:32 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:41:32 0c731 ws -> tcp Boom!
Oct 12 11:41:32 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:41:32 af-za-gp-oracle-287 systemd[1]: Stopping TCP to WebSocket Proxy...
Oct 12 11:41:33 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:41:33 0c731 ws -> tcp Boom!
Oct 12 11:41:33 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:41:33 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:41:33 0c731 ws write err: use of closed network connection
Oct 12 11:41:33 af-za-gp-oracle-287 config.conf[93779]: 2023/10/12 11:41:33 0c731 ws -> tcp Boom!
Oct 12 11:41:33 af-za-gp-oracle-287 config.conf[93779]: repeated read on failed websocket connection
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: TCP over WebSocket (tcp2ws) with UDP support 10.4
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: https://github.com/zanjie1999/tcp-over-websocket
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: Client: ws://tcp2wsUrl localPort
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: Server: ip:port tcp2wsPort
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: Use wss: ip:port tcp2wsPort server.crt server.key
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: Make ssl cert:
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: openssl genrsa -out server.key 2048
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: openssl ecparam -genkey -name secp384r1 -out server.key
Oct 12 11:41:34 af-za-gp-oracle-287 tcp2ws[388953]: openssl req -new -x509 -sha256 -key server.key -out server.crt -days 36500
Oct 12 11:41:34 af-za-gp-oracle-287 systemd[1]: tcp2ws.service: Succeeded.
Oct 12 11:41:34 af-za-gp-oracle-287 systemd[1]: Stopped TCP to WebSocket Proxy.
Oct 12 11:41:34 af-za-gp-oracle-287 systemd[1]: Starting TCP to WebSocket Proxy...
Oct 12 11:41:34 af-za-gp-oracle-287 systemd[1]: Started TCP to WebSocket Proxy.
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 nslookup status.hanada.info
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 Use System DNS 127.0.0.1
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 tcping 104.21.74.113 98ms
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 tcping 172.67.157.238 99ms
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 Prefer IP 104.21.74.113 for xxxxxxxxx.com
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 Client Started 127.0.0.1:35601 -> wss://xxxxxxxxx.com/report
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 DNS TTL:  295 s
Oct 12 11:41:34 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:34 Create UDP Listen: 127.0.0.1:35601
Oct 12 11:41:38 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:38 new tcp conn:
Oct 12 11:41:38 af-za-gp-oracle-287 config.conf[388966]: 2023/10/12 11:41:38 dial 29109

@zanjie1999
Copy link
Owner

可以找到循环之前的日志吗，因为正常情况下另一个线程会标记删除已经关掉的ws连接

@HanadaLee
Copy link
Author

我得找一下，不过日志量太大了有点难找（一天时间貌似产生了几百万条）

@HanadaLee
Copy link
Author

HanadaLee commented Oct 12, 2023 •
edited
Loading

过滤了一上午，发现就这里开始异常的，后面几个小时几百万条都是前面截图那种了（昨晚升级到10.4的，今天上午大概有十几台机有这类问题）
error.log

@zanjie1999
Copy link
Owner

找到问题了，已经改好了，造成这个问题的原因是因为ws被掐断后客户端马上重连也无法连上导致的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zanjie1999 @HanadaLee