Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to connect to server #2489

Closed
huangheding opened this issue Jun 29, 2020 · 10 comments
Closed

failed to connect to server #2489

huangheding opened this issue Jun 29, 2020 · 10 comments
Assignees
@Shawshank-Smile @Aries-Lee1991
Labels
bug Something isn't working

Comments

@huangheding
Copy link

docker镜像里taos客户端连接服务端

taos server版本 1.6.6.0
测试docker镜像ubuntu18.04 taos client 版本1.6.6.0

问题：可以连接 可以use database 可以describe table，但是不能select insert

会直接阻塞在那持续一段时间抛出异常 failed to connect to server

因为项目打包在docker镜像上运行 ，所以也存在该问题
@huangheding huangheding added the bug Something isn't working label Jun 29, 2020
@huangheding
Copy link
Author

toas client log：

            new log file

==================================================
06/29 09:35:14.396613 478 7fb795b39b80 ERROR UTL can't get card name from ip:192.168.1.169
06/29 09:35:14.396792 478 7fb795b39b80 UTL timezone not configured, set to system default: (UTC, +0000)
06/29 09:35:14.396854 478 7fb795b39b80 UTL locale not configured, set to system default:C.UTF-8
06/29 09:35:14.396865 478 7fb795b39b80 UTL charset not configured, set to system default:UTF-8
06/29 09:35:14.396875 478 7fb795b39b80 UTL taos config & system info:
06/29 09:35:14.396877 478 7fb795b39b80 UTL ==================================
06/29 09:35:14.396878 478 7fb795b39b80 UTL masterIp: 192.168.1.169
06/29 09:35:14.396880 478 7fb795b39b80 UTL localIp: 0.0.0.0
06/29 09:35:14.396882 478 7fb795b39b80 UTL mgmtShellPort: 6030
06/29 09:35:14.396883 478 7fb795b39b80 UTL vnodeShellPort: 6035
06/29 09:35:14.396885 478 7fb795b39b80 UTL configDir: /etc/taos
06/29 09:35:14.396886 478 7fb795b39b80 UTL logDir: /var/log/taos
06/29 09:35:14.396888 478 7fb795b39b80 UTL scriptDir: /etc/taos
06/29 09:35:14.396914 478 7fb795b39b80 UTL numOfThreadsPerCore: 1.000000
06/29 09:35:14.396920 478 7fb795b39b80 UTL affectedRowsMod: 0
06/29 09:35:14.396922 478 7fb795b39b80 UTL rpcTimer: 300(ms)
06/29 09:35:14.396924 478 7fb795b39b80 UTL rpcMaxTime: 600(s)
06/29 09:35:14.396925 478 7fb795b39b80 UTL shellActivityTimer: 3(s)
06/29 09:35:14.396926 478 7fb795b39b80 UTL meterMetaKeepTimer: 7200(s)
06/29 09:35:14.396928 478 7fb795b39b80 UTL metricMetaKeepTimer: 600(s)
06/29 09:35:14.396929 478 7fb795b39b80 UTL defaultDB:
06/29 09:35:14.396930 478 7fb795b39b80 UTL defaultUser: root
06/29 09:35:14.396931 478 7fb795b39b80 UTL sockettype: udp
06/29 09:35:14.396932 478 7fb795b39b80 UTL compressMsgSize: -1
06/29 09:35:14.396933 478 7fb795b39b80 UTL maxSQLLength: 65380(byte)
06/29 09:35:14.396934 478 7fb795b39b80 UTL maxAuthRetryTime: 5(byte)
06/29 09:35:14.396935 478 7fb795b39b80 UTL maxNumOfOrderedRes: 100000
06/29 09:35:14.396936 478 7fb795b39b80 UTL timezone: (UTC, +0000)
06/29 09:35:14.396937 478 7fb795b39b80 UTL locale: C.UTF-8
06/29 09:35:14.396938 478 7fb795b39b80 UTL charset: UTF-8
06/29 09:35:14.396939 478 7fb795b39b80 UTL numOfLogLines: 10000000
06/29 09:35:14.396940 478 7fb795b39b80 UTL asyncLog: 1
06/29 09:35:14.396940 478 7fb795b39b80 UTL debugFlag: 131
06/29 09:35:14.396941 478 7fb795b39b80 UTL rpcDebugFlag: 131
06/29 09:35:14.396942 478 7fb795b39b80 UTL tmrDebugFlag: 131
06/29 09:35:14.396943 478 7fb795b39b80 UTL cDebugFlag: 131
06/29 09:35:14.396944 478 7fb795b39b80 UTL jniDebugFlag: 131
06/29 09:35:14.396945 478 7fb795b39b80 UTL odbcDebugFlag: 131
06/29 09:35:14.396946 478 7fb795b39b80 UTL uDebugFlag: 131
06/29 09:35:14.396947 478 7fb795b39b80 UTL qDebugFlag: 131
06/29 09:35:14.396948 478 7fb795b39b80 UTL gitinfo: 1e63827
06/29 09:35:14.396949 478 7fb795b39b80 UTL gitinfoOfInternal:
06/29 09:35:14.396950 478 7fb795b39b80 UTL buildinfo: Built by root at 2020-06-11 14:28
06/29 09:35:14.396951 478 7fb795b39b80 UTL version: 1.6.6.0
06/29 09:35:14.396951 478 7fb795b39b80 UTL dataDir: /var/lib/taos
06/29 09:35:14.396952 478 7fb795b39b80 UTL os pageSize: 4096(KB)
06/29 09:35:14.396953 478 7fb795b39b80 UTL os openMax: 1048576
06/29 09:35:14.396954 478 7fb795b39b80 UTL os streamMax: 16
06/29 09:35:14.396955 478 7fb795b39b80 UTL os numOfCores: 2
06/29 09:35:14.396956 478 7fb795b39b80 UTL os totalDisk: 0.000000(GB)
06/29 09:35:14.396957 478 7fb795b39b80 UTL os totalMemory: 1968(MB)
06/29 09:35:14.396960 478 7fb795b39b80 UTL os sysname: Linux
06/29 09:35:14.396961 478 7fb795b39b80 UTL os nodename: fa756492d0c5
06/29 09:35:14.396965 478 7fb795b39b80 UTL os release: 4.19.76-linuxkit
06/29 09:35:14.396966 478 7fb795b39b80 UTL os version: #1 SMP Tue May 26 11:42:35 UTC 2020
06/29 09:35:14.396967 478 7fb795b39b80 UTL os machine: x86_64
06/29 09:35:14.396967 478 7fb795b39b80 UTL ==================================
06/29 09:35:14.402740 478 7fb795b39b80 ERROR TSC 0xbccfc0 SQL result:76, not supported operation pObj:0xbcce70
06/29 09:35:57.024110 478 7fb794562700 ERROR TSC 0x7fb778030350 reach the max retry:6, code:29
06/29 09:35:57.024110 478 7fb793d61700 ERROR TSC 0x7fb778051720 reach the max retry:6, code:29
06/29 09:35:57.024892 478 7fb794562700 ERROR TSC 0x7fb778036ed0 reach the max retry:6, code:29
06/29 09:35:57.025364 478 7fb794562700 ERROR TSC 0x7fb7780a8c90 reach the max retry:6, code:29
06/29 09:35:57.025521 478 7fb794562700 ERROR TSC 0x7fb7780366b0 reach the max retry:6, code:29
06/29 09:35:57.025690 478 7fb794562700 ERROR TSC 0x7fb7780bca90 reach the max retry:6, code:29
06/29 09:35:57.025776 478 7fb793d61700 ERROR TSC 0x7fb778050d60 reach the max retry:6, code:29
06/29 09:35:57.025873 478 7fb794562700 ERROR TSC 0x7fb7780e4ae0 reach the max retry:6, code:29
06/29 09:35:57.026277 478 7fb794562700 ERROR TSC 0x7fb778051470 reach the max retry:6, code:29
06/29 09:35:57.026533 478 7fb793d61700 ERROR TSC 0x7fb7780d0ad0 reach the max retry:6, code:29
06/29 09:36:07.251041 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb778050d60 retrieve failed,code:-83,orderOfSub:0 failed.no more retry,set global code:-83
06/29 09:36:07.251053 478 7fb794562700 ERROR TSC 0xbccfc0 sub:0x7fb778051470 retrieve failed,code:-83,orderOfSub:1 failed.no more retry,set global code:-83
06/29 09:36:07.251092 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb778051720 retrieve failed,code:-83,orderOfSub:2 failed.no more retry,set global code:-83
06/29 09:36:07.251106 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb7780366b0 retrieve failed,code:-83,orderOfSub:3 failed.no more retry,set global code:-83
06/29 09:36:07.251117 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb778036ed0 retrieve failed,code:-83,orderOfSub:4 failed.no more retry,set global code:-83
06/29 09:36:07.251127 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb778030350 retrieve failed,code:-83,orderOfSub:5 failed.no more retry,set global code:-83
06/29 09:36:07.251136 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb7780a8c90 retrieve failed,code:-83,orderOfSub:6 failed.no more retry,set global code:-83
06/29 09:36:07.251147 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb7780bca90 retrieve failed,code:-83,orderOfSub:7 failed.no more retry,set global code:-83
06/29 09:36:07.251157 478 7fb794562700 ERROR TSC 0xbccfc0 sub:0x7fb7780d0ad0 retrieve failed,code:-83,orderOfSub:8 failed.no more retry,set global code:-83
06/29 09:36:07.251161 478 7fb793d61700 ERROR TSC 0xbccfc0 sub:0x7fb7780e4ae0 retrieve failed,code:-83,orderOfSub:9 failed.no more retry,set global code:-83
06/29 09:36:07.251165 478 7fb793d61700 ERROR TSC 0xbccfc0 retrieve from 10 vnode(s) completed,code:-83.FAILED.
06/29 09:36:07.251204 478 7fb783fff700 ERROR TSC 0xbccfc0 SQL result:83, query cancelled pObj:0xbcce70

@huangheding
Copy link
Author

taos server log：
06/29 17:33:01.130896 8998 7fe859aae700 MND connection from root is closed
06/29 17:35:13.525639 8998 7fe8592ad700 MND pConn:0x7fe85c2ff190 is rebuild, destIp:0xa901a8c0:192.168.1.169 publicIp:0xa901a8c0:192.168.1.169 usePublicIp:1
06/29 17:35:13.525948 8998 7fe8592ad700 MND user:root login from 192.168.1.144, code:0
06/29 17:35:13.527735 8998 7fe859aae700 MND pShow:0x7fe7e00018d0 is allocated
06/29 17:35:13.527761 8998 7fe859aae700 ERROR MND pShow:0x7fe7e00018d0, type:14 vpeers-rsp, failed to get Meta, code:76
06/29 17:35:17.130491 8998 7fe859aae700 MND pConn:0x7fe85c2ff210 is rebuild, destIp:0xa901a8c0:192.168.1.169 publicIp:0xa901a8c0:192.168.1.169 usePublicIp:1
06/29 17:35:17.130532 8998 7fe859aae700 MND login from:9001a8c0:45779
06/29 17:35:17.130610 8998 7fe858aac700 MND DB is change to:0.xiaojin by root
06/29 17:35:23.135896 8998 7fe8592ad700 MND connection from root is closed
06/29 17:35:28.985824 8998 7fe8592ad700 MND pConn:0x7fe85c2ff290 is rebuild, destIp:0xa901a8c0:192.168.1.169 publicIp:0xa901a8c0:192.168.1.169 usePublicIp:1
06/29 17:35:28.985859 8998 7fe8592ad700 MND login from:9001a8c0:45779
06/29 17:35:28.985895 8998 7fe8592ad700 MND 0.xiaojin.xj_ali_point, uid:104363790266625593 meter meta is retrieved
06/29 17:35:28.988807 8998 7fe859aae700 MND metric:0.xiaojin.xj_ali_point retrieve all meter, no query condition
06/29 17:35:28.990539 8998 7fe859aae700 MND metric:0.xiaojin.xj_ali_point numOfRes:9693
06/29 17:35:28.996290 8998 7fe859aae700 MND metric:0.xiaojin.xj_ali_point metric-meta tables:9693, vnode:10
06/29 17:35:28.996309 8998 7fe859aae700 MND metric-meta msg size 194414
06/29 17:35:34.990866 8998 7fe859aae700 MND connection from root is closed
06/29 17:36:14.215894 8998 7fe859aae700 MND connection from root is closed

@Aries-Lee1991
Copy link
Contributor

OK, we'll check it.

@Aries-Lee1991 Aries-Lee1991 self-assigned this Jun 30, 2020
@huangheding
Copy link
Author

你这个是自己制作的镜像文件吗？docker container启动的时候是否做了端口绑定以及绑定了哪些端口？

taos server没用docker，server跑在一台本地服务器上面，为了测试 我构建了一个docker ubuntu 18.04的镜像 然后把taos 客户端1660 copy过去 然后在容器中安装客户端，然后taos -h 服务器这样连接的，执行查询或者插入就会阻塞

@huangheding
Copy link
Author

192.168.1.144是什么ip地址？是不是你的docker的宿主机的地址？这个问题可能是你客户端在docker里，访问taos server时docker网络 的问题。
建议客户端不要放在docker里运行；我们有对应的windows，linux的客户端软件，可以安装个客户端软件后再试试

对的 144是我docker宿主机地址 169是taos server服务器所在地址，不用docker是可以的 这只是我测试来着，但是项目要打包在docker镜像中跑啊，现在的问题就是项目跑着跑着就阻塞了 我才排查测试出这个问题所在，，网络的话 这个taos客户端是可以连接服务端的啊 这代表网络就是通的不是嘛

@huangheding
Copy link
Author

你的安装软件包文件名贴一下，服务端和客户端端的文件名都贴一下

TDengine-server-1.6.6.0-Linux-x64.tar.gz TDengine-client-1.6.6.0-Linux-x64.tar.gz

@huangheding
Copy link
Author

1.你在客户端和server端的/etc/taos/taos.cfg文件中都加上一句：debugFlag 135
2.然后，把两边的/var/log/taos/下面的文件都删掉
3.重启taosd和客户端，再跑一遍，重现故障
4.再吧两边的/var/log/taos/下的文件打包后上传上来

不好意思才回复您，因为一直在测试问题，，
现在把项目(golang)跑在跟taos服务端同一个服务器上面的docker容器内，运行大约几十秒后就会阻塞
阻塞位置是对taos的select max()操作
go连接taos尝试使用连接池、单连接每次执行open close、单连接open一次永远不close 运行结果都是跟上述问题一致

下面是服务端log ，麻烦了~

taosd0.log
taos0.0.log

@ghost ghost assigned Shawshank-Smile Jul 1, 2020
@huangheding
Copy link
Author

抱歉 之前的log修改debugflag没有成功 ，我又重新复现了一下问题 这次是135的log
这次阻塞在 taos insert 执行
taoslog0.0.log

@huangheding
Copy link
Author

从日志里看不出阻塞在insert的现象，要不你把你的写入程序在这里贴一下，我们看看有没有使用上的问题

@huangheding huangheding mentioned this issue Jul 2, 2020
Closed
@huangheding
Copy link
Author

huangheding commented Jul 2, 2020 •
edited
Loading

从日志里看不出阻塞在insert的现象，要不你把你的写入程序在这里贴一下，我们看看有没有使用上的问题

阻塞问题解决了， 我这面把 sockettype 从udp改成了tcp之后不存在阻塞问题了 ，
有一个新问题 现在项目(golang)跑一段时间后就会停掉 抛出的是segmentation fault (core dumped)

应该是C的指针内存问题 或者是堆栈溢出等等 ，咱们这边关注一下

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Shawshank-Smile Shawshank-Smile

@Aries-Lee1991 Aries-Lee1991

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Shawshank-Smile @Aries-Lee1991 @huangheding