Proxy list is empty #10

superkilly · 2015-12-23T03:03:22Z

下面是代码：
JedisResourcePool jedisPool = RoundRobinJedisPool.create().
curatorClient("192.168.25.247:2181", 20000).
zkProxyDir("/zk/codis/db_test/proxy").build();
Jedis jedis = jedisPool.getResource();
jedis.set("foo", "bar");
String value = jedis.get("foo");
System.out.println(value);

下面是日志：
[2015-12-23 11:02:39_434] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl [main.start:238] Starting
[2015-12-23 11:02:39_462] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:zookeeper.version=3.4.7-1713338, built on 11/09/2015 04:32 GMT
[2015-12-23 11:02:39_465] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:host.name=JQ0640.chinapnr.com
[2015-12-23 11:02:39_465] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:java.version=1.7.0_45
[2015-12-23 11:02:39_465] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:java.vendor=Oracle Corporation
[2015-12-23 11:02:39_465] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:java.home=C:\Program Files\Java\jdk1.7.0_45\jre
[2015-12-23 11:02:39_465] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:java.class.path=E:\JavaTestSpace\HuiFuExamPro\bin;E:\JavaTestSpace\HuiFuExamPro\lib\jodis-0.2.2.jar;E:\JavaTestSpace\HuiFuExamPro\lib\jedis-2.7.2.jar;E:\JavaTestSpace\HuiFuExamPro\lib\commons-pool2-2.4.jar;E:\JavaTestSpace\HuiFuExamPro\lib\slf4j-api-1.6.1.jar;E:\JavaTestSpace\HuiFuExamPro\lib\slf4j-log4j12-1.6.1.jar;E:\JavaTestSpace\HuiFuExamPro\lib\log4j-1.2.16.jar;E:\JavaTestSpace\HuiFuExamPro\lib\jackson-annotations-2.7.0-20151116.041133-14.jar;E:\JavaTestSpace\HuiFuExamPro\lib\jackson-core-2.7.0-20151126.183923-32.jar;E:\JavaTestSpace\HuiFuExamPro\lib\jackson-databind-2.7.0-20151125.063421-188.jar;E:\JavaTestSpace\HuiFuExamPro\lib\guava-18.0.jar;E:\JavaTestSpace\HuiFuExamPro\lib\cobertura-2.1.1.jar;E:\JavaTestSpace\HuiFuExamPro\lib\zookeeper-3.4.7.jar;E:\JavaTestSpace\HuiFuExamPro\lib\curator-client-2.5.0.jar;E:\JavaTestSpace\HuiFuExamPro\lib\curator-framework-2.5.0.jar;E:\JavaTestSpace\HuiFuExamPro\lib\curator-recipes-2.5.0.jar
[2015-12-23 11:02:39_465] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:java.library.path=C:\Program Files\Java\jdk1.7.0_45\bin;C:\windows\Sun\Java\bin;C:\windows\system32;C:\windows;D:\app\killy.zhang\product\11.2.0\client_1;C:\oracle\product\10.2.0\client_1\bin;D:\Program Files\Java\jdk1.6.0_22\bin;D:\Program Files\Java\jre6\bin;D:\Software\apache-maven-3.1.1/bin;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;D:\Software\windows linux commands;D:\Program Files\TortoiseSVN\bin;D:\Program Files (x86)\WinSCP;D:\Program Files (x86)\Windows7Master;E:\技术研究\ANT\apache-ant-1.8.2\bin;C:\Program Files (x86)\Python27;.
[2015-12-23 11:02:39_466] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:java.io.tmpdir=C:\Users\KILLY~1.ZHA\AppData\Local\Temp
[2015-12-23 11:02:39_466] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:java.compiler=
[2015-12-23 11:02:39_466] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:os.name=Windows 7
[2015-12-23 11:02:39_466] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:os.arch=amd64
[2015-12-23 11:02:39_466] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:os.version=6.1
[2015-12-23 11:02:39_466] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:user.name=killy.zhang
[2015-12-23 11:02:39_466] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:user.home=C:\Users\killy.zhang
[2015-12-23 11:02:39_467] INFO org.apache.zookeeper.ZooKeeper [main.logEnv:100] Client environment:user.dir=E:\JavaTestSpace\HuiFuExamPro
[2015-12-23 11:02:39_470] INFO org.apache.zookeeper.ZooKeeper [main.:438] Initiating client connection, connectString=192.168.25.247:2181 sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@1f449a28
[2015-12-23 11:02:39_526] INFO org.apache.zookeeper.ClientCnxn [main-SendThread(192.168.25.247:2181).logStartConnect:1032] Opening socket connection to server 192.168.25.247/192.168.25.247:2181. Will not attempt to authenticate using SASL (unknown error)
[2015-12-23 11:02:39_530] INFO org.apache.zookeeper.ClientCnxn [main-SendThread(192.168.25.247:2181).primeConnection:876] Socket connection established to 192.168.25.247/192.168.25.247:2181, initiating session
[2015-12-23 11:02:39_564] INFO org.apache.zookeeper.ClientCnxn [main-SendThread(192.168.25.247:2181).onConnected:1299] Session establishment complete on server 192.168.25.247/192.168.25.247:2181, sessionid = 0x151c4d1663c0037, negotiated timeout = 20000
[2015-12-23 11:02:39_568] INFO org.apache.curator.framework.state.ConnectionStateManager [main-EventThread.postState:194] State change: CONNECTED
log4j:WARN No appenders could be found for logger (com.wandoulabs.jodis.RoundRobinJedisPool).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" redis.clients.jedis.exceptions.JedisException: Proxy list empty
at com.wandoulabs.jodis.RoundRobinJedisPool.getResource(RoundRobinJedisPool.java:286)
at com.huifu.killy.codis.CodisExam.main(CodisExam.java:21)
Unable to get lock channel for E:\Users\psteitz\pool\tags\POOL_2_4_RC1\target\cobertura\cobertura.ser.lock: \Users\psteitz\pool\tags\POOL_2_4_RC1\target\cobertura\cobertura.ser.lock (系统找不到指定的路径。)
lock file could not be deleted

yangzhe1991 · 2015-12-23T14:00:43Z

proxy正确启动了吗？/zk/codis/db_test/proxy这个节点下有child吗？

superkilly · 2015-12-24T01:45:15Z

回复yangzhe1991：下面是有child的
[zk: 192.168.25.247:2181(CONNECTED) 1] ls /zk/codis/db_test
[fence, slots, servers, proxy, migrate_tasks, dashboard, LOCK, actions, ActionResponse]
[zk: 192.168.25.247:2181(CONNECTED) 2] rmr /zk/codis/db_test/dashboard
[zk: 192.168.25.247:2181(CONNECTED) 3] ls /zk/codis/db_test/proxy
[proxy_1]
我的情况是这样，jodis（Java代码）是在本地，而zookeeper、codis以及redis都是在一台服务器上，两者之间的网络是OK的。

hellomsg · 2016-03-10T03:04:51Z

你好，我也出现这个问题。修改Jodis后用jdk1.6编译的，报同样的错误。情况跟 @superkilly 相同。

Ericj100 · 2016-03-12T09:25:58Z

对于这个问题，目前我们也碰到了
环境：jodis 0.3 ， codis3.0.2
根据源代码跟踪测试，发现jodis中RoundRobinJedisPool初始化时，执行resetPools方法中，需要从Zookeeper中获取proxy字节，然后利用json转换为CodisProxyInfo对象，然而问题就在这。
CodisProxyInfo需要获取2个数据：1.proxy是否在线状态，2.proxy的ip和端口号。
获取1状态时，发现Zookeeper中根本没有此状态。Zookeeper中存储数据结构如下：

{
    "id": 1,
    "token": "5061161396d59489c34c6ec31506f94e",
    "start_time": "2016-03-12 14:58:18.198193631 +0800 CST",
    "admin_addr": "172.16.54.135:11080",
    "proto_type": "tcp4",
    "proxy_addr": "172.16.54.135:19000",
    "product_name": "demo",
    "pid": 23002,
    "pwd": "/usr/local/gopath/src/github.com/CodisLabs/codis",
    "sys": "Linux dev-1 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux"
}

导致jodis判断proxy状态是否为在线状态时，为false，不再进行初始化操作。
获取ip和端口号是，CodisProxyInfo的字段名为addr，与Zookeeper中的proxy_addr名称不一致，导致json序列化无法初始化addr字段。
以上2个问题，导致了Proxy list is empty问题。
第二个问题，可以通过在CodisProxyInfo加上注解解决：

@JsonProperty("proxy_addr")
    private String addr;

第一个问题，等待专家解决，目前看到dashboard可以通过http api获取状态。

Apache9 · 2016-03-12T09:50:22Z

你这个json文件的内容是proxy目录下的么？感觉这个是给dashboard看的？
@spinlock 你确认一下？

spinlock · 2016-03-12T11:09:28Z

codis 2.x 里面 proxy 注册的地址是 /zk/codis/db_xxx/proxy/yyy，jodis 监视的目录也是 /zk/codis/db_xxx/proxy/yyy，来获取 proxy 列表。

codis 3.x 里面，proxy 注册目录换成了 /codis3/xxx/proxy/yyy，同时也会写入 /zk/codis/db_xxx/proxy/yyy 以兼容 jodis。

其中 xxx 是 product name，yyy 是 proxy 名称。所以先确认一下 zk 下对应目录是不是有正确的数据？

Ericj100 · 2016-03-12T11:44:24Z

[zk: localhost:2181(CONNECTED) 0] get /codis3/codis-demo/proxy/proxy-878c84e8cb91a032733ced45fff90fa7
{
    "id": 1,
    "token": "878c84e8cb91a032733ced45fff90fa7",
    "start_time": "2016-03-12 10:28:44.195026896 +0800 CST",
    "admin_addr": "192.168.92.147:11080",
    "proto_type": "tcp4",
    "proxy_addr": "192.168.92.147:19000",
    "product_name": "codis-demo",
    "pid": 1450,
    "pwd": "/usr/local/gopath/src/github.com/CodisLabs/codis",
    "sys": "Linux localhost.localdomain 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux"
}
cZxid = 0x1000000017
ctime = Sat Mar 12 18:29:19 CST 2016
mZxid = 0x1000000018
mtime = Sat Mar 12 18:29:19 CST 2016
pZxid = 0x1000000017
cversion = 0
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 480
numChildren = 0

从Zookeeper中获取的proxy信息，完整如上，这个的确是Zookeeper中的数据

jodis可以从 /zk/codis/db_xxx/proxy/yyy中获取，这个还没有尝试过，我们都是从/codis3/xxx/proxy/yyy中获取的，回头我尝试下。
由于安装在公司服务器上，所以我周一上班时再测试从 /zk/codis/db_xxx/proxy/yyy中获取

Thank you for your reply!

spinlock · 2016-03-13T01:03:29Z

@Ericj100 对 codis 3.x 而言，jodis 是可选参数，必须启动的时候指定才行。如果找不到 /zk/codis/db_xxx 的话应该是 proxy 启动参数不对造成的哦。

Ericj100 · 2016-03-14T01:19:49Z

hi,everyone
经过spinlock和Apache9建议，进行2点修改
1.修改codis3的proxy.toml文件

# Set jodis address & session timeout.
jodis_addr = "172.16.54.135:2181"
jodis_timeout = 10

jodis_addr必须设置
2.修改jodis的zkProxyDir
zkProxyDir("/zk/codis/db_xxx/proxy")
测试通过。
结论：
1.codis 3.x 里面，proxy 注册目录换成了 /codis3/xxx/proxy/yyy，同时也会写入 /zk/codis/db_xxx/proxy/yyy 以兼容 jodis。
Zookeeper在 /zk/codis/db_xxx/proxy下存储的消息如下，满足jodis获取CodisProxyInfo消息体

[zk: localhost:2181(CONNECTED) 9] get /zk/codis/db_demo/proxy/c60ed6906ff3c080bc4b1a78676660c5
{
    "addr": "172.16.54.135:19000",
    "start_at": "2016-03-14 09:09:30.151749896 +0800 CST",
    "state": "online",
    "token": "c60ed6906ff3c080bc4b1a78676660c5"
}

2.codis的proxy需要指定jodis

Thank you for your reply and help!

yangyuxia · 2017-04-12T09:42:16Z

codis3中proxy的状态记录在路径/jodis/{PRODUCT_NAME}中。注意路径中最后不带"/"符号。
修改jodis的zkProxyDir: zkProxyDir("/jodis/{PRODUCT_NAME}")

CrazySuper · 2018-08-15T06:54:27Z

楼主还在吗

zzqw · 2020-06-17T10:00:22Z

codis3中proxy的状态记录在路径/jodis/{PRODUCT_NAME}中。注意路径中最后不带"/"符号。
修改jodis的zkProxyDir: zkProxyDir("/jodis/{PRODUCT_NAME}")

亲测有效。。。上面的解决办法都不对的。。。竟然在closed下面找到真正的解决办法

superkilly closed this as completed Dec 24, 2015

superkilly reopened this Dec 24, 2015

yangzhe1991 closed this as completed Apr 25, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proxy list is empty #10

Proxy list is empty #10

superkilly commented Dec 23, 2015

yangzhe1991 commented Dec 23, 2015

superkilly commented Dec 24, 2015

hellomsg commented Mar 10, 2016

Ericj100 commented Mar 12, 2016

Apache9 commented Mar 12, 2016

spinlock commented Mar 12, 2016

Ericj100 commented Mar 12, 2016

spinlock commented Mar 13, 2016

Ericj100 commented Mar 14, 2016

yangyuxia commented Apr 12, 2017

CrazySuper commented Aug 15, 2018

zzqw commented Jun 17, 2020

Proxy list is empty #10

Proxy list is empty #10

Comments

superkilly commented Dec 23, 2015

yangzhe1991 commented Dec 23, 2015

superkilly commented Dec 24, 2015

hellomsg commented Mar 10, 2016

Ericj100 commented Mar 12, 2016

Apache9 commented Mar 12, 2016

spinlock commented Mar 12, 2016

Ericj100 commented Mar 12, 2016

spinlock commented Mar 13, 2016

Ericj100 commented Mar 14, 2016

yangyuxia commented Apr 12, 2017

CrazySuper commented Aug 15, 2018

zzqw commented Jun 17, 2020