Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

优化代码的properties载入，增加新的全局ID生成算法，改变Curator的Maven依赖 #850

Merged
merged 2 commits into from May 4, 2016
Merged

优化代码的properties载入，增加新的全局ID生成算法，改变Curator的Maven依赖 #850

merged 2 commits into from May 4, 2016

Conversation

HashZhang
Copy link
Contributor

  1. 优化代码的properties载入，增加载入工具类PropertiesUtil
  2. 增加新的全局ID生成算法：基于ZK与本地配置的分布式ID生成器(可以通过ZK获取集群（机房）唯一InstanceID，也可以通过配置文件配置InstanceID)
    ID结构：long 64位，ID最大可占63位
    |current time millis(微秒时间戳41位)|clusterId（机房或者ZKid，通过配置文件配置4位）|instanceId（实例ID，可以通过ZK或者配置文件获取，4位）|threadId（线程ID，7位）|increment(自增,7位)
    一共63位，可以承受单机房单机器单线程1000*(2^7)=1280000的并发。
    无悲观锁，无强竞争，吞吐量更高
    1. 改变Curator的Maven依赖，修改为内置最新稳定版本zookeeper3.4.8的2.10.0. 增加Curator-recipe的依赖

@HashZhang
Merge pull request #4 from MyCATApache/1.6
bf3093c 
merger
@HashZhang
1. 优化代码的properties载入，增加载入工具类PropertiesUtil
a828bdd 
2. 增加新的quanjuID生成算法：基于ZK与本地配置的分布式ID生成器(可以通过ZK获取集群（机房）唯一InstanceID，也可以通过配置文件配置InstanceID)
 ID结构：long 64位，ID最大可占63位
 |current time millis(微秒时间戳41位)|clusterId（机房或者ZKid，通过配置文件配置4位）|instanceId（实例ID，可以通过ZK或者配置文件获取，4位）|threadId（线程ID，7位）|increment(自增,7位)
  一共63位，可以承受单机房单机器单线程1000*(2^7)=1280000的并发。
  无悲观锁，无强竞争，吞吐量更高
  3. 改变Curator的Maven依赖，修改为内置最新稳定版本zookeeper3.4.8的2.10.0. 增加Curator-recipe的依赖
@magicdoom
Copy link
Contributor

这个生成的id是否是局部有序的？

@magicdoom magicdoom merged commit cefaca3 into MyCATApache:1.6 May 4, 2016
@magicdoom
Copy link
Contributor

加下志愿者群 478119364 加好后跟我说下 另外你这个单元测试失败 你再测下

@magicdoom
Copy link
Contributor

单元测试貌似用到了zk 最好自己test代码里自己启个zk 不然别人会报错 自动化编译也过不去

@HashZhang
Copy link
Contributor Author

不是自增的。我已在测试中自己启动了个zk，怎么会测试不通过呢？

@magicdoom
Copy link
Contributor

测试现在通过了 但是执行单元测试的时候会报以下异常：
2016-05-06 19:02:27.164 WARN Curator-LeaderSelector-0 - Unexpected thread interruption!
2016-05-06 19:02:27.167 ERROR Curator-LeaderSelector-0 - The leader threw an exception
java.lang.InterruptedException
at java.lang.Object.wait(Native Method) ~[?:1.7.0_25]
at java.lang.Object.wait(Object.java:503) ~[?:1.7.0_25]
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:871) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:243) ~[curator-framework-2.10.0.jar:?]
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:237) ~[curator-framework-2.10.0.jar:?]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108) ~[curator-client-2.10.0.jar:?]
at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:234) ~[curator-framework-2.10.0.jar:?]
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:218) ~[curator-framework-2.10.0.jar:?]
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:42) ~[curator-framework-2.10.0.jar:?]
at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:339) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:123) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:425) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:441) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) ~[curator-recipes-2.10.0.jar:?]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[?:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[?:1.7.0_25]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[?:1.7.0_25]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[?:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[?:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) [?:1.7.0_25]
�ϵ�һ��leader�ڵ��
0 used to be leader

2016-05-06 19:02:27.172 WARN Curator-LeaderSelector-1 - Unexpected thread interruption!
2016-05-06 19:02:27.172 ERROR Curator-LeaderSelector-1 - The leader threw an exception
java.lang.IllegalStateException: instance must be started before calling this method
at com.google.common.base.Preconditions.checkState(Preconditions.java:174) ~[guava-19.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.delete(CuratorFrameworkImpl.java:359) ~[curator-framework-2.10.0.jar:?]
at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:339) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:123) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:425) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:441) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) ~[curator-recipes-2.10.0.jar:?]
at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) ~[curator-recipes-2.10.0.jar:?]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[?:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[?:1.7.0_25]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[?:1.7.0_25]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[?:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[?:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) [?:1.7.0_25]
�ϵ�����leader�ڵ��

2016-05-06 19:02:27.319 INFO main - total resouces of dataHost localhost1 is :1
2016-05-06 19:02:28.048 WARN main-SendThread(127.0.0.1:54838) - Session 0x15485bb7e980000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.7.0_25]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708) ~[?:1.7.0_25]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
160000
Time elapsed:0.209s
ID/s:765550.2392344498
2016-05-06 19:02:29.373 INFO main - total resouces of dataHost localhost1 is :1
2016-05-06 19:02:30.849 WARN main-SendThread(127.0.0.1:54838) - Session 0x15485bb7e980000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.7.0_25]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708) ~[?:1.7.0_25]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.991 sec

@HashZhang
Copy link
Contributor Author

这个是会报的，因为中途我做了client异常断链。

@magicdoom
Copy link
Contributor

这个你要不在捕捉异常的地方 Assert 不要抛出来

@HashZhang
Copy link
Contributor Author

我刚申请加群

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@HashZhang @magicdoom