Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gremlin(g.V().count())执行报错 #102

Closed
yuyang0 opened this issue Oct 11, 2018 · 6 comments

Comments

@yuyang0
Copy link

@yuyang0 yuyang0 commented Oct 11, 2018

Expected behavior 期望表现

g.V().count() 这样一个很简单的统计

Actual behavior 实际表现

2018-10-10 21:28:14 78398733 [gremlin-server-worker-1] [WARN ] org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler [] - Invalid request - responding with 500 Internal Server Error and Error encountered evaluating script: g.V().count()
org.apache.tinkerpop.gremlin.process.traversal.util.TraversalInterruptedException: null
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:142) ~[gremlin-core-3.2.5.jar:3.2.5]
at
org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.hasNext(ExpandableStepIterator.java:42) ~[gremlin-core-3.2.5.jar:3.2.5]
at
org.apache.tinkerpop.gremlin.process.traversal.step.util.ReducingBarrierStep.processAllStarts(ReducingBarrierStep.java:83) ~[gremlin-core-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.process.traversal.step.util.ReducingBarrierStep.processNextStart(ReducingBarrierStep.java:113) ~[gremlin-core-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143) ~[gremlin-core-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192) ~[gremlin-core-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.fill(IteratorUtils.java:62) ~[gremlin-core-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.list(IteratorUtils.java:85) ~[gremlin-core-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.asList(IteratorUtils.java:382) ~[gremlin-core-3.2.5.jar:3.2.5]
at
org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:239) ~[gremlin-server-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36) ~[gremlin-core-3.2.5.jar:3.2.5]
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:320) ~[gremlin-groovy-3.2.5.jar:3.2.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_151]
at com.baidu.hugegraph.auth.HugeGraphAuthProxy$ContextTask.run(HugeGraphAuthProxy.java:290) ~[hugegraph-api-0.7.4.jar:0.27.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Vertex/Edge summary 数据量

数据量大概1000w节点

Specifications of environment 环境信息

  • hugegraph version: {like v0.7.4}
  • operating system: {ubuntu 16.04, 32 CPUs, 128G RAM}
  • hugegraph backend: {rocksdb}
@yuyang0 yuyang0 changed the title gremlin执行报错 gremlin(g.V().count())执行报错 Oct 11, 2018
@javeme

This comment has been minimized.

Copy link
Contributor

@javeme javeme commented Oct 11, 2018

这个错误是因为gremlin语句执行30s超时导致的,count()是属于全表遍历性质的操作(与mysql等传统数据库不一样的地方),对于这类耗时操作建议使用异步gremlin执行(OLAP),详细用法请参考gremlin-schedule

@javeme

This comment has been minimized.

Copy link
Contributor

@javeme javeme commented Oct 11, 2018

类似的表现:#37

@yuyang0

This comment has been minimized.

Copy link
Author

@yuyang0 yuyang0 commented Oct 12, 2018

@javeme 这样直接500,并且抛出异常不太好吧,如果有明确的日志那就完美了

@javeme

This comment has been minimized.

Copy link
Contributor

@javeme javeme commented Oct 15, 2018

@yuyang0 感谢你反馈的建议。确实我们第一次遇到这个异常也很疑惑,但是由于tinkerpop框架中gremlin-server设计的这个流程,所以在hugegraph层面无法修改(可以参考Java Thread Interrupt)。我们后续会尝试往tinkerpop社区提交我们的修改,期待能有一个更加易于理解的错误信息。谢谢。

@yuyang0

This comment has been minimized.

Copy link
Author

@yuyang0 yuyang0 commented Oct 15, 2018

@javeme 希望hugegraph越做越好,我打算小范围上线hugegraph,现在唯一的顾虑就是不能部署多台服务器来扩展处理能力,这以后肯定会遇到瓶颈

@yuyang0 yuyang0 closed this Oct 18, 2018
@javeme

This comment has been minimized.

Copy link
Contributor

@javeme javeme commented Oct 18, 2018

@yuyang0 谢谢。针对你说到的顾虑,提供一些我们的想法供参考:
hugegraph是支持存储的扩展(支持分布式后端存储),我猜你是指目前hugegraph只支持单服务无法扩展计算能力。计算的话主要分2部分:

  1. OLTP部分:目前遇到的场景来看,在线及时分析这一块(包括索引及ID查询、多度查询、最短路径等),一般单台hugegraph服务基本能cover十亿级别的数据插入及查询分析,瓶颈基本在后端存储上(比如20台Cassandra集群)。
  2. OLAP部分:对于长时间的离线任务(如pagerank、louvain),单机的计算能力确实显得非常不足,这部分目前我们内部是依赖Spark集群来完成部分工作的,当然将来会通过分布式的hugegraph服务来完成OLAP(预计明年会支持)。

另外,hugegraph服务HA预计今年底会支持。感谢你的反馈。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.