Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'Failed to do request' error when loading #309

Closed
suesunss opened this issue Dec 30, 2018 · 2 comments

Comments

@suesunss
Copy link

@suesunss suesunss commented Dec 30, 2018

Expected behavior 期望表现

{type something here...}

用HugeGraphLoader导入数据进HugeGraph的时候,有时会出现如下exception:

2018-12-30 04:28:40 119108 [pool-3-thread-20] [ERROR] com.baidu.hugegraph.loader.task.TaskManager [] - Vertex insert error
com.baidu.hugegraph.rest.ClientException: Failed to do request
	at com.baidu.hugegraph.rest.RestClient.request(RestClient.java:69) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	at com.baidu.hugegraph.rest.RestClient.post(RestClient.java:115) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	at com.baidu.hugegraph.rest.RestClient.post(RestClient.java:79) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	at com.baidu.hugegraph.rest.RestClient.post(RestClient.java:74) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	at com.baidu.hugegraph.api.graph.VertexAPI.create(VertexAPI.java:48) ~[hugegraph-client-1.6.4.jar:1.6.4.0]
	at com.baidu.hugegraph.driver.GraphManager.addVertex(GraphManager.java:56) ~[hugegraph-client-1.6.4.jar:1.6.4.0]
	at com.baidu.hugegraph.loader.task.TaskManager.lambda$submitVerticesInSingleMode$0(TaskManager.java:206) ~[hugegraph-loader-0.8.0.jar:?]
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [guava-19.0.jar:?]
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [guava-19.0.jar:?]
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [guava-19.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: javax.ws.rs.ProcessingException: java.net.ConnectException: Cannot assign requested address (connect failed)
	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:228) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:437) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:343) ~[jersey-client-2.25.1.jar:?]
	at com.baidu.hugegraph.rest.RestClient.lambda$post$0(RestClient.java:116) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	at com.baidu.hugegraph.rest.RestClient.request(RestClient.java:67) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	... 12 more
Caused by: java.net.ConnectException: Cannot assign requested address (connect failed)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_144]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_144]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_144]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:1.8.0_144]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_144]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_144]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:1.8.0_144]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:1.8.0_144]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ~[?:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ~[?:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ~[?:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ~[?:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) ~[?:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) ~[?:1.8.0_144]
	at org.glassfish.jersey.client.internal.HttpUrlConnector$4.getOutputStream(HttpUrlConnector.java:390) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:200) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:194) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:228) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[jersey-common-2.25.1.jar:?]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2033) ~[jackson-core-2.8.4.jar:2.8.4]
	at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1045) ~[jackson-core-2.8.4.jar:2.8.4]
	at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:942) ~[jackson-databind-2.8.4.jar:2.8.4]
	at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:618) ~[jackson-jaxrs-base-2.8.4.jar:2.8.4]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.spi.ContentEncoder.aroundWriteTo(ContentEncoder.java:138) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.client.ClientRequest.doWriteEntity(ClientRequest.java:517) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:499) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:393) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:285) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:228) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444) ~[jersey-common-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:437) ~[jersey-client-2.25.1.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:343) ~[jersey-client-2.25.1.jar:?]
	at com.baidu.hugegraph.rest.RestClient.lambda$post$0(RestClient.java:116) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	at com.baidu.hugegraph.rest.RestClient.request(RestClient.java:67) ~[hugegraph-common-1.5.3.jar:1.5.3.0]
	... 12 more

个人感觉是HugeGraphServer端无法及时响应Loader的请求造成失败,请问有哪些方法可以尝试解决这个问题?

我个人能想到的方法:

  1. 增加loader的超时时间阈值
  2. 增加每次load请求的batch size
  3. 增加server端的硬件资源(如核心数)
  4. 分布式的方式去部署server(但这个目前的HugeGraph不支持)

谢谢。

Actual behavior 实际表现

{type something here...}

Steps to reproduce the problem 复现步骤

  1. {step 1}
  2. {step 2}
  3. {step 3}

Status of loaded data 数据状态

Vertex/Edge summary 数据量

  • loaded vertices amount: {like 10 million}
  • loaded edges amount: {like 20 million}
  • loaded time: {like 200s}

点的数据量在一千万左右,边的数据量在百万级每次。

Vertex/Edge example 数据示例

{type something here...}

Schema(VertexLabel, EdgeLabel, IndexLabel) 元数据结构

{type something here...}

Specifications of environment 环境信息

  • hugegraph version: {like v0.7.4}
  • operating system: {like centos 7.4, 32 CPUs, 64G RAM}
  • hugegraph backend: {like cassandra 3.10, cluster with 20 nodes, 3 x 1TB HDD disk each node}
@zhoney

This comment has been minimized.

Copy link
Contributor

@zhoney zhoney commented Jan 2, 2019

@suesunss 感谢您的反馈。

这个问题推断是由于短时间内大量建立tcp连接导致的。可以尝试开启Linux内核参数:

net.ipv4.tcp_tw_reuse = 1 #表示重用TIME-WAIT socket
net.ipv4.tcp_tw_recycle = 1 #表示回收TIME-WAIT socket

在/proc/sys/net/ipv4/ 目录下

后续我们会尝试使用长连接进行优化,避免该问题的发生。@Linary

@suesunss

This comment has been minimized.

Copy link
Author

@suesunss suesunss commented Jan 7, 2019

@zhoney thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.