java.net.SocketException: Connection reset

[hksinsinwar]

Hi,
Connection to arangoDB is getting reset frequently.
My environment is --
ArangoDB : 3.3.14
Java Driver : 5.0.0.RC1

Received following exception
`Caused by: java.util.concurrent.ExecutionException: com.arangodb.ArangoDBException: java.net.SocketException: Connection reset
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_111-internal]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_111-internal]
at com.arangodb.internal.velocystream.internal.VstConnectionSync.write(VstConnectionSync.java:99) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
... 26 common frames omitted
Caused by: com.arangodb.ArangoDBException: java.net.SocketException: Connection reset
at com.arangodb.internal.velocystream.internal.MessageStore.get(MessageStore.java:73) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at com.arangodb.internal.velocystream.internal.VstConnectionSync$1.call(VstConnectionSync.java:93) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at com.arangodb.internal.velocystream.internal.VstConnectionSync$1.call(VstConnectionSync.java:90) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111-internal]
at com.arangodb.internal.velocystream.internal.MessageStore.clear(MessageStore.java:96) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:150) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:124) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
... 1 common frames omitted
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209) ~[na:1.8.0_111-internal]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_111-internal]
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_111-internal]
at sun.security.ssl.InputRecord.read(InputRecord.java:503) ~[na:1.8.0_111-internal]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[na:1.8.0_111-internal]
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) ~[na:1.8.0_111-internal]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_111-internal]
at com.arangodb.internal.velocystream.internal.VstConnection.readBytesIntoBuffer(VstConnection.java:260) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at com.arangodb.internal.velocystream.internal.VstConnection.readBytes(VstConnection.java:254) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at com.arangodb.internal.velocystream.internal.VstConnection.readChunk(VstConnection.java:231) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
at com.arangodb.internal.velocystream.internal.VstConnection$1.call(VstConnection.java:141) ~[arangodb-java-driver-5.0.0-RC.1.jar!/:na]
... 5 common frames omitted

`

[mvollmary]

Hi @hksinsinwar,

Which server setup do you have? (Single server or cluster)

Do you see any ERROR/FATAL log messages in the server logs? (if you have a cluster the coordinator logs are important)

Which options do you set in ArangoDB.Builder? (e.g. loadBalancingStrategy)

How frequently do you see that behavior?

[philschaller]

We ran into the same problem with almost the same environment:
ArangoDB 3.3.14
Java Driver: 5.0.1

It always happened at the first request after a long idle time, maybe an hour. The subsequent requests where successful. Changing the protocol to HTTP_VPACK seems to help, at least for us.

[mvollmary]

As far as I remember @jsteemann told me about a bug in 3.3.14 which could cause to this problem. I recommend you to update ArangoDB

[jsteemann]

@mpv1989 : we recently released 3.3.17, which fixes a known issue related to SSL/TLS errors when using VST over an SSL connection.
For 3.3.17, we are not aware of any other known issues with protocol type VST, but we did not fix anything inactivity-related. This particular issue looks like a hangup from either side (client application or ArangoDB server) after a long time of connection inactivity. I don't remember a dedicated fix for this, so it may be still relevant.
Anyway, if you could try 3.3.17 to rule out the already fixed known issue that would be great. Thanks!

[zqhxuyuan]

Hi @hksinsinwar,

Which server setup do you have? (Single server or cluster)

Do you see any ERROR/FATAL log messages in the server logs? (if you have a cluster the coordinator logs are important)

Which options do you set in ArangoDB.Builder? (e.g. loadBalancingStrategy)

How frequently do you see that behavior?

we also has this problem when stress testing on 3.3.17. We notice coordinator is down and restart because supervisor(2018-10-16T11:37:33). but there're no ERROR log. our log level is INFO.

2018-10-15T14:13:52+08:00 |INFO| using `arangosh --server.endpoint tcp://172.16.48.178:8529`. component=arangodb
2018-10-16T11:37:33+08:00 |INFO| coordinator has terminated component=arangodb

[crash-g]

Hi, we encountered a similar problem too. Actually, in our case the issue can be split in two:

1. As @zqhxuyuan, we often get a connection reset.
2. After the first failed request, all subsequent requests will fail.

As for 1, we could not assess why this happens. We suspect it may be related to the fact that we use Docker (https://forums.docker.com/t/running-a-simple-client-server-program-in-docker/7321), but it is also somehow influenced by the database and driver versions (using ArangoDB 3.3.16 and driver 5.0.1, it used to happen after a long period of inactivity, but after upgrading it happens randomly during normal use).

As for 2, it may be related to how the message store used by VstConnection is cleared in case of errors. In our scenario, readChunk(); throws a SocketException (Connection reset). Then the catch block executes messageStore.clear(e); which throws an exception too, and the connection is not closed. Subsequent requests will try to use the same connection and get the same SocketException (Connection reset). Could this be the root of the issue?
Thanks!

Our environment is --
ArangoDB : 3.4.1
Java Driver : 5.0.3
Protocol: VST
Connection pool size: 1
ConnectionTtl: 5 minutes

[bischoje]

We seem to be having the same issue as originally reported, though we are using ArangoDB version 3.3.23-1 and the stack trace is slightly different. The symptom is that after a long period of inactivity (let's say an hour), any attempt to utilize the connections (by calling ArangoDatabase::query) results in "Connection reset" error messages in multiple threads in the connection pool. Those threads are never heard from again, apparently being replaced with new threads with live connections. After enough threads have errored out and died, further ArangoDB access is successful without errors... until the next long period of inactivity.

Here is an example pair of related stack traces:

repository-6885744d46-qcxmr: 2020-01-24 19:39:11,797 ERROR [pool-58-thread-1] [MessageStore]    : Connection reset
repository-6885744d46-qcxmr: java.net.SocketException: Connection reset
repository-6885744d46-qcxmr:    at java.base/java.net.SocketInputStream.read(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.net.SocketInputStream.read(Unknown Source)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection.readBytesIntoBuffer(Connection.java:266)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection.readBytes(Connection.java:260)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection.readChunk(Connection.java:237)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection$1.call(Connection.java:148)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection$1.call(Connection.java:137)
repository-6885744d46-qcxmr:    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.lang.Thread.run(Unknown Source)
repository-6885744d46-qcxmr: 2020-01-24 19:39:12,186 ERROR [pool-59-thread-1] [MessageStore]    : Connection reset
repository-6885744d46-qcxmr: java.net.SocketException: Connection reset
repository-6885744d46-qcxmr:    at java.base/java.net.SocketInputStream.read(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.net.SocketInputStream.read(Unknown Source)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection.readBytesIntoBuffer(Connection.java:266)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection.readBytes(Connection.java:260)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection.readChunk(Connection.java:237)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection$1.call(Connection.java:148)
repository-6885744d46-qcxmr:    at com.arangodb.internal.velocystream.internal.Connection$1.call(Connection.java:137)
repository-6885744d46-qcxmr:    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
repository-6885744d46-qcxmr:    at java.base/java.lang.Thread.run(Unknown Source)

I will try switching from VST to HTTP_VPACK, as one of the previous comments mentioned and see if that helps. I'll report back.

[rashtao]

Using Http connection could solve the problem, since it is a stateless connection. Nevertheless I would not expect this problem to happen in VST either, since the driver sets:

arangodb-java-driver/src/main/java/com/arangodb/internal/velocystream/internal/VstConnection.java

Line 110 in d00ddb1

socket.setKeepAlive(true);

Can you please provide more details to reproduce the problem? eg. db version, db deploment topology, driver version, driver configuration, java version...

[rashtao]

closing due to inactivity

@hksinsinwar
feel free to reopen in case of further problems