Too many "ECONNRESET" error

[metalshan]

neo4j-javascript-driver - 1.0.1

I am using neo4j with enterprise-3.0.1 edition with your nodejs driver and getting too many (over 90%) ECONNRESET errors from the driver. As the same queries are working perfectly fine if used in neo4j browser client (which uses http protocol), I am expecting the problem is in the driver and hence besides mailing the issue to neo4j support I'm also adding the same here.

[oskarhane]

Thanks for the report @metalshan.
What version of node are you using?
Could you show a code example that have this behavior?

[metalshan]

Thanks for the quick response @oskarhane

I am using node 6.2.1

And about code, the code is simple. Below is the snippet.
The node part

doDatabaseOperation(query, params, operation) {
    if(!operation){
        operation = "operation";
    }
    return new Promise((resolve, reject) => {
        var neo4j = require('neo4j-driver').v1;
        var driver = neo4j.driver(dbPath, neo4j.auth.basic(username, password));
        var session = driver.session();
        session.run(query).then(result=>{
            // I am handling my success here
            resolve(result);
        }, err=>{
            //The ECONNRESET error is being caught here
            reject(err);
        });
    }); 
}

cypher query example

MATCH p = (topic:TOPIC)-[:SUBTOPICOF*0..]->(tBase:TOPIC) WHERE tBase.id in [\'t00000000000000002\']
MATCH (topic) <-[:BELONGSTO]- (n:LINK) 

WITH DISTINCT n, p ORDER BY n.addedOn desc SKIP 0 LIMIT 30
MATCH (site:SITE) <-[:BELONGSTO]- (n)
OPTIONAL MATCH (topics:TOPIC) <-[topicRelationalObj:BELONGSTO]- (n)
OPTIONAL MATCH (bands:BAND) <-[:BELONGSTO]- (n)
RETURN DISTINCT n, site, topicRelationalObj, topics, bands

[oskarhane]

Thanks,
Do you call doDatabaseOperation for every query you run (I'm reading the code like that but just want to confirm)?

It's enough that you:

var neo4j = require('neo4j-driver').v1;
var driver = neo4j.driver(dbPath, neo4j.auth.basic(username, password));

one time and the just create sessions from that.
Also, you should close the session when done.

I would write your code like this:

var neo4j = require('neo4j-driver').v1;
var driver = neo4j.driver(dbPath, neo4j.auth.basic(username, password));

doDatabaseOperation(query, params, operation) {
    if(!operation){
        operation = "operation";
    }
    return new Promise((resolve, reject) => {
        var session = driver.session();
        session.run(query)
          .then(result => {
            // I am handling my success here
            session.close();
            resolve(result);
          }).catch(err => {
            //The ECONNRESET error is being caught here
            reject(err);
          });
    }); 
}

Could you try that and see if the behavior changes?
Also, try to use the js driver 1.0.3 if possible since it's the latest on the 1.0.X branch.

[metalshan]

Yes, every neo4j query is executed by that function.
I've changed the the code to declare neo4j and driver once (as you said) and also upgraded the driver version to 1.0.3, but still the issue exists :(

[oskarhane]

Is it the same for any query you run or just the example you showed?
Could you try to test with really simple queries to see if this is a driver or Bolt issue?

Do you get the same behavior when connecting to any neo4j database or just the one your seeing this on?
If possible it would be helpful if you could setup a new neo4j instance with the same or parts of the same dataset to rule out any issues being with this specific instance.

[metalshan]

For any query I am facing the issue. With simple queries the failure rates are may be less, but they are there.

Yesterday night I modified the doDatabaseOperation() function to use http endpoints of neo, rather than bolt driver and everything is working pretty good. No failures. So the issue is probably with the driver only.

Another thing is since last 1 months the driver was also working fine for same queries. The only difference is, number of data in the database were less. Right now I have 500+ nodes and I'm facing this issue.

I restarted and also reinstalled the neo4j containers, but the issue remains with the driver.

[arbythree]

Having the same problem with this code:

router.get('/', function(req, res, next) {
  var driver  = neo4j.driver("bolt://ip.address", neo4j.auth.basic("neo4j", "neo4j"));
  var session = driver.session();
  var query   = "MATCH (p:Person {name:'Sally'})-[r:KNOWS]->(f:Person) RETURN f";

  session
    .run(query)
    .then(function(result) {
      result.records.forEach(function(record) {
        console.log(record._fields);
      });
      session.close();
    })
    .catch(function(error) {
      console.log(error);
    });

  driver.close();
});

I've tried local and remote servers, http and bolt, and have confirmed connectivity in golang.

[oskarhane]

@raybradley Your issue is that you're potentially closing the driver before the query is finished.
Do you also do this @metalshan?

router.get('/', function(req, res, next) {
  var driver  = neo4j.driver("bolt://ip.address", neo4j.auth.basic("neo4j", "neo4j"));
  var session = driver.session();
  var query   = "MATCH (p:Person {name:'Sally'})-[r:KNOWS]->(f:Person) RETURN f";

  session
    .run(query)
    .then(function(result) {
      result.records.forEach(function(record) {
        console.log(record._fields);
      });
      session.close();
      driver.close();
    })
    .catch(function(error) {
      console.log(error);
      driver.close();
    });
});

@raybradley your code should look something like the above.

[arbythree]

That was it, @oskarhane. Thank you!

[metalshan]

Hi,
Sorry for the late response.
No I am not closing the session or driver at all.
After reading your previous comments I thought is it because I was creating sessions again and again and never closed them? I thought I will test it by closing the sessions in the callback, but it seems in my local machine it's working with the previous code. However I haven't test in the server where the error were actually causing. Will try to test in the server again with previous code + session.close() and let you know in few days.

[oskarhane]

Since we never heard back from @metalshan, I'm closing this issue.
Please open it again if the issue still exists.

[metalshan]

Probably it was an environmental issue... I removed every single thing and set up the server again. Haven't faced this issue after that.

[stormit-vn]

Hi all,

We also encounter this issue, here is the stack trace of the error. Hopefully, someone can take a look and figure out the root cause

Neo4jError: read ECONNRESET\n\n at captureStacktrace (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/result.js:200:15)\n at new Result (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/result.js:73:19)\n at _newRunResult (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/transaction.js:348:10)\n at Object.run (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/transaction.js:259:14)\n at Transaction.run (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/transaction.js:103:26)\n at Function.runInTransaction (/var/neo4j-sample-app/node_modules/ogmneo/lib/ogmneo-operation-executer.js:154:32)\n at session.readTransaction (/var/neo4j-sample-app/node_modules/ogmneo/lib/ogmneo-operation-executer.js:129:40)\n at TransactionExecutor._executeTransactionInsidePromise (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/internal/transaction-executor.js:115:37)\n at /var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/internal/transaction-executor.js:67:15\n at new Promise ()\n at new F (/var/neo4j-sample-app/node_modules/core-js/library/modules/_export.js:36:28)\n at TransactionExecutor.execute (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/internal/transaction-executor.js:66:14)\n at Session._runTransaction (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/session.js:214:40)\n at Session.readTransaction (/var/neo4j-sample-app/node_modules/neo4j-driver/lib/v1/session.js:187:19)\n at Promise (/var/neo4j-sample-app/node_modules/ogmneo/lib/ogmneo-operation-executer.js:128:25)\n at new Promise ()\n at Function._executeRead (/var/neo4j-sample-app/node_modules/ogmneo/lib/ogmneo-operation-executer.js:122:16)\n at Function.execute (/var/neo4j-sample-app/node_modules/ogmneo/lib/ogmneo-operation-executer.js:110:29)\n at OGMNeoModel.findOne (/var/neo4j-sample-app/src/api/ogm-neo/ogmneo-model.js:399:44)\n at OAuthAccessTokenManager.findOne (/var/neo4j-sample-app/src/api/managers/BaseManager.js:242:27)\n at onJwtBearerStrategyAuth (/var/neo4j-sample-app/src/config/passport.js:102:52)\n at /var/neo4j-sample-app/node_modules/passport-http-jwt-bearer/lib/strategy.js:99:11

This is running on the environment below:
NodeJS v8.9.3
Neo4J v3.4.0
neo4j-driver v1.6.2
The app is running on a Docker container with Centos 7

[stormit-vn]

We cannot go-live without getting this issue fixed. So does anyone has a resolution can fix this issue?

[lutovich]

Hi @stormit-vn,

ECONNRESET means the server side abruptly closed the connection. It can be either neo4j database, container or something else in the middle. Neo4j database should log when it kills connections. Logs should be available in the neo4j.log & debug.log. Do you know if container can go to sleep after some inactivity and terminate TCP connections as the result of this?

Please consider configuring maxConnectionLifetime to a lower value than the default 1 hour. See more details here.

Hope this helps.

[stormit-vn]

@lutovich Thank you for your suggestion. I have applied above configuration and this issue has no longer occurred. I am still following our application, hopefully, it will work

Here is the configuration object that I am using

{
                encrypted: false,
                // The maximum total number of connections allowed to be managed by the connection pool, per host.
                // This includes both in-use and idle connections. No maximum connection pool size is imposed
                // by default.
                maxConnectionPoolSize: 100,
                // The maximum allowed lifetime for a pooled connection in milliseconds. Pooled connections older than this
                // threshold will be closed and removed from the pool. Such discarding happens during connection acquisition
                // so that new session is never backed by an old connection. Setting this option to a low value will cause
                // a high connection churn and might result in a performance hit. It is recommended to set maximum lifetime
                // to a slightly smaller value than the one configured in network equipment (load balancer, proxy, firewall,
                // etc. can also limit maximum connection lifetime). No maximum lifetime limit is imposed by default. Zero
                // and negative values result in lifetime not being checked.
                maxConnectionLifetime: 0,
                // The maximum amount of time to wait to acquire a connection from the pool (to either create a new
                // connection or borrow an existing one.
                connectionAcquisitionTimeout: 60000,

                // Specify the maximum time in milliseconds transactions are allowed to retry via
                // <code>Session#readTransaction()</code> and <code>Session#writeTransaction()</code> functions.
                // These functions will retry the given unit of work on `ServiceUnavailable`, `SessionExpired` and transient
                // errors with exponential back-off using initial delay of 1 second.
                // Default value is 30000 which is 30 seconds.
                maxTransactionRetryTime: 60000,

                // Provide an alternative load balancing strategy for the routing driver to use.
                // Driver uses "least_connected" by default.
                // <b>Note:</b> We are experimenting with different strategies. This could be removed in the next minor
                // version.
                // loadBalancingStrategy: 'least_connected' | 'round_robin',

                // Specify socket connection timeout in milliseconds. Numeric values are expected. Negative and zero values
                // result in no timeout being applied. Connection establishment will be then bound by the timeout configured
                // on the operating system level. Default value is 5000, which is 5 seconds.
                connectionTimeout: 30000
            }