Deadlock when multiple connections make simultaneous requests

[flyingrobots]

NOTE I'm not sure if this is a bug in OrientDB or if it has anything to do with OrientJS so I've created this issue in both repositories. Please see orientechnologies/orientdb#4625 for the OrientDB issue

We are running OrientDB 2.1-rc4 (this has also attempted with 2.1-rc5) as a distributed database and our driver is NodeJS pacakge orientjs (latest version from npm OR from github at commit 5b0035aec7fdcab75ff0e1492117ad7d881dd434).

If more than one connection attempts to query the database at once, the database appears to become deadlocked. No responses are sent to the clients and furthermore we must stop and terminate the server process manually - in other words, the java process becomes unresponsive and doesn't shut down when it receives SIGTERM.

We are running the server process on OSX (and we've tried this on linux as well) with Java 8 and are using the following distributed-db-config.json:

{
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "undefined",
  "readQuorum": 1,
  "writeQuorum": 1,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "servers": {
    "*": "master"
  },
  "clusters": {
    "internal": {
    },
    "index": {
    },
    "*": {
      "servers": ["<NEW_NODE>"]
    }
  }
}

We've noticed that if the connections do NOT query the database simultaneously, then there is no problem. The deadlock only appears to occur if more than one client issues a request to the database at the same time. We've also noted that this problem appears to have been introduced in 2.1-rc4, it did not exist in 2.1-rc3.

Here is an example script which you can use to reproduce the issue yourself:

#!/usr/bin/env node
'use strict';

var orientjs = require('orientjs');
var Promise = require('bluebird');

// NOTE this works successfully only if the connection count is 1.
var CONNECTION_COUNT = 2;

class Connection {
    constructor() {
        // NOTE change these settings to match your own.
        this.server = orientjs({
            host: 'localhost',
            port: 2424,
            username: 'root',
            password: 'password'
        });

        // NOTE change these settings to match your own.
        this.db = this.server.use({
            name: 'GratefulDeadConcerts',
            username: 'admin',
            password: 'admin'
        });
    }
}

var connections = [];
var promises = [];

// Create some number of connections...
for(var i = 0; i < CONNECTION_COUNT; i++) {
    connections.push(new Connection());
    console.info(`Created connection: ${i}`);
}

// For each connection, query the number of vertices in the database.
// Collect the Promises as we go...
connections.forEach(function(connection) {
    promises.push(
        connection.db.select('count(*)').from('V').scalar()
        // NOTE if CONNECTION_COUNT > 1, none of the following happens:
        .then(function(count) {
            console.info(`count: ${count}`);
        })
        .catch(function(err) {
            console.error(err);
        })
    );
});

// Wait for all Promises to resolve and exit.
Promise.all(promises)
.finally(function(err) {
    // NOTE this never happens!
    console.info('All OK!');
    process.exit(0);
});

[flyingrobots]

This issue was on OrientDB's end - closing as it has been fixed.