New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem while executing queries against distributed database with 2.1.7 version #5470

Closed
sandhya-inbetween opened this Issue Dec 15, 2015 · 4 comments

Comments

Projects
None yet
3 participants
@sandhya-inbetween

sandhya-inbetween commented Dec 15, 2015

Hi, we are trying to setup orientdb cluster with (3 nodes) running on Docker containers , on three different VM(multi-host orientdb cluster with docker, with one container per VM).

Orientdb Version: 2.1.7
Docker container OS: ubuntu 14.04

default-distributed-db-config.json

{
"autoDeploy": true,
"hotAlignment": true,
"executionMode": "asynchronous",
"readQuorum": 1,
"writeQuorum": 2,
"failureAvailableNodesLessQuorum": false,
"readYourWrites":true,
"servers": {
"": "master"
},
"clusters": {
"internal": {
},
"index": {
},
"
": {
"servers": ["<NEW_NODE>"]
}
}
}

Load Balancer

We are using HAPROXY as a load balancer, where we have configured the three orientdb nodes.

We get the following error while executing queries against database.
Is our load balancing/clustering technique the right way ? Or do we need to change anything?

As per documentation provided on orientdb site we need to set property of ODatabaseDocumentTx object:
http://orientdb.com/docs/2.1/Distributed-Configuration.html#load-balancing

currently in our code we have not done this configuration.
Few lines from our code are as below:
[
ODatabaseDocumentTx db = null;
OrientGraph graphDb = null;
db = getProfiledDatabaseInstance(iRequest);
db.activateOnCurrentThread();
graphDb = new OrientGraph(db);
]
But as per documentation ,we tried with setting property for db object but it is not supported in latest release.

Error stacktrace:
[


Node:1


2015-12-15 10:46:16:785 INFO [node1450176340446]--[node1450176348454] update configuration db=Test1 [OHazelcastPlugin]
2015-12-15 10:46:49:578 WARNI [node1450176340446] timeout (10000ms) on waiting for synchronous responses from nodes=[node1450176335911] responsesSoFar=[] request=id=34 from=node1450176340446 task=command_sql(create class V cluster 14) [OHazelcastDistributedDatabase]
2015-12-15 10:46:49:579 WARNI [node1450176340446] detected 1 node(s) in timeout or in conflict and quorum (1) has not been reached, rolling back changes for request (id=34 from=node1450176340446 task=command_sql(create class V cluster 14)) [ODistributedResponseManager]
2015-12-15 10:46:49:580 WARNI [node1450176340446] Quorum 1 not reached for request (id=34 from=node1450176340446 task=command_sql(create class V cluster 14)). Elapsed=10009ms No server in conflict. Received: {node1450176335911=waiting-for-response} [ODistributedResponseManager]com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing distributed request (id=34 from=node1450176340446 task=command_sql(create class V cluster 14)) against database 'Test1.[]' to nodes [node1450176335911]
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:189)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:360)
at com.orientechnologies.orient.server.distributed.ODistributedStorage.command(ODistributedStorage.java:333)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:67)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.doCreateClass(OSchemaShared.java:1031)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:415)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:400)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:275)
at com.orientechnologies.orient.core.metadata.schema.OSchemaProxy.createClass(OSchemaProxy.java:71)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.checkForGraphSchema(OrientBaseGraph.java:386)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:139)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:61)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:46)
at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:216)
at com.cs.usecase.managetag.GetAllTags.execute(GetAllTags.java:39)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:180)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:627)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Quorum 1 not reached for request (id=34 from=node1450176340446 task=command_sql(create class V cluster 14)). Elapsed=10009ms No server in conflict. Received: {node1450176335911=waiting-for-response}
at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.manageConflicts(ODistributedResponseManager.java:570)
at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.getFinalResponse(ODistributedResponseManager.java:339)
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.waitForResponse(OHazelcastDistributedDatabase.java:423)
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:186)
... 17 more


Node:2


2015-12-15 10:46:16:800 INFO [node1450176335911] Saving distributed configuration file for database 'Test1' to: /orientdb/databases/Test1/distributed-config.json [OHazelcastPlugin]
2015-12-15 10:46:16:802 INFO [node1450176335911]<-[node1450176340446] received updated status node1450176340446.Test1=ONLINE [OHazelcastPlugin]
2015-12-15 10:46:49:599 WARNI [node1450176335911] timeout (10002ms) on waiting for synchronous responses from nodes=[node1450176340446] responsesSoFar=[] request=id=33 from=node1450176335911 task=command_sql(create class V cluster 14) [OHazelcastDistributedDatabase]
2015-12-15 10:46:49:602 WARNI [node1450176335911] detected 1 node(s) in timeout or in conflict and quorum (1) has not been reached, rolling back changes for request (id=33 from=node1450176335911 task=command_sql(create class V cluster 14)) [ODistributedResponseManager]
2015-12-15 10:46:49:602 WARNI [node1450176335911] Quorum 1 not reached for request (id=33 from=node1450176335911 task=command_sql(create class V cluster 14)). Elapsed=10012ms No server in conflict. Received: {node1450176340446=waiting-for-response} [ODistributedResponseManager]com.orientechnologies.orient.server.distributed.ODistributedException: Error on executing distributed request (id=33 from=node1450176335911 task=command_sql(create class V cluster 14)) against database 'Test1.[]' to nodes [node1450176340446]
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:189)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.sendRequest(OHazelcastPlugin.java:360)
at com.orientechnologies.orient.server.distributed.ODistributedStorage.command(ODistributedStorage.java:333)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:67)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.doCreateClass(OSchemaShared.java:1031)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:415)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:400)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:275)
at com.orientechnologies.orient.core.metadata.schema.OSchemaProxy.createClass(OSchemaProxy.java:71)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.checkForGraphSchema(OrientBaseGraph.java:386)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:139)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:61)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:46)
at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:216)
at com.cs.usecase.manageattribute.GetAllAttributes.execute(GetAllAttributes.java:39)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:180)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:627)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Quorum 1 not reached for request (id=33 from=node1450176335911 task=command_sql(create class V cluster 14)). Elapsed=10012ms No server in conflict. Received: {node1450176340446=waiting-for-response}
at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.manageConflicts(ODistributedResponseManager.java:570)
at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.getFinalResponse(ODistributedResponseManager.java:339)
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.waitForResponse(OHazelcastDistributedDatabase.java:423)
at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.send2Nodes(OHazelcastDistributedDatabase.java:186)
... 17 more

Node 3:

2015-12-15 10:46:22:250 INFO [node1450176348454] Saving distributed configuration file for database 'Test1' to: /orientdb/databases/Test1/distributed-config.json [OHazelcastPlugin]com.orientechnologies.orient.server.distributed.ODistributedException: Error on execution distributed COMMAND
at com.orientechnologies.orient.server.distributed.ODistributedStorage.command(ODistributedStorage.java:345)
at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1334)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClusters(OSchemaShared.java:1132)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:419)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:400)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:275)
at com.orientechnologies.orient.core.metadata.schema.OSchemaProxy.createClass(OSchemaProxy.java:71)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.checkForGraphSchema(OrientBaseGraph.java:386)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:139)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:61)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.(OrientTransactionalGraph.java:46)
at com.tinkerpop.blueprints.impls.orient.OrientGraph.(OrientGraph.java:216)
at com.cs.usecase.manageklass.GetAllKlasses.execute(GetAllKlasses.java:35)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:180)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:627)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)
Caused by: com.orientechnologies.orient.core.sql.OCommandSQLParsingException: Error on parsing command at position #0: Cluster 'v' already exists
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLCreateCluster.execute(OCommandExecutorSQLCreateCluster.java:95)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:90)
at com.orientechnologies.orient.server.distributed.task.OSQLCommandTask.execute(OSQLCommandTask.java:116)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:810)
at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:299)
at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:119)

When we try to add any new record we got following stacktrace in any one of the node out of three:

2015-12-15 10:51:08:246 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 0/10... [ODistributedStorage]
2015-12-15 10:51:08:741 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 1/10... [ODistributedStorage]
2015-12-15 10:51:09:264 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 2/10... [ODistributedStorage]
2015-12-15 10:51:09:792 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 3/10... [ODistributedStorage]
2015-12-15 10:51:10:339 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 4/10... [ODistributedStorage]
2015-12-15 10:51:10:845 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 5/10... [ODistributedStorage]
2015-12-15 10:51:11:383 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 6/10... [ODistributedStorage]
2015-12-15 10:51:11:883 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 7/10... [ODistributedStorage]
2015-12-15 10:51:12:390 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 8/10... [ODistributedStorage]
2015-12-15 10:51:12:899 WARNI Error on creating cluster on distributed nodes: ids are different (local=13 and remote=25). Retrying 9/10... [ODistributedStorage]com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1364)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClusters(OSchemaShared.java:1132)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:419)
at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.createClass(OSchemaShared.java:400)
at com.orientechnologies.orient.core.metadata.schema.OSchemaProxy.createClass(OSchemaProxy.java:100)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph$7.call(OrientBaseGraph.java:1433)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph$7.call(OrientBaseGraph.java:1430)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.executeOutsideTx(OrientBaseGraph.java:1782)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.createEdgeType(OrientBaseGraph.java:1430)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.createEdgeType(OrientBaseGraph.java:1414)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.createEdgeType(OrientBaseGraph.java:1399)
at com.cs.usecase.managetag.TagUtil.getTagEdgeSuperType(TagUtil.java:29)
at com.cs.usecase.managetag.SaveTag.createTag(SaveTag.java:95)
at com.cs.usecase.managetag.SaveTag.execute(SaveTag.java:54)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:180)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:627)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

]

*hazelcast.conf *

[



orientdb
orientdb


2434


235.1.1.1
2434


192.168.134.81:2434
192.168.134.79:2434
192.168.132.97:2434
192.168.132.161:2434


192.168.134.79


16


]

@sandhya-inbetween sandhya-inbetween changed the title from Problem while executing queries against distributed database with 3 nodes to Problem while executing queries against distributed database with 2.1.7 version Dec 15, 2015

@luigidellaquila

This comment has been minimized.

Show comment
Hide comment
@luigidellaquila

luigidellaquila Dec 15, 2015

Member

Hi @sandhya-inbetween

I see you have hotAlignment=true in your configuration. It has some known issues, probably the problem is related to that.

Please set it to false and restart all the cluster. Let me know if it fixes the problem.

Thanks

Luigi

Member

luigidellaquila commented Dec 15, 2015

Hi @sandhya-inbetween

I see you have hotAlignment=true in your configuration. It has some known issues, probably the problem is related to that.

Please set it to false and restart all the cluster. Let me know if it fixes the problem.

Thanks

Luigi

@luigidellaquila luigidellaquila self-assigned this Dec 15, 2015

@sandhya-inbetween

This comment has been minimized.

Show comment
Hide comment
@sandhya-inbetween

sandhya-inbetween Dec 15, 2015

hi Luigi ,

Even after keeping "hotAlignment=false" I get same error.

And one more error occurs , in one of the nodes, when i start all three nodes and try to create DB. But this error does not seem to block further operations on the DB.

Error Stacktrace:

[
2015-12-15 15:12:19:277 INFO  [node1450192266989] class 'OFunction', creation of new local cluster 'ofunction_node1450192266989' (id=-1) [OHazelcastPlugin]
2015-12-15 15:12:19:522 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 0/10... [ODistributedStorage]
2015-12-15 15:12:20:253 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 1/10... [ODistributedStorage]
2015-12-15 15:12:20:978 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 2/10... [ODistributedStorage]
2015-12-15 15:12:21:663 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 3/10... [ODistributedStorage]
2015-12-15 15:12:22:373 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 4/10... [ODistributedStorage]
2015-12-15 15:12:23:076 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 5/10... [ODistributedStorage]
2015-12-15 15:12:23:764 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 6/10... [ODistributedStorage]
2015-12-15 15:12:24:502 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 7/10... [ODistributedStorage]
2015-12-15 15:12:25:215 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 8/10... [ODistributedStorage]
2015-12-15 15:12:25:943 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 9/10... [ODistributedStorage]
2015-12-15 15:12:26:548 SEVER [node1450192266989] error on creating cluster 'ofunction_node1450192266989' in class 'OFunction':  [OHazelcastPlugin][192.168.132.84]:2434 [orientdb] [3.5.3] hz._hzInstance_1_orientdb.event-4 caught an exception while processing task:com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher@8515e7e
com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'ofunction_node1450192266989' in class 'OFunction'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:241)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1124)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.requestDatabase(OHazelcastPlugin.java:964)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:901)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1461)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.checkDatabaseEvent(OHazelcastPlugin.java:1271)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.entryAdded(OHazelcastPlugin.java:677)
        at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:63)
        at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:51)
        at com.hazelcast.map.impl.MapEventPublishingService.callListener(MapEventPublishingService.java:90)
        at com.hazelcast.map.impl.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:102)
        at com.hazelcast.map.impl.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:46)
        at com.hazelcast.map.impl.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:33)
        at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:91)
        at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:61)
        at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:63)
        at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)
        at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'ofunction_node1450192266989' in class 'OFunction'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1624)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1293)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1127)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1124)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:239)
        ... 17 more
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
        at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1364)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2102)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1068)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1617)
        ... 21 more

2015-12-15 15:12:26:560 INFO  [node1450192266989]--[node1450192284546] update configuration db=Test1 [OHazelcastPlugin]
2015-12-15 15:12:26:584 INFO  [node1450192266989]--[node1450192284546] update configuration db=Test1 [OHazelcastPlugin]
2015-12-15 15:12:26:590 WARNI [node1450192266989] updated distributed configuration for database: Test1:

{
  "version": 7,
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "asynchronous",
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "servers": {
    "*": "master"
  },
  "clusters": {
    "internal": {
    },
    "index": {
    },
    "*": {
      "servers": ["node1450192252787","node1450192284546","<NEW_NODE>"]
    },
    "ofunction_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "oschedule_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "orole_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "orids_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "ouser_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    }
  }
}

] 

sandhya-inbetween commented Dec 15, 2015

hi Luigi ,

Even after keeping "hotAlignment=false" I get same error.

And one more error occurs , in one of the nodes, when i start all three nodes and try to create DB. But this error does not seem to block further operations on the DB.

Error Stacktrace:

[
2015-12-15 15:12:19:277 INFO  [node1450192266989] class 'OFunction', creation of new local cluster 'ofunction_node1450192266989' (id=-1) [OHazelcastPlugin]
2015-12-15 15:12:19:522 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 0/10... [ODistributedStorage]
2015-12-15 15:12:20:253 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 1/10... [ODistributedStorage]
2015-12-15 15:12:20:978 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 2/10... [ODistributedStorage]
2015-12-15 15:12:21:663 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 3/10... [ODistributedStorage]
2015-12-15 15:12:22:373 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 4/10... [ODistributedStorage]
2015-12-15 15:12:23:076 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 5/10... [ODistributedStorage]
2015-12-15 15:12:23:764 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 6/10... [ODistributedStorage]
2015-12-15 15:12:24:502 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 7/10... [ODistributedStorage]
2015-12-15 15:12:25:215 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 8/10... [ODistributedStorage]
2015-12-15 15:12:25:943 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=14). Retrying 9/10... [ODistributedStorage]
2015-12-15 15:12:26:548 SEVER [node1450192266989] error on creating cluster 'ofunction_node1450192266989' in class 'OFunction':  [OHazelcastPlugin][192.168.132.84]:2434 [orientdb] [3.5.3] hz._hzInstance_1_orientdb.event-4 caught an exception while processing task:com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher@8515e7e
com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'ofunction_node1450192266989' in class 'OFunction'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:241)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1124)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.requestDatabase(OHazelcastPlugin.java:964)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:901)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1461)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.checkDatabaseEvent(OHazelcastPlugin.java:1271)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.entryAdded(OHazelcastPlugin.java:677)
        at com.hazelcast.map.impl.MapListenerAdaptors$1$1.onEvent(MapListenerAdaptors.java:63)
        at com.hazelcast.map.impl.InternalMapListenerAdapter.onEvent(InternalMapListenerAdapter.java:51)
        at com.hazelcast.map.impl.MapEventPublishingService.callListener(MapEventPublishingService.java:90)
        at com.hazelcast.map.impl.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:102)
        at com.hazelcast.map.impl.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:46)
        at com.hazelcast.map.impl.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:33)
        at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:91)
        at com.hazelcast.map.impl.MapService.dispatchEvent(MapService.java:61)
        at com.hazelcast.spi.impl.eventservice.impl.LocalEventDispatcher.run(LocalEventDispatcher.java:63)
        at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:190)
        at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:174)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'ofunction_node1450192266989' in class 'OFunction'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1624)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1293)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1127)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1124)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:239)
        ... 17 more
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
        at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1364)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2102)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1068)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1617)
        ... 21 more

2015-12-15 15:12:26:560 INFO  [node1450192266989]--[node1450192284546] update configuration db=Test1 [OHazelcastPlugin]
2015-12-15 15:12:26:584 INFO  [node1450192266989]--[node1450192284546] update configuration db=Test1 [OHazelcastPlugin]
2015-12-15 15:12:26:590 WARNI [node1450192266989] updated distributed configuration for database: Test1:

{
  "version": 7,
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "asynchronous",
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "servers": {
    "*": "master"
  },
  "clusters": {
    "internal": {
    },
    "index": {
    },
    "*": {
      "servers": ["node1450192252787","node1450192284546","<NEW_NODE>"]
    },
    "ofunction_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "oschedule_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "orole_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "orids_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    },
    "ouser_node1450192284546": {
      "servers": ["node1450192284546","node1450192252787","<NEW_NODE>"]
    }
  }
}

] 
@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Dec 16, 2015

Member

Can you try by creating the database before to go in cluster?

Member

lvca commented Dec 16, 2015

Can you try by creating the database before to go in cluster?

@lvca lvca added the bug label Jan 12, 2016

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Feb 2, 2016

Member

HAProxy cannot be used with binary protocol. Is this the case? In v2.2 we supported load balancing at client side, so it's configurable and transparent.

Member

lvca commented Feb 2, 2016

HAProxy cannot be used with binary protocol. Is this the case? In v2.2 we supported load balancing at client side, so it's configurable and transparent.

@lvca lvca closed this Feb 2, 2016

@lvca lvca added question and removed bug labels Feb 2, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment