New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error on creating cluster on distributed nodes: ids are different #4767

Closed
ashokrenukappa opened this Issue Aug 10, 2015 · 34 comments

Comments

Projects
None yet
@ashokrenukappa

ashokrenukappa commented Aug 10, 2015

Am using Orient DB 2.1.0 and running in clustered environment. I tried with 2.0.13, same issue.
Am trying to setup 3 Orient DB servers and create database in one of them

hazelcast.xml looks like this:

<hazelcast xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.h
azelcast.com/schema/config hazelcast-config-3.0.xsd">
  <group>
    <name>orient-db</name>
    <password>cluster-db</password>
  </group>
  <network>
    <port auto-increment="false">2434</port>
    <join>
      <multicast enabled="false">
        <multicast-group>235.1.1.1</multicast-group>
        <multicast-port>2434</multicast-port>
      </multicast>
      <tcp-ip enabled="true">
          <!-- Host names of 3 OrientDB servers and port number -->
          <member>orientdb1-prod-dev-brn:2434</member>
          <member>orientdb2-prod-dev-brn:2434</member>
          <member>orientdb3-prod-dev-brn:2434</member>       
      </tcp-ip>
    </join>
    <symmetric-encryption enabled="true">
      <algorithm>PBEWithMD5AndDES</algorithm>
      <!-- salt value to use when generating the secret key -->
      <salt>orientdb-cluster-salt</salt>
      <!-- pass phrase to use when generating the secret key -->
      <password>orientdb-cluster-pp</password>
      <!-- iteration count to use when generating the secret key -->
      <iteration-count>19</iteration-count>
    </symmetric-encryption>
  </network>
  <executor-service>
    <pool-size>16</pool-size>
  </executor-service>
</hazelcast>

Handler OHazelcastPlugin in orientdb-server-config.xml looks like this

        <!-- CLUSTER PLUGIN, TO TURN ON SET THE 'ENABLED' PARAMETER TO 'true' -->
        <handler class="com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin">
            <parameters>
                <parameter name="nodeName" value="orientdb3-prod-dev-brn" />
                <parameter name="enabled" value="true"/>
                <parameter name="configuration.db.default"
                           value="${ORIENTDB_HOME}/config/default-distributed-db-config.json"/>
                <parameter name="configuration.hazelcast" value="${ORIENTDB_HOME}/config/hazelcast.xml"/>
            </parameters>
        </handler>

default-distributed-db-config.json looks like this

{
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "asynchronous",
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "servers": {
        "*": "master"
  },
  "clusters": {
    "internal": {},
    "index": {},
    "*": {
      "servers": [
        "<NEW_NODE>"
      ]
    }
  }
}

ERROR in logs

2015-08-10 21:03:58:208 INFO  [product-orientdb3-prod-dev-brn]--[product-orientdb2-prod-dev-brn] update configuration db=product [OHazelcastPlugin]
2015-08-10 21:04:06:407 INFO  [product-orientdb3-prod-dev-brn] class 'someEdgeFrom', creation of new local cluster 'someedgefrom_product-orientdb3-prod-dev-brn' (id=-1) [OHazelcastPlugin]
2015-08-10 21:04:09:992 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 0/10... [ODistributedStorage]
2015-08-10 21:04:16:949 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 1/10... [ODistributedStorage]
2015-08-10 21:04:20:619 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 2/10... [ODistributedStorage]
2015-08-10 21:04:22:369 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 3/10... [ODistributedStorage]
2015-08-10 21:04:24:767 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 4/10... [ODistributedStorage]
2015-08-10 21:04:26:191 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 5/10... [ODistributedStorage]
2015-08-10 21:04:27:593 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 6/10... [ODistributedStorage]
2015-08-10 21:04:28:956 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 7/10... [ODistributedStorage]
2015-08-10 21:04:30:331 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 8/10... [ODistributedStorage]
2015-08-10 21:04:31:699 WARNI Error on creating cluster on distributed nodes: ids are different (local=114 and remote=170). Retrying 9/10... [ODistributedStorage]
2015-08-10 21:04:32:799 SEVER [product-orientdb3-prod-dev-brn] error on creating cluster 'someedgefrom_product-orientdb3-prod-dev-brn' in class 'someEdgeFrom':  [OHazelcastPlugin]Exception in thread "main" com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'someedgefrom_product-orientdb3-prod-dev-brn' in class 'someEdgeFrom'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:224)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1055)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:937)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1350)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:184)
        at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:971)
        at com.orientechnologies.orient.server.OServer.activate(OServer.java:338)
        at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'someedgefrom_product-orientdb3-prod-dev-brn' in class 'someEdgeFrom'
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1494)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1207)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$1.call(OHazelcastPlugin.java:1058)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$1.call(OHazelcastPlugin.java:1055)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:222)
        ... 7 more
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
        at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1268)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1253)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2124)
        at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1105)
        at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1487)
        ... 11 more

@lvca lvca added the waiting reply label Aug 11, 2015

@ashokrenukappa

This comment has been minimized.

Show comment
Hide comment
@ashokrenukappa

ashokrenukappa Aug 11, 2015

I used the snapshot and again Orient DB 1 and 2 works fine 3rd one has problem.

Logs, does not move after it "sent request"

2015-08-11 15:52:54:402 INFO  [product-orientdb3-prod-dev-brn] found no previous messages in queue orientdb.node.product-orientdb3-prod-dev-brn.response [OHazelcastDistributedMessageService]
2015-08-11 15:52:54:463 INFO  [product-orientdb3-prod-dev-brn] loaded database configuration from active cluster [OHazelcastPlugin]
2015-08-11 15:52:54:666 WARNI [product-orientdb3-prod-dev-brn] updated distributed configuration for database: product:
----------
{^M
  "version": 0,^M
  "autoDeploy": true,^M
  "hotAlignment": true,^M
  "executionMode": "asynchronous",^M
  "readQuorum": 1,^M
  "writeQuorum": 2,^M
  "failureAvailableNodesLessQuorum": false,^M
  "readYourWrites": true,"servers":{^M
    "*": "master"^M
    },"clusters":{^M
    "internal": {^M
},^M
    "index": {^M
},^M
    "*": {^M
  "servers": ["product-orientdb1-prod-dev-brn","product-orientdb2-prod-dev-brn","product-orientdb3-prod-dev-brn","<NEW_NODE>"]^M
},^M
    "seenat_product-orientdb2-prod-dev-brn": {^M
  "servers": ["product-orientdb2-prod-dev-brn","product-orientdb1-prod-dev-brn","product-orientdb3-prod-dev-brn","<NEW_NODE>"]^M
},^M
    "ouser_product-orientdb2-prod-dev-brn": {^M
  "servers": ["product-orientdb2-prod-dev-brn","product-orientdb1-prod-dev-brn","product-orientdb3-prod-dev-brn","<NEW_NODE>"]^M
},^M
########## Many lines regarding schema classes

---------- [OHazelcastPlugin]

2015-08-11 15:53:00:681 WARNI [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] requesting deploy of database 'product' on local server... [OHazelcastPlugin]
2015-08-11 15:53:00:717 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sending request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]
2015-08-11 15:53:00:730 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sent request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]

When it try to access the studio, i get following error in logs

2015-08-11 15:53:00:717 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sending request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]
2015-08-11 15:53:00:730 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sent request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]
2015-08-11 16:00:16:824 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52177. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
2015-08-11 16:00:16:926 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52178. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
2015-08-11 16:00:17:414 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52179. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
2015-08-11 16:00:17:532 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52180. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]

But if I wait for both 1/2 servers to come online then start the 3rd one. It seems to work. But in a clustered environment it is possible for servers to go up and down any time.

ashokrenukappa commented Aug 11, 2015

I used the snapshot and again Orient DB 1 and 2 works fine 3rd one has problem.

Logs, does not move after it "sent request"

2015-08-11 15:52:54:402 INFO  [product-orientdb3-prod-dev-brn] found no previous messages in queue orientdb.node.product-orientdb3-prod-dev-brn.response [OHazelcastDistributedMessageService]
2015-08-11 15:52:54:463 INFO  [product-orientdb3-prod-dev-brn] loaded database configuration from active cluster [OHazelcastPlugin]
2015-08-11 15:52:54:666 WARNI [product-orientdb3-prod-dev-brn] updated distributed configuration for database: product:
----------
{^M
  "version": 0,^M
  "autoDeploy": true,^M
  "hotAlignment": true,^M
  "executionMode": "asynchronous",^M
  "readQuorum": 1,^M
  "writeQuorum": 2,^M
  "failureAvailableNodesLessQuorum": false,^M
  "readYourWrites": true,"servers":{^M
    "*": "master"^M
    },"clusters":{^M
    "internal": {^M
},^M
    "index": {^M
},^M
    "*": {^M
  "servers": ["product-orientdb1-prod-dev-brn","product-orientdb2-prod-dev-brn","product-orientdb3-prod-dev-brn","<NEW_NODE>"]^M
},^M
    "seenat_product-orientdb2-prod-dev-brn": {^M
  "servers": ["product-orientdb2-prod-dev-brn","product-orientdb1-prod-dev-brn","product-orientdb3-prod-dev-brn","<NEW_NODE>"]^M
},^M
    "ouser_product-orientdb2-prod-dev-brn": {^M
  "servers": ["product-orientdb2-prod-dev-brn","product-orientdb1-prod-dev-brn","product-orientdb3-prod-dev-brn","<NEW_NODE>"]^M
},^M
########## Many lines regarding schema classes

---------- [OHazelcastPlugin]

2015-08-11 15:53:00:681 WARNI [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] requesting deploy of database 'product' on local server... [OHazelcastPlugin]
2015-08-11 15:53:00:717 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sending request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]
2015-08-11 15:53:00:730 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sent request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]

When it try to access the studio, i get following error in logs

2015-08-11 15:53:00:717 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sending request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]
2015-08-11 15:53:00:730 FINE  [product-orientdb3-prod-dev-brn]->[[product-orientdb1-prod-dev-brn]] sent request id=4 from=product-orientdb3-prod-dev-brn task=deploy_db [OHazelcastDistributedDatabase]
2015-08-11 16:00:16:824 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52177. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
2015-08-11 16:00:16:926 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52178. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
2015-08-11 16:00:17:414 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52179. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]
2015-08-11 16:00:17:532 WARNI Distributed server is not yet ONLINE (status=STARTING), reject incoming connection from /10.88.31.131:52180. If you are trying to shutdown the server, please kill the process [OServerNetworkListener]

But if I wait for both 1/2 servers to come online then start the 3rd one. It seems to work. But in a clustered environment it is possible for servers to go up and down any time.

@prjhub prjhub removed the waiting reply label Aug 11, 2015

@ashokrenukappa

This comment has been minimized.

Show comment
Hide comment
@ashokrenukappa

ashokrenukappa Aug 13, 2015

@lvca Do you have any update on this ?

ashokrenukappa commented Aug 13, 2015

@lvca Do you have any update on this ?

@fishtek

This comment has been minimized.

Show comment
Hide comment
@fishtek

fishtek Aug 27, 2015

👍
We have also run into this during our evaluation of OrientDB!

fishtek commented Aug 27, 2015

👍
We have also run into this during our evaluation of OrientDB!

@lvca lvca added the bug label Sep 21, 2015

@lvca lvca added this to the 2.1.x (next hotfix) milestone Sep 21, 2015

@lvca lvca self-assigned this Sep 21, 2015

@boardthatpowder

This comment has been minimized.

Show comment
Hide comment
@boardthatpowder

boardthatpowder Nov 4, 2015

+1 same occurring with 2.1.4

Steps to reproduce:

  • Start first instance of orientdb in distributed mode. Wait for it to finish
  • Start second instance of orientdb in distributed mode, wait for it to finish.
  • In Studio for the first instance, create a new database, and add a record.

Logs from server1:


2015-11-04 01:38:50:638 INFO  [node1446601006190] Saving distributed configuration file for database 'deansdb' to: /opt/orientdb/databases/deansdb/distributed-config.json [OHazelcastPlugin]
2015-11-04 01:38:50:649 INFO  [node1446601006190] received updated status node1446601038783.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:50:651 INFO  [node1446601006190] received updated status node1446601006190.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:50:652 WARNI [node1446601006190]->[node1446601038783] deploying database deansdb... [OSyncDatabaseTask]
2015-11-04 01:38:50:652 INFO  [node1446601006190]->[node1446601038783] creating backup of database 'deansdb' in directory: /tmp/orientdb/backup_deansdb.zip... [OSyncDatabaseTask]
2015-11-04 01:38:50:743 INFO  - Compressing file ouser.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:750 INFO  ok size=65.00KB compressedSize=132 ratio=100% elapsed=8ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:750 INFO  - Compressing file ORole.name.sbt... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:751 INFO  ok size=65.00KB compressedSize=178 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:754 INFO  - Compressing file dictionary.sbt... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:755 INFO  ok size=65.00KB compressedSize=129 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:756 INFO  - Compressing file default.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:763 INFO  ok size=65.00KB compressedSize=120 ratio=100% elapsed=6ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:763 INFO  - Compressing file index.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:765 INFO  ok size=65.00KB compressedSize=136 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:765 INFO  - Compressing file deansdb.wmr... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:768 INFO  ok size=40b compressedSize=19 ratio=53% elapsed=3ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:768 INFO  - Compressing file manindex.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:776 INFO  ok size=129.00KB compressedSize=245 ratio=100% elapsed=8ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:777 INFO  - Compressing file ofunction.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:778 INFO  ok size=65.00KB compressedSize=123 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:778 INFO  - Compressing file distributed-config.json... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:779 INFO  ok size=405b compressedSize=221 ratio=46% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:779 INFO  - Compressing file manindex.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:780 INFO  ok size=65.00KB compressedSize=120 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:783 INFO  - Compressing file orole.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:785 INFO  ok size=129.00KB compressedSize=581 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:787 INFO  - Compressing file index.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:789 INFO  ok size=129.00KB compressedSize=275 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:789 INFO  - Compressing file database.ocf... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:791 INFO  ok size=1.62KB compressedSize=272 ratio=84% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:795 INFO  - Compressing file oschedule.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:796 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:796 INFO  - Compressing file name_id_map.cm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:797 INFO  ok size=1.10KB compressedSize=270 ratio=76% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:798 INFO  - Compressing file v.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:799 INFO  ok size=65.00KB compressedSize=122 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:800 INFO  - Compressing file ouser.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:802 INFO  ok size=129.00KB compressedSize=547 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:803 INFO  - Compressing file internal.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:806 INFO  ok size=129.00KB compressedSize=8388 ratio=94% elapsed=4ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:811 INFO  - Compressing file OUser.name.sbt... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:814 INFO  ok size=65.00KB compressedSize=179 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:814 INFO  - Compressing file dirty.fl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:815 INFO  ok size=1b compressedSize=3 ratio=-200% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:815 INFO  - Compressing file default.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:816 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:817 INFO  - Compressing file orole.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:818 INFO  ok size=65.00KB compressedSize=131 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:819 INFO  - Compressing file v.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:820 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:821 INFO  - Compressing file orids.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:822 INFO  ok size=65.00KB compressedSize=116 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:824 INFO  - Compressing file orids.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:826 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:826 INFO  - Compressing file oschedule.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:829 INFO  ok size=65.00KB compressedSize=122 ratio=100% elapsed=3ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:829 INFO  - Compressing file ofunction.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:830 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:830 INFO  - Compressing file internal.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:831 INFO  ok size=65.00KB compressedSize=129 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:835 INFO  [node1446601006190]->[node1446601038783] backup of database 'deansdb' completed. lastOperationId=0... [OSyncDatabaseTask$1]
2015-11-04 01:38:51:058 INFO  [node1446601006190]->[node1446601038783] - transferring chunk #1 offset=0 size=15920... [OSyncDatabaseTask]
2015-11-04 01:38:51:061 INFO  [node1446601006190] received updated status node1446601006190.deansdb=ONLINE [OHazelcastPlugin]
2015-11-04 01:38:51:063 INFO  [node1446601006190]->[node1446601038783] deploy database task completed [OSyncDatabaseTask]
2015-11-04 01:38:51:315 SEVER Internal server error:
com.orientechnologies.orient.core.exception.OQueryParsingException: Error on parsing query at position #8: Error on parsing command at position #0: Error on parsing query
Query:  _studio WHERE user.name = 'admin' AND type = 'GraphConfig'
-------------^
--> com.orientechnologies.orient.core.exception.OCommandExecutionException: Class '_STUDIO' was not found in current database [ONetworkProtocolHttpDb]
2015-11-04 01:38:51:386 WARNI ->172.20.0.171: Command not found: GET.sqlProfiler/deansdb [ONetworkProtocolHttpDb]
2015-11-04 01:38:56:416 WARNI [node1446601006190] detected 1 node(s) in timeout or in conflict and quorum (2) has not been reached, rolling back changes for request (id=2 from=node1446601006190 task=command_sql(CREATE CLASS _studio) user=#5:0) [ODistributedResponseManager]
2015-11-04 01:38:56:417 WARNI [node1446601006190] Quorum 2 not reached for request (id=2 from=node1446601006190 task=command_sql(CREATE CLASS _studio) user=#5:0). Elapsed=5029ms Servers in timeout/conflict are:
 - node1446601038783: 9

Logs from server2:


2015-11-04 01:38:50:584 INFO  [node1446601038783] Saving distributed configuration file for database 'deansdb' to: /opt/orientdb/databases/deansdb/distributed-config.json [OHazelcastPlugin]
2015-11-04 01:38:50:586 INFO  [node1446601038783] received new status node1446601038783.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:50:600 WARNI [node1446601038783]->[[node1446601006190]] requesting deploy of database 'deansdb' on local server... [OHazelcastPlugin]
2015-11-04 01:38:50:657 INFO  [node1446601038783]<-[node1446601006190] received updated status node1446601038783.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:51:079 INFO  [node1446601038783]<-[node1446601006190] copying remote database 'deansdb' to: /tmp/orientdb/install_deansdb.zip [OHazelcastPlugin]
2015-11-04 01:38:51:081 WARNI [node1446601038783]<-[node1446601006190] installing database 'deansdb' to: /opt/orientdb/databases/deansdb... [OHazelcastPlugin]
2015-11-04 01:38:51:087 WARNI [node1446601038783] - writing chunk #1 offset=0 size=15.55KB [OHazelcastPlugin]
2015-11-04 01:38:51:088 INFO  [node1446601038783] database copied correctly, size=15.55KB [OHazelcastPlugin$1]
2015-11-04 01:38:51:108 INFO  {db=deansdb} - Uncompressing file ouser.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:108 INFO  {db=deansdb} - Uncompressing file ORole.name.sbt... [OHazelcastPlugin]
2015-11-04 01:38:51:109 INFO  {db=deansdb} - Uncompressing file dictionary.sbt... [OHazelcastPlugin]
2015-11-04 01:38:51:110 INFO  {db=deansdb} - Uncompressing file default.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:111 INFO  {db=deansdb} - Uncompressing file index.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:112 INFO  {db=deansdb} - Uncompressing file deansdb.wmr... [OHazelcastPlugin]
2015-11-04 01:38:51:113 INFO  {db=deansdb} - Uncompressing file manindex.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:119 INFO  {db=deansdb} - Uncompressing file ofunction.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:120 INFO  {db=deansdb} - Uncompressing file distributed-config.json... [OHazelcastPlugin]
2015-11-04 01:38:51:121 INFO  {db=deansdb} - Uncompressing file manindex.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:121 INFO  {db=deansdb} - Uncompressing file orole.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:122 INFO  {db=deansdb} - Uncompressing file index.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:128 INFO  {db=deansdb} - Uncompressing file database.ocf... [OHazelcastPlugin]
2015-11-04 01:38:51:128 INFO  {db=deansdb} - Uncompressing file oschedule.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:129 INFO  {db=deansdb} - Uncompressing file name_id_map.cm... [OHazelcastPlugin]
2015-11-04 01:38:51:129 INFO  {db=deansdb} - Uncompressing file v.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:132 INFO  {db=deansdb} - Uncompressing file ouser.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:133 INFO  {db=deansdb} - Uncompressing file internal.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:136 INFO  {db=deansdb} - Uncompressing file OUser.name.sbt... [OHazelcastPlugin]
2015-11-04 01:38:51:139 INFO  {db=deansdb} - Uncompressing file dirty.fl... [OHazelcastPlugin]
2015-11-04 01:38:51:139 INFO  {db=deansdb} - Uncompressing file default.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:140 INFO  {db=deansdb} - Uncompressing file orole.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:141 INFO  {db=deansdb} - Uncompressing file v.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:141 INFO  {db=deansdb} - Uncompressing file orids.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:142 INFO  {db=deansdb} - Uncompressing file orids.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:144 INFO  {db=deansdb} - Uncompressing file oschedule.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:145 INFO  {db=deansdb} - Uncompressing file ofunction.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:152 INFO  {db=deansdb} - Uncompressing file internal.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:152 INFO  {db=deansdb} [node1446601038783] installed database 'deansdb' [OHazelcastPlugin]
2015-11-04 01:38:51:167 INFO  {db=deansdb} [node1446601038783] found no previous messages in queue orientdb.node.node1446601038783.deansdb.request [OHazelcastDistributedMessageService]
2015-11-04 01:38:51:635 INFO  [node1446601038783] class 'ORole', creation of new local cluster 'orole_node1446601038783' (id=-1) [OHazelcastPlugin]
2015-11-04 01:38:51:748 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 0/10... [ODistributedStorage]
2015-11-04 01:38:52:236 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 1/10... [ODistributedStorage]
2015-11-04 01:38:52:704 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 2/10... [ODistributedStorage]
2015-11-04 01:38:53:162 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 3/10... [ODistributedStorage]
2015-11-04 01:38:53:617 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 4/10... [ODistributedStorage]
2015-11-04 01:38:54:069 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 5/10... [ODistributedStorage]
2015-11-04 01:38:54:520 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 6/10... [ODistributedStorage]
2015-11-04 01:38:54:982 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 7/10... [ODistributedStorage]
2015-11-04 01:38:55:445 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 8/10... [ODistributedStorage]
2015-11-04 01:38:55:905 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 9/10... [ODistributedStorage]
2015-11-04 01:38:56:364 SEVER [node1446601038783] error on creating cluster 'orole_node1446601038783' in class 'ORole':  [OHazelcastPlugin][10.0.58.237]:2434 [brevity] [3.3.5] hz._hzInstance_1_brevity.event-3 caught an exception while processing task:com.hazelcast.spi.impl.EventServiceImpl$LocalEventDispatcher@3aeb8205
com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node1446601038783' in class 'ORole'
    at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:240)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1111)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.requestDatabase(OHazelcastPlugin.java:956)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:893)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1426)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.checkDatabaseEvent(OHazelcastPlugin.java:1252)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.entryAdded(OHazelcastPlugin.java:672)
    at com.hazelcast.map.MapEventPublishingService.dispatch0(MapEventPublishingService.java:85)
    at com.hazelcast.map.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:64)
    at com.hazelcast.map.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:39)
    at com.hazelcast.map.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:17)
    at com.hazelcast.map.MapService.dispatchEvent(MapService.java:76)
    at com.hazelcast.map.MapService.dispatchEvent(MapService.java:51)
    at com.hazelcast.spi.impl.EventServiceImpl$LocalEventDispatcher.run(EventServiceImpl.java:650)
    at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:189)
    at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:173)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node1446601038783' in class 'ORole'
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1590)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1274)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1114)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1111)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:238)
    ... 15 more
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
    at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1326)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1253)
    at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2081)
    at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1058)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1583)
    ... 19 more

boardthatpowder commented Nov 4, 2015

+1 same occurring with 2.1.4

Steps to reproduce:

  • Start first instance of orientdb in distributed mode. Wait for it to finish
  • Start second instance of orientdb in distributed mode, wait for it to finish.
  • In Studio for the first instance, create a new database, and add a record.

Logs from server1:


2015-11-04 01:38:50:638 INFO  [node1446601006190] Saving distributed configuration file for database 'deansdb' to: /opt/orientdb/databases/deansdb/distributed-config.json [OHazelcastPlugin]
2015-11-04 01:38:50:649 INFO  [node1446601006190] received updated status node1446601038783.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:50:651 INFO  [node1446601006190] received updated status node1446601006190.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:50:652 WARNI [node1446601006190]->[node1446601038783] deploying database deansdb... [OSyncDatabaseTask]
2015-11-04 01:38:50:652 INFO  [node1446601006190]->[node1446601038783] creating backup of database 'deansdb' in directory: /tmp/orientdb/backup_deansdb.zip... [OSyncDatabaseTask]
2015-11-04 01:38:50:743 INFO  - Compressing file ouser.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:750 INFO  ok size=65.00KB compressedSize=132 ratio=100% elapsed=8ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:750 INFO  - Compressing file ORole.name.sbt... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:751 INFO  ok size=65.00KB compressedSize=178 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:754 INFO  - Compressing file dictionary.sbt... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:755 INFO  ok size=65.00KB compressedSize=129 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:756 INFO  - Compressing file default.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:763 INFO  ok size=65.00KB compressedSize=120 ratio=100% elapsed=6ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:763 INFO  - Compressing file index.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:765 INFO  ok size=65.00KB compressedSize=136 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:765 INFO  - Compressing file deansdb.wmr... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:768 INFO  ok size=40b compressedSize=19 ratio=53% elapsed=3ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:768 INFO  - Compressing file manindex.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:776 INFO  ok size=129.00KB compressedSize=245 ratio=100% elapsed=8ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:777 INFO  - Compressing file ofunction.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:778 INFO  ok size=65.00KB compressedSize=123 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:778 INFO  - Compressing file distributed-config.json... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:779 INFO  ok size=405b compressedSize=221 ratio=46% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:779 INFO  - Compressing file manindex.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:780 INFO  ok size=65.00KB compressedSize=120 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:783 INFO  - Compressing file orole.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:785 INFO  ok size=129.00KB compressedSize=581 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:787 INFO  - Compressing file index.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:789 INFO  ok size=129.00KB compressedSize=275 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:789 INFO  - Compressing file database.ocf... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:791 INFO  ok size=1.62KB compressedSize=272 ratio=84% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:795 INFO  - Compressing file oschedule.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:796 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:796 INFO  - Compressing file name_id_map.cm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:797 INFO  ok size=1.10KB compressedSize=270 ratio=76% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:798 INFO  - Compressing file v.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:799 INFO  ok size=65.00KB compressedSize=122 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:800 INFO  - Compressing file ouser.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:802 INFO  ok size=129.00KB compressedSize=547 ratio=100% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:803 INFO  - Compressing file internal.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:806 INFO  ok size=129.00KB compressedSize=8388 ratio=94% elapsed=4ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:811 INFO  - Compressing file OUser.name.sbt... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:814 INFO  ok size=65.00KB compressedSize=179 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:814 INFO  - Compressing file dirty.fl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:815 INFO  ok size=1b compressedSize=3 ratio=-200% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:815 INFO  - Compressing file default.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:816 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:817 INFO  - Compressing file orole.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:818 INFO  ok size=65.00KB compressedSize=131 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:819 INFO  - Compressing file v.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:820 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:821 INFO  - Compressing file orids.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:822 INFO  ok size=65.00KB compressedSize=116 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:824 INFO  - Compressing file orids.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:826 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=2ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:826 INFO  - Compressing file oschedule.pcl... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:829 INFO  ok size=65.00KB compressedSize=122 ratio=100% elapsed=3ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:829 INFO  - Compressing file ofunction.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:830 INFO  ok size=1024b compressedSize=16 ratio=99% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:830 INFO  - Compressing file internal.cpm... [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:831 INFO  ok size=65.00KB compressedSize=129 ratio=100% elapsed=1ms [OSyncDatabaseTask$1$2]
2015-11-04 01:38:50:835 INFO  [node1446601006190]->[node1446601038783] backup of database 'deansdb' completed. lastOperationId=0... [OSyncDatabaseTask$1]
2015-11-04 01:38:51:058 INFO  [node1446601006190]->[node1446601038783] - transferring chunk #1 offset=0 size=15920... [OSyncDatabaseTask]
2015-11-04 01:38:51:061 INFO  [node1446601006190] received updated status node1446601006190.deansdb=ONLINE [OHazelcastPlugin]
2015-11-04 01:38:51:063 INFO  [node1446601006190]->[node1446601038783] deploy database task completed [OSyncDatabaseTask]
2015-11-04 01:38:51:315 SEVER Internal server error:
com.orientechnologies.orient.core.exception.OQueryParsingException: Error on parsing query at position #8: Error on parsing command at position #0: Error on parsing query
Query:  _studio WHERE user.name = 'admin' AND type = 'GraphConfig'
-------------^
--> com.orientechnologies.orient.core.exception.OCommandExecutionException: Class '_STUDIO' was not found in current database [ONetworkProtocolHttpDb]
2015-11-04 01:38:51:386 WARNI ->172.20.0.171: Command not found: GET.sqlProfiler/deansdb [ONetworkProtocolHttpDb]
2015-11-04 01:38:56:416 WARNI [node1446601006190] detected 1 node(s) in timeout or in conflict and quorum (2) has not been reached, rolling back changes for request (id=2 from=node1446601006190 task=command_sql(CREATE CLASS _studio) user=#5:0) [ODistributedResponseManager]
2015-11-04 01:38:56:417 WARNI [node1446601006190] Quorum 2 not reached for request (id=2 from=node1446601006190 task=command_sql(CREATE CLASS _studio) user=#5:0). Elapsed=5029ms Servers in timeout/conflict are:
 - node1446601038783: 9

Logs from server2:


2015-11-04 01:38:50:584 INFO  [node1446601038783] Saving distributed configuration file for database 'deansdb' to: /opt/orientdb/databases/deansdb/distributed-config.json [OHazelcastPlugin]
2015-11-04 01:38:50:586 INFO  [node1446601038783] received new status node1446601038783.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:50:600 WARNI [node1446601038783]->[[node1446601006190]] requesting deploy of database 'deansdb' on local server... [OHazelcastPlugin]
2015-11-04 01:38:50:657 INFO  [node1446601038783]<-[node1446601006190] received updated status node1446601038783.deansdb=SYNCHRONIZING [OHazelcastPlugin]
2015-11-04 01:38:51:079 INFO  [node1446601038783]<-[node1446601006190] copying remote database 'deansdb' to: /tmp/orientdb/install_deansdb.zip [OHazelcastPlugin]
2015-11-04 01:38:51:081 WARNI [node1446601038783]<-[node1446601006190] installing database 'deansdb' to: /opt/orientdb/databases/deansdb... [OHazelcastPlugin]
2015-11-04 01:38:51:087 WARNI [node1446601038783] - writing chunk #1 offset=0 size=15.55KB [OHazelcastPlugin]
2015-11-04 01:38:51:088 INFO  [node1446601038783] database copied correctly, size=15.55KB [OHazelcastPlugin$1]
2015-11-04 01:38:51:108 INFO  {db=deansdb} - Uncompressing file ouser.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:108 INFO  {db=deansdb} - Uncompressing file ORole.name.sbt... [OHazelcastPlugin]
2015-11-04 01:38:51:109 INFO  {db=deansdb} - Uncompressing file dictionary.sbt... [OHazelcastPlugin]
2015-11-04 01:38:51:110 INFO  {db=deansdb} - Uncompressing file default.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:111 INFO  {db=deansdb} - Uncompressing file index.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:112 INFO  {db=deansdb} - Uncompressing file deansdb.wmr... [OHazelcastPlugin]
2015-11-04 01:38:51:113 INFO  {db=deansdb} - Uncompressing file manindex.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:119 INFO  {db=deansdb} - Uncompressing file ofunction.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:120 INFO  {db=deansdb} - Uncompressing file distributed-config.json... [OHazelcastPlugin]
2015-11-04 01:38:51:121 INFO  {db=deansdb} - Uncompressing file manindex.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:121 INFO  {db=deansdb} - Uncompressing file orole.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:122 INFO  {db=deansdb} - Uncompressing file index.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:128 INFO  {db=deansdb} - Uncompressing file database.ocf... [OHazelcastPlugin]
2015-11-04 01:38:51:128 INFO  {db=deansdb} - Uncompressing file oschedule.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:129 INFO  {db=deansdb} - Uncompressing file name_id_map.cm... [OHazelcastPlugin]
2015-11-04 01:38:51:129 INFO  {db=deansdb} - Uncompressing file v.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:132 INFO  {db=deansdb} - Uncompressing file ouser.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:133 INFO  {db=deansdb} - Uncompressing file internal.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:136 INFO  {db=deansdb} - Uncompressing file OUser.name.sbt... [OHazelcastPlugin]
2015-11-04 01:38:51:139 INFO  {db=deansdb} - Uncompressing file dirty.fl... [OHazelcastPlugin]
2015-11-04 01:38:51:139 INFO  {db=deansdb} - Uncompressing file default.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:140 INFO  {db=deansdb} - Uncompressing file orole.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:141 INFO  {db=deansdb} - Uncompressing file v.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:141 INFO  {db=deansdb} - Uncompressing file orids.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:142 INFO  {db=deansdb} - Uncompressing file orids.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:144 INFO  {db=deansdb} - Uncompressing file oschedule.pcl... [OHazelcastPlugin]
2015-11-04 01:38:51:145 INFO  {db=deansdb} - Uncompressing file ofunction.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:152 INFO  {db=deansdb} - Uncompressing file internal.cpm... [OHazelcastPlugin]
2015-11-04 01:38:51:152 INFO  {db=deansdb} [node1446601038783] installed database 'deansdb' [OHazelcastPlugin]
2015-11-04 01:38:51:167 INFO  {db=deansdb} [node1446601038783] found no previous messages in queue orientdb.node.node1446601038783.deansdb.request [OHazelcastDistributedMessageService]
2015-11-04 01:38:51:635 INFO  [node1446601038783] class 'ORole', creation of new local cluster 'orole_node1446601038783' (id=-1) [OHazelcastPlugin]
2015-11-04 01:38:51:748 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 0/10... [ODistributedStorage]
2015-11-04 01:38:52:236 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 1/10... [ODistributedStorage]
2015-11-04 01:38:52:704 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 2/10... [ODistributedStorage]
2015-11-04 01:38:53:162 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 3/10... [ODistributedStorage]
2015-11-04 01:38:53:617 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 4/10... [ODistributedStorage]
2015-11-04 01:38:54:069 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 5/10... [ODistributedStorage]
2015-11-04 01:38:54:520 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 6/10... [ODistributedStorage]
2015-11-04 01:38:54:982 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 7/10... [ODistributedStorage]
2015-11-04 01:38:55:445 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 8/10... [ODistributedStorage]
2015-11-04 01:38:55:905 WARNI Error on creating cluster on distributed nodes: ids are different (local=10 and remote=12). Retrying 9/10... [ODistributedStorage]
2015-11-04 01:38:56:364 SEVER [node1446601038783] error on creating cluster 'orole_node1446601038783' in class 'ORole':  [OHazelcastPlugin][10.0.58.237]:2434 [brevity] [3.3.5] hz._hzInstance_1_brevity.event-3 caught an exception while processing task:com.hazelcast.spi.impl.EventServiceImpl$LocalEventDispatcher@3aeb8205
com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node1446601038783' in class 'ORole'
    at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:240)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1111)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.requestDatabase(OHazelcastPlugin.java:956)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:893)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1426)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.checkDatabaseEvent(OHazelcastPlugin.java:1252)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.entryAdded(OHazelcastPlugin.java:672)
    at com.hazelcast.map.MapEventPublishingService.dispatch0(MapEventPublishingService.java:85)
    at com.hazelcast.map.MapEventPublishingService.dispatchEntryEventData(MapEventPublishingService.java:64)
    at com.hazelcast.map.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:39)
    at com.hazelcast.map.MapEventPublishingService.dispatchEvent(MapEventPublishingService.java:17)
    at com.hazelcast.map.MapService.dispatchEvent(MapService.java:76)
    at com.hazelcast.map.MapService.dispatchEvent(MapService.java:51)
    at com.hazelcast.spi.impl.EventServiceImpl$LocalEventDispatcher.run(EventServiceImpl.java:650)
    at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:189)
    at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:173)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node1446601038783' in class 'ORole'
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1590)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1274)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1114)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1111)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:238)
    ... 15 more
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different
    at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1326)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1253)
    at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2081)
    at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1058)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1583)
    ... 19 more
@Waldiii

This comment has been minimized.

Show comment
Hide comment
@Waldiii

Waldiii Nov 10, 2015

+1 with 2.1.5

My steps are the sames as dhart-alldigitals.

It doesn't matter if I create the database via Studio or directly per Java API (which is my usual usecase)

Waldiii commented Nov 10, 2015

+1 with 2.1.5

My steps are the sames as dhart-alldigitals.

It doesn't matter if I create the database via Studio or directly per Java API (which is my usual usecase)

@rajkumargithub

This comment has been minimized.

Show comment
Hide comment
@rajkumargithub

rajkumargithub Nov 10, 2015

+1, i am running into the same problem with 3 servers. But still i see the newly created database available in all 3 servers. Would that be a potential problem for a production environment? Any fix coming soon?

rajkumargithub commented Nov 10, 2015

+1, i am running into the same problem with 3 servers. But still i see the newly created database available in all 3 servers. Would that be a potential problem for a production environment? Any fix coming soon?

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Nov 12, 2015

Member

@dhart-alldigital there is an issue on creating database in cluster setup: the new database is not fully distributed to all the servers. While this is an issue, you can easily bypass it by creating the database before to run a distributed configuration. Could you please try with 2.1.5?

@rajkumargithub is your problem the same? Are you creating the database in distributed mode?

Member

lvca commented Nov 12, 2015

@dhart-alldigital there is an issue on creating database in cluster setup: the new database is not fully distributed to all the servers. While this is an issue, you can easily bypass it by creating the database before to run a distributed configuration. Could you please try with 2.1.5?

@rajkumargithub is your problem the same? Are you creating the database in distributed mode?

@lvca lvca added the in progress label Nov 12, 2015

@Waldiii

This comment has been minimized.

Show comment
Hide comment
@Waldiii

Waldiii Nov 12, 2015

@ivca This may be an options for the others but I am creating databases on demand. So I'm depending on the fix

Waldiii commented Nov 12, 2015

@ivca This may be an options for the others but I am creating databases on demand. So I'm depending on the fix

@smolinari

This comment has been minimized.

Show comment
Hide comment
@smolinari

smolinari Nov 12, 2015

Contributor

👍 to what @Waldiii said.

Once our system is up and running, we'd also be needing the same thing at some point too.

Databases on demand!

Scott

Contributor

smolinari commented Nov 12, 2015

👍 to what @Waldiii said.

Once our system is up and running, we'd also be needing the same thing at some point too.

Databases on demand!

Scott

@aleksnikolic

This comment has been minimized.

Show comment
Hide comment
@aleksnikolic

aleksnikolic Nov 12, 2015

+1 The same issue as others.

I have upgraded my docker image to 2.1.5 version and running it in distributed mode directly from docker-compose file. I can put some startup script to create databases before running it in distributed mode but I don't think that's a good option for production.

Do you guys think you can fix this quickly?

aleksnikolic commented Nov 12, 2015

+1 The same issue as others.

I have upgraded my docker image to 2.1.5 version and running it in distributed mode directly from docker-compose file. I can put some startup script to create databases before running it in distributed mode but I don't think that's a good option for production.

Do you guys think you can fix this quickly?

@boardthatpowder

This comment has been minimized.

Show comment
Hide comment
@boardthatpowder

boardthatpowder Nov 12, 2015

@lvca. I was in fact (and would prefer) to create the databases beforehand as mine are fixed. In 2.1.4 I was initially creating them at startup by specifying them in orientdb-server-config.xml. However when launching in distributed mode hazelcast would fail distributing the databases. I tracked this down to the database path in the config not starting with "plocal:./databases/" which the hazelcast plugin had an explicit check for but even after updating the path for some reason the databases were still not distributing (remaining in plocal mode), hence why I tried creating manually after clustering the servers.

Will try again with 2.1.5 and report back.

boardthatpowder commented Nov 12, 2015

@lvca. I was in fact (and would prefer) to create the databases beforehand as mine are fixed. In 2.1.4 I was initially creating them at startup by specifying them in orientdb-server-config.xml. However when launching in distributed mode hazelcast would fail distributing the databases. I tracked this down to the database path in the config not starting with "plocal:./databases/" which the hazelcast plugin had an explicit check for but even after updating the path for some reason the databases were still not distributing (remaining in plocal mode), hence why I tried creating manually after clustering the servers.

Will try again with 2.1.5 and report back.

@boardthatpowder

This comment has been minimized.

Show comment
Hide comment
@boardthatpowder

boardthatpowder Nov 12, 2015

@lvca. Using 2.1.5, If I configure my databases to create at startup as follows:

orientdb-server-config.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<orient-server>
    ...
    <storages>
        <storage name="organizations" path="plocal:./databases/test"
            userName="admin" userPassword="admin"
            loaded-at-startup="true" />
    </storages>
    ...
</orient-server>

If I start instance A in distributed mode, then followed by instance B in distributed mode, I get the following exceptions on instanceA:

Members [2] {
    Member [10.0.165.95]:2434 this
    Member [10.0.159.32]:2434
}
 [ClusterService]
2015-11-12 19:03:09:355 INFO  [10.0.165.95]:2434 [brevity] [3.3.5] Remaining migration tasks in queue => 1 [InternalPartitionService]
2015-11-12 19:03:09:466 INFO  [10.0.165.95]:2434 [brevity] [3.3.5] Re-partitioning cluster data... Migration queue size: 135 [InternalPartitionService]
2015-11-12 19:03:10:863 INFO  [10.0.165.95]:2434 [brevity] [3.3.5] All migration tasks have been completed, queues are empty. [InternalPartitionService]
2015-11-12 19:03:11:292 INFO  [node1447354913827]<-[node1447354982571] added node configuration id=Member [10.0.159.32]:2434 name=node1447354982571, now 2 nodes are configured [OHazelcastPlugin]
2015-11-12 19:03:11:413 INFO  [node1447354913827]<-[node1447354982571] received new status node1447354982571.test=SYNCHRONIZING [OHazelcastPlugin]
2015-11-12 19:03:11:424 INFO  [node1447354913827]--[node1447354982571] update configuration db=test [OHazelcastPlugin]
2015-11-12 19:03:11:441 WARNI [node1447354913827] updated distributed configuration for database: test:
----------
{
  "version": 2,
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "undefined",
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,"clusters":{
    "internal": {
},
    "index": {
},
    "*": {
  "servers": ["node1447354913827","node1447354982571","<NEW_NODE>"]
}
    }
}
---------- [OHazelcastPlugin]
2015-11-12 19:03:11:443 INFO  [node1447354913827] Saving distributed configuration file for database 'test' to: /opt/orientdb/databases/test/distributed-config.json [OHazelcastPlugin]{db=test} [node1447354913827]<-[node1447354982571] error on executing distributed request 0: deploy_db
com.orientechnologies.orient.server.distributed.ODistributedException: Distributed storage was not installed for database 'test'. Implementation found: com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:794)
    at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:296)
    at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:119)

The code that I found that was restricting hazelcast distribution based on path name was here:
com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.java

@Override
  public void onCreate(final ODatabaseInternal iDatabase) {
    final String dbUrl = OSystemVariableResolver.resolveSystemVariables(iDatabase.getURL());
    if (dbUrl.startsWith("plocal:")) {
      // CHECK SPECIAL CASE WITH MULTIPLE SERVER INSTANCES ON THE SAME JVM
      final String dbDirectory = serverInstance.getDatabaseDirectory();
      if (!dbUrl.substring("plocal:".length()).startsWith(dbDirectory))
        // SKIP IT: THIS HAPPENS ONLY ON MULTIPLE SERVER INSTANCES ON THE SAME JVM
        return;
    } else if (dbUrl.startsWith("remote:"))
      return;

    final ODatabaseDocumentInternal currDb = ODatabaseRecordThreadLocal.INSTANCE.getIfDefined();
    try {

      if (getConfigurationMap().containsKey(OHazelcastPlugin.CONFIG_DATABASE_PREFIX + iDatabase.getName()))
        throw new ODistributedException("Cannot create a new database with the same name of one available distributed");

      final OHazelcastDistributedDatabase distribDatabase = messageService.registerDatabase(iDatabase.getName());
      distribDatabase.configureDatabase(false, false, null).setOnline();
      onOpen(iDatabase);

    } finally {
      // RESTORE ORIGINAL DATABASE INSTANCE IN TL
      ODatabaseRecordThreadLocal.INSTANCE.set(currDb);
    }
  }

boardthatpowder commented Nov 12, 2015

@lvca. Using 2.1.5, If I configure my databases to create at startup as follows:

orientdb-server-config.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<orient-server>
    ...
    <storages>
        <storage name="organizations" path="plocal:./databases/test"
            userName="admin" userPassword="admin"
            loaded-at-startup="true" />
    </storages>
    ...
</orient-server>

If I start instance A in distributed mode, then followed by instance B in distributed mode, I get the following exceptions on instanceA:

Members [2] {
    Member [10.0.165.95]:2434 this
    Member [10.0.159.32]:2434
}
 [ClusterService]
2015-11-12 19:03:09:355 INFO  [10.0.165.95]:2434 [brevity] [3.3.5] Remaining migration tasks in queue => 1 [InternalPartitionService]
2015-11-12 19:03:09:466 INFO  [10.0.165.95]:2434 [brevity] [3.3.5] Re-partitioning cluster data... Migration queue size: 135 [InternalPartitionService]
2015-11-12 19:03:10:863 INFO  [10.0.165.95]:2434 [brevity] [3.3.5] All migration tasks have been completed, queues are empty. [InternalPartitionService]
2015-11-12 19:03:11:292 INFO  [node1447354913827]<-[node1447354982571] added node configuration id=Member [10.0.159.32]:2434 name=node1447354982571, now 2 nodes are configured [OHazelcastPlugin]
2015-11-12 19:03:11:413 INFO  [node1447354913827]<-[node1447354982571] received new status node1447354982571.test=SYNCHRONIZING [OHazelcastPlugin]
2015-11-12 19:03:11:424 INFO  [node1447354913827]--[node1447354982571] update configuration db=test [OHazelcastPlugin]
2015-11-12 19:03:11:441 WARNI [node1447354913827] updated distributed configuration for database: test:
----------
{
  "version": 2,
  "autoDeploy": true,
  "hotAlignment": false,
  "executionMode": "undefined",
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,"clusters":{
    "internal": {
},
    "index": {
},
    "*": {
  "servers": ["node1447354913827","node1447354982571","<NEW_NODE>"]
}
    }
}
---------- [OHazelcastPlugin]
2015-11-12 19:03:11:443 INFO  [node1447354913827] Saving distributed configuration file for database 'test' to: /opt/orientdb/databases/test/distributed-config.json [OHazelcastPlugin]{db=test} [node1447354913827]<-[node1447354982571] error on executing distributed request 0: deploy_db
com.orientechnologies.orient.server.distributed.ODistributedException: Distributed storage was not installed for database 'test'. Implementation found: com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.executeOnLocalNode(OHazelcastPlugin.java:794)
    at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.onMessage(ODistributedWorker.java:296)
    at com.orientechnologies.orient.server.hazelcast.ODistributedWorker.run(ODistributedWorker.java:119)

The code that I found that was restricting hazelcast distribution based on path name was here:
com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.java

@Override
  public void onCreate(final ODatabaseInternal iDatabase) {
    final String dbUrl = OSystemVariableResolver.resolveSystemVariables(iDatabase.getURL());
    if (dbUrl.startsWith("plocal:")) {
      // CHECK SPECIAL CASE WITH MULTIPLE SERVER INSTANCES ON THE SAME JVM
      final String dbDirectory = serverInstance.getDatabaseDirectory();
      if (!dbUrl.substring("plocal:".length()).startsWith(dbDirectory))
        // SKIP IT: THIS HAPPENS ONLY ON MULTIPLE SERVER INSTANCES ON THE SAME JVM
        return;
    } else if (dbUrl.startsWith("remote:"))
      return;

    final ODatabaseDocumentInternal currDb = ODatabaseRecordThreadLocal.INSTANCE.getIfDefined();
    try {

      if (getConfigurationMap().containsKey(OHazelcastPlugin.CONFIG_DATABASE_PREFIX + iDatabase.getName()))
        throw new ODistributedException("Cannot create a new database with the same name of one available distributed");

      final OHazelcastDistributedDatabase distribDatabase = messageService.registerDatabase(iDatabase.getName());
      distribDatabase.configureDatabase(false, false, null).setOnline();
      onOpen(iDatabase);

    } finally {
      // RESTORE ORIGINAL DATABASE INSTANCE IN TL
      ODatabaseRecordThreadLocal.INSTANCE.set(currDb);
    }
  }
@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Nov 12, 2015

Member

@dhart-alldigital I think the path should be ../databases and not ./databases if the script starts from bin directory.

About @aleksnikolic, @smolinari and @Waldiii the fix should be straightforward, I count to do that in the next 24h for 2.1.6.

Member

lvca commented Nov 12, 2015

@dhart-alldigital I think the path should be ../databases and not ./databases if the script starts from bin directory.

About @aleksnikolic, @smolinari and @Waldiii the fix should be straightforward, I count to do that in the next 24h for 2.1.6.

@smolinari

This comment has been minimized.

Show comment
Hide comment
@smolinari

smolinari Nov 12, 2015

Contributor

@lvca - Sounds good. Thanks for the attention.

Scott

Contributor

smolinari commented Nov 12, 2015

@lvca - Sounds good. Thanks for the attention.

Scott

@boardthatpowder

This comment has been minimized.

Show comment
Hide comment
@boardthatpowder

boardthatpowder Nov 12, 2015

@lvca, "../databases" did not work, but you did point me in the right direction with it being based on the bin directory.

For info incase anyone else needs it....

The validation that OHazelcastPlugin is making on the database path is determined by OServer.startup() which is output to the log as:

2015-11-12 17:22:35:139 INFO  Databases directory: /opt/orientdb/databases [OServer]

Therefore setting the auto-create databases with a path that starts with plocal:/opt/orientdb/databases in orientdb-server-config.xml did fix my problem. My auto-created databases are now distributing. My limited testing seems to be working.

Thanks @lvca!

boardthatpowder commented Nov 12, 2015

@lvca, "../databases" did not work, but you did point me in the right direction with it being based on the bin directory.

For info incase anyone else needs it....

The validation that OHazelcastPlugin is making on the database path is determined by OServer.startup() which is output to the log as:

2015-11-12 17:22:35:139 INFO  Databases directory: /opt/orientdb/databases [OServer]

Therefore setting the auto-create databases with a path that starts with plocal:/opt/orientdb/databases in orientdb-server-config.xml did fix my problem. My auto-created databases are now distributing. My limited testing seems to be working.

Thanks @lvca!

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Nov 13, 2015

Member

Cool you resolved.

Member

lvca commented Nov 13, 2015

Cool you resolved.

lvca added a commit that referenced this issue Nov 13, 2015

@lvca lvca closed this Nov 13, 2015

@lvca lvca removed the in progress label Nov 13, 2015

lvca added a commit that referenced this issue Nov 13, 2015

@lvca lvca modified the milestones: 2.1.x (next hotfix), 2.1.6 Nov 22, 2015

@lvca lvca removed this from the 2.1.x (next hotfix) milestone Nov 22, 2015

@hey-mx

This comment has been minimized.

Show comment
Hide comment
@hey-mx

hey-mx Jan 27, 2016

Hi,

I have the same problem but with the version 2.1.7 I have 3 nodes and I tried to create a database in the first node with:

CREATE DATABASE remote:localhost/JobomasGraph root xxxxxxx plocal

But in the others nodes I have noticed this on the log file:

---------- [OHazelcastPlugin] 2016-01-27 14:59:22:908 INFO [node3] Saving distributed configuration file for database 'JobomasGraph' to: /usr/local/orientdb/databases/JobomasGraph/distributed-config.json [OHazelcastPlugin] 2016-01-27 14:59:22:921 WARNI [node3]->[[node1]] requesting deploy of database 'JobomasGraph' on local server... [OHazelcastPlugin] 2016-01-27 14:59:22:946 INFO [node3]<-[node1] received updated status node3.JobomasGraph=SYNCHRONIZING [OHazelcastPlugin] 2016-01-27 14:59:22:947 INFO [node3]<-[node1] received updated status node1.JobomasGraph=SYNCHRONIZING [OHazelcastPlugin] 2016-01-27 14:59:22:948 INFO [node3]<-[node1] received updated status node1.JobomasGraph=ONLINE [OHazelcastPlugin] 2016-01-27 14:59:22:955 WARNI [node3] moving existent database 'JobomasGraph' located in '/usr/local/orientdb/databases/JobomasGraph' to '/usr/local/orientdb/databases/../backup/databases/JobomasGraph' and get a fresh copy from a remote node... [OHazelcastPlugin] 2016-01-27 14:59:22:956 INFO [node3]<-[node1] copying remote database 'JobomasGraph' to: /tmp/orientdb/install_JobomasGraph.zip [OHazelcastPlugin] 2016-01-27 14:59:22:957 WARNI [node3]<-[node1] installing database 'JobomasGraph' to: /usr/local/orientdb/databases/JobomasGraph... [OHazelcastPlugin] 2016-01-27 14:59:22:961 WARNI [node3] - writing chunk #1 offset=0 size=13.37KB [OHazelcastPlugin] 2016-01-27 14:59:22:962 INFO [node3] database copied correctly, size=13.37KB [OHazelcastPlugin$1] 2016-01-27 14:59:22:992 INFO {db=JobomasGraph} - Uncompressing file orole.pcl... [OHazelcastPlugin] 2016-01-27 14:59:22:994 INFO {db=JobomasGraph} - Uncompressing file default.cpm... [OHazelcastPlugin] 2016-01-27 14:59:22:994 INFO {db=JobomasGraph} - Uncompressing file oschedule.cpm... [OHazelcastPlugin] 2016-01-27 14:59:22:995 INFO {db=JobomasGraph} - Uncompressing file JobomasGraph.wmr... [OHazelcastPlugin] 2016-01-27 14:59:22:995 INFO {db=JobomasGraph} - Uncompressing file default.pcl... [OHazelcastPlugin] 2016-01-27 14:59:22:996 INFO {db=JobomasGraph} - Uncompressing file dictionary.sbt... [OHazelcastPlugin] 2016-01-27 14:59:22:996 INFO {db=JobomasGraph} - Uncompressing file OUser.name.sbt... [OHazelcastPlugin] 2016-01-27 14:59:22:997 INFO {db=JobomasGraph} - Uncompressing file internal.pcl... [OHazelcastPlugin] 2016-01-27 14:59:22:998 INFO {db=JobomasGraph} - Uncompressing file dirty.fl... [OHazelcastPlugin] 2016-01-27 14:59:22:999 INFO {db=JobomasGraph} - Uncompressing file orids.cpm... [OHazelcastPlugin] 2016-01-27 14:59:22:999 INFO {db=JobomasGraph} - Uncompressing file ouser.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:000 INFO {db=JobomasGraph} - Uncompressing file name_id_map.cm... [OHazelcastPlugin] 2016-01-27 14:59:23:000 INFO {db=JobomasGraph} - Uncompressing file ORole.name.sbt... [OHazelcastPlugin] 2016-01-27 14:59:23:001 INFO {db=JobomasGraph} - Uncompressing file oschedule.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:001 INFO {db=JobomasGraph} - Uncompressing file ouser.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:002 INFO {db=JobomasGraph} - Uncompressing file internal.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:003 INFO {db=JobomasGraph} - Uncompressing file ofunction.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:004 INFO {db=JobomasGraph} - Uncompressing file orids.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:004 INFO {db=JobomasGraph} - Uncompressing file index.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:005 INFO {db=JobomasGraph} - Uncompressing file manindex.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:006 INFO {db=JobomasGraph} - Uncompressing file database.ocf... [OHazelcastPlugin] 2016-01-27 14:59:23:006 INFO {db=JobomasGraph} - Uncompressing file distributed-config.json... [OHazelcastPlugin] 2016-01-27 14:59:23:006 INFO {db=JobomasGraph} - Uncompressing file orole.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:007 INFO {db=JobomasGraph} - Uncompressing file ofunction.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:007 INFO {db=JobomasGraph} - Uncompressing file manindex.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:008 INFO {db=JobomasGraph} - Uncompressing file index.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:008 INFO {db=JobomasGraph} [node3] installed database 'JobomasGraph' [OHazelcastPlugin] 2016-01-27 14:59:23:012 INFO {db=JobomasGraph} [node3] found no previous messages in queue orientdb.node.node3.JobomasGraph.request [OHazelcastDistributedMessageService] 2016-01-27 14:59:23:066 WARNI {db=JobomasGraph} Storage 'JobomasGraph' was not closed properly. Will try to recover from write ahead log [OLocalPaginatedStorage] 2016-01-27 14:59:23:067 SEVER {db=JobomasGraph} Restore is not possible because write ahead log is empty. [OLocalPaginatedStorage] 2016-01-27 14:59:23:067 INFO {db=JobomasGraph} Storage data recover was completed [OLocalPaginatedStorage] 2016-01-27 14:59:23:477 INFO [node3] class 'ORole', creation of new local cluster 'orole_node3' (id=-1) [OHazelcastPlugin] 2016-01-27 14:59:23:555 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 0/10... [ODistributedStorage] 2016-01-27 14:59:23:961 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 1/10... [ODistributedStorage] 2016-01-27 14:59:24:383 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 2/10... [ODistributedStorage] 2016-01-27 14:59:24:777 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=10). Retrying 3/10... [ODistributedStorage] 2016-01-27 14:59:25:165 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 4/10... [ODistributedStorage] 2016-01-27 14:59:25:558 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 5/10... [ODistributedStorage] 2016-01-27 14:59:25:948 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 6/10... [ODistributedStorage] 2016-01-27 14:59:26:343 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 7/10... [ODistributedStorage] 2016-01-27 14:59:26:727 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 8/10... [ODistributedStorage] 2016-01-27 14:59:27:110 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 9/10... [ODistributedStorage] 2016-01-27 14:59:27:474 SEVER [node3] error on creating cluster 'orole_node3' in class 'ORole': [OHazelcastPlugin][node3] Error on starting distributed plugin com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node3' in class 'ORole' at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:241) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1124) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.requestDatabase(OHazelcastPlugin.java:964) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:901) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1461) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:185) at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:979) at com.orientechnologies.orient.server.OServer.activate(OServer.java:346) at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41) Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node3' in class 'ORole' at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1624) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1293) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1127) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1124) at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:239) ... 8 more Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1364) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2102) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1068) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1617) ... 12 more

Is it an issue that was not fixed on this version?

hey-mx commented Jan 27, 2016

Hi,

I have the same problem but with the version 2.1.7 I have 3 nodes and I tried to create a database in the first node with:

CREATE DATABASE remote:localhost/JobomasGraph root xxxxxxx plocal

But in the others nodes I have noticed this on the log file:

---------- [OHazelcastPlugin] 2016-01-27 14:59:22:908 INFO [node3] Saving distributed configuration file for database 'JobomasGraph' to: /usr/local/orientdb/databases/JobomasGraph/distributed-config.json [OHazelcastPlugin] 2016-01-27 14:59:22:921 WARNI [node3]->[[node1]] requesting deploy of database 'JobomasGraph' on local server... [OHazelcastPlugin] 2016-01-27 14:59:22:946 INFO [node3]<-[node1] received updated status node3.JobomasGraph=SYNCHRONIZING [OHazelcastPlugin] 2016-01-27 14:59:22:947 INFO [node3]<-[node1] received updated status node1.JobomasGraph=SYNCHRONIZING [OHazelcastPlugin] 2016-01-27 14:59:22:948 INFO [node3]<-[node1] received updated status node1.JobomasGraph=ONLINE [OHazelcastPlugin] 2016-01-27 14:59:22:955 WARNI [node3] moving existent database 'JobomasGraph' located in '/usr/local/orientdb/databases/JobomasGraph' to '/usr/local/orientdb/databases/../backup/databases/JobomasGraph' and get a fresh copy from a remote node... [OHazelcastPlugin] 2016-01-27 14:59:22:956 INFO [node3]<-[node1] copying remote database 'JobomasGraph' to: /tmp/orientdb/install_JobomasGraph.zip [OHazelcastPlugin] 2016-01-27 14:59:22:957 WARNI [node3]<-[node1] installing database 'JobomasGraph' to: /usr/local/orientdb/databases/JobomasGraph... [OHazelcastPlugin] 2016-01-27 14:59:22:961 WARNI [node3] - writing chunk #1 offset=0 size=13.37KB [OHazelcastPlugin] 2016-01-27 14:59:22:962 INFO [node3] database copied correctly, size=13.37KB [OHazelcastPlugin$1] 2016-01-27 14:59:22:992 INFO {db=JobomasGraph} - Uncompressing file orole.pcl... [OHazelcastPlugin] 2016-01-27 14:59:22:994 INFO {db=JobomasGraph} - Uncompressing file default.cpm... [OHazelcastPlugin] 2016-01-27 14:59:22:994 INFO {db=JobomasGraph} - Uncompressing file oschedule.cpm... [OHazelcastPlugin] 2016-01-27 14:59:22:995 INFO {db=JobomasGraph} - Uncompressing file JobomasGraph.wmr... [OHazelcastPlugin] 2016-01-27 14:59:22:995 INFO {db=JobomasGraph} - Uncompressing file default.pcl... [OHazelcastPlugin] 2016-01-27 14:59:22:996 INFO {db=JobomasGraph} - Uncompressing file dictionary.sbt... [OHazelcastPlugin] 2016-01-27 14:59:22:996 INFO {db=JobomasGraph} - Uncompressing file OUser.name.sbt... [OHazelcastPlugin] 2016-01-27 14:59:22:997 INFO {db=JobomasGraph} - Uncompressing file internal.pcl... [OHazelcastPlugin] 2016-01-27 14:59:22:998 INFO {db=JobomasGraph} - Uncompressing file dirty.fl... [OHazelcastPlugin] 2016-01-27 14:59:22:999 INFO {db=JobomasGraph} - Uncompressing file orids.cpm... [OHazelcastPlugin] 2016-01-27 14:59:22:999 INFO {db=JobomasGraph} - Uncompressing file ouser.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:000 INFO {db=JobomasGraph} - Uncompressing file name_id_map.cm... [OHazelcastPlugin] 2016-01-27 14:59:23:000 INFO {db=JobomasGraph} - Uncompressing file ORole.name.sbt... [OHazelcastPlugin] 2016-01-27 14:59:23:001 INFO {db=JobomasGraph} - Uncompressing file oschedule.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:001 INFO {db=JobomasGraph} - Uncompressing file ouser.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:002 INFO {db=JobomasGraph} - Uncompressing file internal.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:003 INFO {db=JobomasGraph} - Uncompressing file ofunction.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:004 INFO {db=JobomasGraph} - Uncompressing file orids.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:004 INFO {db=JobomasGraph} - Uncompressing file index.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:005 INFO {db=JobomasGraph} - Uncompressing file manindex.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:006 INFO {db=JobomasGraph} - Uncompressing file database.ocf... [OHazelcastPlugin] 2016-01-27 14:59:23:006 INFO {db=JobomasGraph} - Uncompressing file distributed-config.json... [OHazelcastPlugin] 2016-01-27 14:59:23:006 INFO {db=JobomasGraph} - Uncompressing file orole.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:007 INFO {db=JobomasGraph} - Uncompressing file ofunction.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:007 INFO {db=JobomasGraph} - Uncompressing file manindex.pcl... [OHazelcastPlugin] 2016-01-27 14:59:23:008 INFO {db=JobomasGraph} - Uncompressing file index.cpm... [OHazelcastPlugin] 2016-01-27 14:59:23:008 INFO {db=JobomasGraph} [node3] installed database 'JobomasGraph' [OHazelcastPlugin] 2016-01-27 14:59:23:012 INFO {db=JobomasGraph} [node3] found no previous messages in queue orientdb.node.node3.JobomasGraph.request [OHazelcastDistributedMessageService] 2016-01-27 14:59:23:066 WARNI {db=JobomasGraph} Storage 'JobomasGraph' was not closed properly. Will try to recover from write ahead log [OLocalPaginatedStorage] 2016-01-27 14:59:23:067 SEVER {db=JobomasGraph} Restore is not possible because write ahead log is empty. [OLocalPaginatedStorage] 2016-01-27 14:59:23:067 INFO {db=JobomasGraph} Storage data recover was completed [OLocalPaginatedStorage] 2016-01-27 14:59:23:477 INFO [node3] class 'ORole', creation of new local cluster 'orole_node3' (id=-1) [OHazelcastPlugin] 2016-01-27 14:59:23:555 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 0/10... [ODistributedStorage] 2016-01-27 14:59:23:961 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 1/10... [ODistributedStorage] 2016-01-27 14:59:24:383 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 2/10... [ODistributedStorage] 2016-01-27 14:59:24:777 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=10). Retrying 3/10... [ODistributedStorage] 2016-01-27 14:59:25:165 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 4/10... [ODistributedStorage] 2016-01-27 14:59:25:558 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 5/10... [ODistributedStorage] 2016-01-27 14:59:25:948 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 6/10... [ODistributedStorage] 2016-01-27 14:59:26:343 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 7/10... [ODistributedStorage] 2016-01-27 14:59:26:727 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 8/10... [ODistributedStorage] 2016-01-27 14:59:27:110 WARNI Error on creating cluster on distributed nodes: ids are different (local=9 and remote=12). Retrying 9/10... [ODistributedStorage] 2016-01-27 14:59:27:474 SEVER [node3] error on creating cluster 'orole_node3' in class 'ORole': [OHazelcastPlugin][node3] Error on starting distributed plugin com.orientechnologies.orient.server.distributed.ODistributedException: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node3' in class 'ORole' at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:241) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1124) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.requestDatabase(OHazelcastPlugin.java:964) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:901) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1461) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:185) at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:979) at com.orientechnologies.orient.server.OServer.activate(OServer.java:346) at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41) Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster 'orole_node3' in class 'ORole' at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1624) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDbClustersForLocalNode(OHazelcastPlugin.java:1293) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1127) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin$2.call(OHazelcastPlugin.java:1124) at com.orientechnologies.orient.server.hazelcast.OHazelcastDistributedDatabase.configureDatabase(OHazelcastDistributedDatabase.java:239) ... 8 more Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: Error on creating cluster on distributed nodes: local and remote ids assigned are different at com.orientechnologies.orient.server.distributed.ODistributedStorage.addCluster(ODistributedStorage.java:1364) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.addCluster(ODatabaseDocumentTx.java:1291) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createClusterIfNeeded(OClassImpl.java:2102) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.addCluster(OClassImpl.java:1068) at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installLocalClusterPerClass(OHazelcastPlugin.java:1617) ... 12 more

Is it an issue that was not fixed on this version?

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Aug 31, 2017

Facing issue -- Error on creating cluster 'lives' on distributed nodes: local and remote ids assigned are different
while creating edge. Dont know whether problem in api or in the config file.Please reply asap.
Using 2.2.26 version.

shubhamod1 commented Aug 31, 2017

Facing issue -- Error on creating cluster 'lives' on distributed nodes: local and remote ids assigned are different
while creating edge. Dont know whether problem in api or in the config file.Please reply asap.
Using 2.2.26 version.

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Aug 31, 2017

Member

Are you creating a class or cluster? If you try to re-execute the same command do you have the same error?

Member

lvca commented Aug 31, 2017

Are you creating a class or cluster? If you try to re-execute the same command do you have the same error?

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 1, 2017

I am creating clusters on distributed mode like this:

"autoDeploy": true,
"hotAlignment": false,
"readQuorum": 1,
"writeQuorum": "majority",
"executionMode": "undefined",
"readYourWrites": true,
"newNodeStrategy": "dynamic",
"clusters": {
"internal": {
},
"index": {
},
"client_1": {
"servers" : [ "1","2" ]
},
"client_2": {
"servers" : [ "2","3" ]
},
"client_3": {
"servers" : [ "3","1" ]
},
"*": {
"servers" : [ "<NEW_NODE>" ]
}
}
}

I am able to create nodes on the servers ,but while creating edge i get this error. I get the same error while re-executing the same command.
Command is -- OrientEdge e=graph.addEdge(null, vertex1, vertex2, "lives");

shubhamod1 commented Sep 1, 2017

I am creating clusters on distributed mode like this:

"autoDeploy": true,
"hotAlignment": false,
"readQuorum": 1,
"writeQuorum": "majority",
"executionMode": "undefined",
"readYourWrites": true,
"newNodeStrategy": "dynamic",
"clusters": {
"internal": {
},
"index": {
},
"client_1": {
"servers" : [ "1","2" ]
},
"client_2": {
"servers" : [ "2","3" ]
},
"client_3": {
"servers" : [ "3","1" ]
},
"*": {
"servers" : [ "<NEW_NODE>" ]
}
}
}

I am able to create nodes on the servers ,but while creating edge i get this error. I get the same error while re-executing the same command.
Command is -- OrientEdge e=graph.addEdge(null, vertex1, vertex2, "lives");

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Sep 1, 2017

Member

Please try to create the class "lives" beforehand with this command:

create class lives extends e

Member

lvca commented Sep 1, 2017

Please try to create the class "lives" beforehand with this command:

create class lives extends e

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 1, 2017

As per this documentation, "lives" is just a label and "null" is because i am creating edge extending the default class.

shubhamod1 commented Sep 1, 2017

As per this documentation, "lives" is just a label and "null" is because i am creating edge extending the default class.

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Sep 1, 2017

Member

In OrientDB, by default, edge labels are edge classes.

Member

lvca commented Sep 1, 2017

In OrientDB, by default, edge labels are edge classes.

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 1, 2017

okay.
So what is the java api command line for creating an edge type?

shubhamod1 commented Sep 1, 2017

okay.
So what is the java api command line for creating an edge type?

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Sep 1, 2017

Member

g.createEdgeType("lives");

Member

lvca commented Sep 1, 2017

g.createEdgeType("lives");

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 4, 2017

Hey lvca,
I did what you ask . I get this error now--
Error while removing cluster '-1'
Cluster id '-1' is outside the of range of configured clusters (0-25) in database 'graph'

please look at the code--
Vertex vAddress = graph.addVertex(null);
vAddress.setProperty("name", "P");
graph.commit();
System.out.println("Created vertex: ");

	Vertex vPerson = graph.addVertex(null);
	vPerson.setProperty("name", "Q");
	graph.commit();
	System.out.println("Created vertex: ");

	 Vertex vertex1=null;
	for (Vertex v : graph.getVertices("name", "P")) {
	    System.out.println("Found vertex: " + v);
	    vertex1=v;
	}
	Vertex vertex2=null;
	for (Vertex v : graph.getVertices("name", "P")) {
	    System.out.println("Found vertex: " + v);
	    vertex2=v;
	}
	graph.createEdgeType("lives");
	 Edge e = graph.addEdge(null, vertex1, vertex2, "lives");
	graph.commit();
	System.out.println("Created edge: ");

shubhamod1 commented Sep 4, 2017

Hey lvca,
I did what you ask . I get this error now--
Error while removing cluster '-1'
Cluster id '-1' is outside the of range of configured clusters (0-25) in database 'graph'

please look at the code--
Vertex vAddress = graph.addVertex(null);
vAddress.setProperty("name", "P");
graph.commit();
System.out.println("Created vertex: ");

	Vertex vPerson = graph.addVertex(null);
	vPerson.setProperty("name", "Q");
	graph.commit();
	System.out.println("Created vertex: ");

	 Vertex vertex1=null;
	for (Vertex v : graph.getVertices("name", "P")) {
	    System.out.println("Found vertex: " + v);
	    vertex1=v;
	}
	Vertex vertex2=null;
	for (Vertex v : graph.getVertices("name", "P")) {
	    System.out.println("Found vertex: " + v);
	    vertex2=v;
	}
	graph.createEdgeType("lives");
	 Edge e = graph.addEdge(null, vertex1, vertex2, "lives");
	graph.commit();
	System.out.println("Created edge: ");
@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 8, 2017

Hey ,
Its working now with 2 machine setup.
But i just want to know if there is any way we can speedup the process of creating nodes and edges.It is quite slow . like heap size or something .

shubhamod1 commented Sep 8, 2017

Hey ,
Its working now with 2 machine setup.
But i just want to know if there is any way we can speedup the process of creating nodes and edges.It is quite slow . like heap size or something .

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Sep 8, 2017

Member

What's the throughput? Are you connected remote? Are you grouping operations in transactions?

Member

lvca commented Sep 8, 2017

What's the throughput? Are you connected remote? Are you grouping operations in transactions?

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 8, 2017

I am just trying to create 100k dummy nodes with random connected edges. Yes i am connected remote.

shubhamod1 commented Sep 8, 2017

I am just trying to create 100k dummy nodes with random connected edges. Yes i am connected remote.

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Sep 8, 2017

Member

What about grouping in TXs?

Member

lvca commented Sep 8, 2017

What about grouping in TXs?

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 8, 2017

can you tell me how to do that? if you have any feature available like this?

shubhamod1 commented Sep 8, 2017

can you tell me how to do that? if you have any feature available like this?

@lvca

This comment has been minimized.

Show comment
Hide comment
@lvca

lvca Sep 8, 2017

Member

Just put like 1000 vertices/edges in one transaction. In this way, the message to the server will be 1, not 1000.

Member

lvca commented Sep 8, 2017

Just put like 1000 vertices/edges in one transaction. In this way, the message to the server will be 1, not 1000.

@shubhamod1

This comment has been minimized.

Show comment
Hide comment
@shubhamod1

shubhamod1 Sep 11, 2017

done that also.Initially speed is fine but later it depreciates and after some time the process stops.

shubhamod1 commented Sep 11, 2017

done that also.Initially speed is fine but later it depreciates and after some time the process stops.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment