Permalink
Browse files

Support for HA cluster with neo4j, 1.9.X, #228, #99, #223

* Fixed for IRB/Pry console using HA to concurrently access the database
* Updated the HA examples
* Removed ZooKeeper and its configuration
  • Loading branch information...
1 parent f6b40a9 commit 5895b86addadcfbde2df0ef66078733038b3b388 @andreasronge andreasronge committed Dec 28, 2012
View
@@ -1,3 +1,11 @@
+== 2.2.3 / 2012-12-28
+* Support for HA cluster with neo4j 1.9.X, #228, #99, #223
+* Make sure the Identity map is cleared after an exception, #214
+* Relationship other_node should return wrapped node, #226
+* Automatically convert DateTimes to UTC, (neo4j-wrapper #7)
+* get_or_create should return a wrapped node (neo4j-wrapper #8)
+* Make it work with Neo4j 1.7.1 (neo4j-core, #19)
+
== 2.2.1 / 2012-12-18
* Fix for JRuby 1.7.1 and Equal #225
* Fix for create nodes and relationship using Cypher (neo4j-core #17)
@@ -24,79 +24,75 @@ lucene: { fulltext: { provider: lucene,
type: exact}
}
-# If migrations should be run in a new thread
-# If set to true it you can use the database while it is migrating.
-migration_thread: false
-
-# If online backup should be available, if it is the Online JAR file will be loaded,
-# Notice it must be either 'true' or 'false' as string
-enable_online_backup: 'false'
-
-#use the clustered Neo4j GraphDatabase (org.neo4j.kernel.HighlyAvailableGraphDatabase)
-ha.db: false
-
-# Example of HA Configuration, see http://wiki.neo4j.org/content/High_Availability_Cluster
-# This is only used when ha.db is set to true
-ha.server_id: 2
-ha.server: 'localhost:6002'
-ha.coordinators: 'localhost:2181,localhost:2182,localhost:2183'
-
-# if enabled you can use the bin/neo4j-shell command to access the database
-enable_remote_shell: "port=9332"
-
-#===Memory mapped I/O settings===
-
-#Each file in the Neo store can use memory mapped I/O for reading/writing.
-#Best performance is achived if the full file can be memory mapped but if
-#there isn't enough memory for that Neo will try and make the best use of
-#the memory it gets (regions of the file that get accessed often will more
-#likley be memory mapped).
-
-#For high traversal speed it is important to have the nodestore.db and
-#relationshipstore.db files.
-
-neostore.nodestore.db.mapped_memory: 25M
-neostore.relationshipstore.db.mapped_memory: 50M
-neostore.propertystore.db.mapped_memory: 90M
-neostore.propertystore.db.index.mapped_memory: 1M
-neostore.propertystore.db.index.keys.mapped_memory: 1M
-neostore.propertystore.db.strings.mapped_memory: 130M
-neostore.propertystore.db.arrays.mapped_memory: 130M
-
-
-#: ": ": "Cache settings: ": ": "
-
-#use adaptive caches YES|NO. Let Neo try make best use of available heap.
-use_adaptive_cache: YES
-
-#heap usage/max heap size ratio. Neo will increase caches while ratio
-#is less and decrease if greater. Default 0.77 seems to be a good over
-#all ratio of heap usage to avoid GC trashing. Larger heaps may allow for
-#a higher ratio while tiny heaps may need even less.
-adaptive_cache_heap_ratio: 0.77
-
-#how aggressive Neo will decrease caches once heap ratio reached
-adaptive_cache_manager_decrease_ratio: 1.15
-
-#how aggresive Neo will increase caches if ratio isn't yet reached
-adaptive_cache_manager_increase_ratio: 1.1
-
-#if no requests are made to Neo this is the amount of time in ms Neo will wait
-#before it checks the heap usage and adapts the caches if needed
-adaptive_cache_worker_sleep_time: 3000
-
-#minimum size (number of nodes) of node cache. If adaptive cache is in use
-#node cache will not be decreased under this value
-min_node_cache_size: 0
-
-#minimum size (number of relationships) of relationship cache. If adaptive
-#cache is in use relationship cache will not be decreased under this value
-min_relationship_cache_size: 0
-
-#maximum size (number of nodes) of node cache. If adaptive cache is not in
-#use the node cache will not be increased above this value
-max_node_cache_size: 1500
-
-#maximum size (number of relationship) of node cache. If adaptive cache is
-#not in use the relationship cache will not be increased above this value
-max_relationship_cache_size: 3500
+## If online backup should be available, if it is the Online JAR file will be loaded,
+## Notice it must be either 'true' or 'false' as string
+#enable_online_backup: 'false'
+#
+##use the clustered Neo4j GraphDatabase (org.neo4j.kernel.HighlyAvailableGraphDatabase)
+#ha.db: false
+#
+## Example of HA Configuration, see http://wiki.neo4j.org/content/High_Availability_Cluster
+## This is only used when ha.db is set to true
+#ha.server_id: 2
+#ha.server: 'localhost:6002'
+#ha.coordinators: 'localhost:2181,localhost:2182,localhost:2183'
+#
+## if enabled you can use the bin/neo4j-shell command to access the database
+#enable_remote_shell: "port=9332"
+#
+##===Memory mapped I/O settings===
+#
+##Each file in the Neo store can use memory mapped I/O for reading/writing.
+##Best performance is achived if the full file can be memory mapped but if
+##there isn't enough memory for that Neo will try and make the best use of
+##the memory it gets (regions of the file that get accessed often will more
+##likley be memory mapped).
+#
+##For high traversal speed it is important to have the nodestore.db and
+##relationshipstore.db files.
+#
+#neostore.nodestore.db.mapped_memory: 25M
+#neostore.relationshipstore.db.mapped_memory: 50M
+#neostore.propertystore.db.mapped_memory: 90M
+#neostore.propertystore.db.index.mapped_memory: 1M
+#neostore.propertystore.db.index.keys.mapped_memory: 1M
+#neostore.propertystore.db.strings.mapped_memory: 130M
+#neostore.propertystore.db.arrays.mapped_memory: 130M
+#
+#
+##: ": ": "Cache settings: ": ": "
+#
+##use adaptive caches YES|NO. Let Neo try make best use of available heap.
+#use_adaptive_cache: YES
+#
+##heap usage/max heap size ratio. Neo will increase caches while ratio
+##is less and decrease if greater. Default 0.77 seems to be a good over
+##all ratio of heap usage to avoid GC trashing. Larger heaps may allow for
+##a higher ratio while tiny heaps may need even less.
+#adaptive_cache_heap_ratio: 0.77
+#
+##how aggressive Neo will decrease caches once heap ratio reached
+#adaptive_cache_manager_decrease_ratio: 1.15
+#
+##how aggresive Neo will increase caches if ratio isn't yet reached
+#adaptive_cache_manager_increase_ratio: 1.1
+#
+##if no requests are made to Neo this is the amount of time in ms Neo will wait
+##before it checks the heap usage and adapts the caches if needed
+#adaptive_cache_worker_sleep_time: 3000
+#
+##minimum size (number of nodes) of node cache. If adaptive cache is in use
+##node cache will not be decreased under this value
+#min_node_cache_size: 0
+#
+##minimum size (number of relationships) of relationship cache. If adaptive
+##cache is in use relationship cache will not be decreased under this value
+#min_relationship_cache_size: 0
+#
+##maximum size (number of nodes) of node cache. If adaptive cache is not in
+##use the node cache will not be increased above this value
+#max_node_cache_size: 1500
+#
+##maximum size (number of relationship) of node cache. If adaptive cache is
+##not in use the relationship cache will not be increased above this value
+#max_relationship_cache_size: 3500
@@ -1,6 +1,7 @@
source 'http://rubygems.org'
gem 'neo4j', :path => "../.."
+#gem 'neo4j-core', :path => '../../../neo4j-core'
-gem 'neo4j-advanced', '1.8' # you might have to change this so that it is the same version as neo4j-community
-gem 'neo4j-enterprise', '1.8'
+#gem 'neo4j-advanced', '1.9.M03' # you might have to change this so that it is the same version as neo4j-community
+#gem 'neo4j-enterprise', '1.9.M03'
@@ -3,13 +3,7 @@ A simple High Availability Cluster Example
To run this:
-0. First you need the neo4j enterprise JAR files
- mkdir -p lib
- bundle install
- bundle exec ../../bin/neo4j-jars enterprise
-
-1. type: ./start_zookeeper.sh
-This starts 3 java processes (ps -ef | grep java)
+1. Nothing here :)
2. type: irb
@@ -32,13 +26,7 @@ This defines the method start which configures neo4j.rb database as a HA databas
11. type: Neo4j.ref_node.props.inspect
-Notice that you only can do modification slaves. You should not write to the master.
-The master machine is usually the machine that starts first.
-
-If you need to start with a new database, you should clean the zookeeper data folders, type: clean.sh
-
-NOTICE - there might be problem running zookeeper on Mac Os X, see https://groups.google.com/forum/?fromgroups=#!topic/neo4j/Kn_tHt3wPL0
-
-For more info, see the HA chapter in the Neo4j Manual, http://docs.neo4j.org/
+To check if which machine is master:
+ Neo4j.db.graph.isMaster()
@@ -1,6 +0,0 @@
-#!/bin/sh
-
-rm -rf data/zookeeper1/version-2
-rm -rf data/zookeeper2/version-2
-rm -rf data/zookeeper3/version-2
-rm -rf db
@@ -1,11 +0,0 @@
-#server1.cfg
-tickTime=2000
-initLimit=10
-syncLimit=5
-
-dataDir=data/zookeeper1
-clientPort=2181
-
-server.1=localhost:2888:3888
-server.2=localhost:2889:3889
-server.3=localhost:2890:3890
@@ -1,11 +0,0 @@
-#server2.cfg
-tickTime=2000
-initLimit=10
-syncLimit=5
-
-dataDir=data/zookeeper2
-clientPort=2182
-
-server.1=localhost:2888:3888
-server.2=localhost:2889:3889
-server.3=localhost:2890:3890
@@ -1,11 +0,0 @@
-#server3.cfg
-tickTime=2000
-initLimit=10
-syncLimit=5
-
-dataDir=data/zookeeper3
-clientPort=2183
-
-server.1=localhost:2888:3888
-server.2=localhost:2889:3889
-server.3=localhost:2890:3890
@@ -1,97 +0,0 @@
-#=== Neo4j.rb configuration settings
-
-# The folder location of the neo4j and lucene database
-storage_path: db
-
-# When using the Neo4j::Model you can let neo4j automatically set timestamps when updating/creating nodes.
-# If set to true neo4j.rb automatically timestamps create and update operations if the model has properties named created_at/created_on or updated_at/updated_on
-# (similar to ActiveRecord).
-timestamps: true
-
-# Configuration for lucene
-lucene: { fulltext: { provider: lucene,
- type: fulltext },
- exact: { provider: lucene,
- type: exact}
-}
-
-# If migrations should be run in a new thread
-# If set to true it you can use the database while it is migrating.
-migration_thread: false
-
-#use the clustered Neo4j GraphDatabase (org.neo4j.kernel.HighlyAvailableGraphDatabase)
-ha.db: true
-
-# If online backup should be available, if it is the Online JAR file will be loaded.
-online_backup_enabled: false
-
-# If enable neo4j.rb will create _all relationship to all instances inheriting from Neo4j::Rails::Model
-# If disabled all custom rules will also be unavailable.
-enable_rules: true
-
-
-# Example of HA Configuration, see http://wiki.neo4j.org/content/High_Availability_Cluster
-# This is only used when ha.db is set to true
-ha.server_id: 1
-ha.server: 'localhost:6001'
-ha.coordinators: 'localhost:2181,localhost:2182,localhost:2183'
-
-# if enabled you can use the bin/neo4j-shell command to access the database
-enable_remote_shell: "port=9331"
-
-#===Memory mapped I/O settings===
-
-#Each file in the Neo store can use memory mapped I/O for reading/writing.
-#Best performance is achived if the full file can be memory mapped but if
-#there isn't enough memory for that Neo will try and make the best use of
-#the memory it gets (regions of the file that get accessed often will more
-#likley be memory mapped).
-
-#For high traversal speed it is important to have the nodestore.db and
-#relationshipstore.db files.
-
-neostore.nodestore.db.mapped_memory: 25M
-neostore.relationshipstore.db.mapped_memory: 50M
-neostore.propertystore.db.mapped_memory: 90M
-neostore.propertystore.db.index.mapped_memory: 1M
-neostore.propertystore.db.index.keys.mapped_memory: 1M
-neostore.propertystore.db.strings.mapped_memory: 130M
-neostore.propertystore.db.arrays.mapped_memory: 130M
-
-
-#: ": ": "Cache settings: ": ": "
-
-#use adaptive caches YES|NO. Let Neo try make best use of available heap.
-use_adaptive_cache: YES
-
-#heap usage/max heap size ratio. Neo will increase caches while ratio
-#is less and decrease if greater. Default 0.77 seems to be a good over
-#all ratio of heap usage to avoid GC trashing. Larger heaps may allow for
-#a higher ratio while tiny heaps may need even less.
-adaptive_cache_heap_ratio: 0.77
-
-#how aggressive Neo will decrease caches once heap ratio reached
-adaptive_cache_manager_decrease_ratio: 1.15
-
-#how aggresive Neo will increase caches if ratio isn't yet reached
-adaptive_cache_manager_increase_ratio: 1.1
-
-#if no requests are made to Neo this is the amount of time in ms Neo will wait
-#before it checks the heap usage and adapts the caches if needed
-adaptive_cache_worker_sleep_time: 3000
-
-#minimum size (number of nodes) of node cache. If adaptive cache is in use
-#node cache will not be decreased under this value
-min_node_cache_size: 0
-
-#minimum size (number of relationships) of relationship cache. If adaptive
-#cache is in use relationship cache will not be decreased under this value
-min_relationship_cache_size: 0
-
-#maximum size (number of nodes) of node cache. If adaptive cache is not in
-#use the node cache will not be increased above this value
-max_node_cache_size: 1500
-
-#maximum size (number of relationship) of node cache. If adaptive cache is
-#not in use the relationship cache will not be increased above this value
-max_relationship_cache_size: 3500
@@ -1 +0,0 @@
-1
@@ -1 +0,0 @@
-2
@@ -1 +0,0 @@
-3
@@ -5,15 +5,18 @@
require 'neo4j-enterprise'
def start(machine_id)
- Neo4j::Config.default_file = 'config.yml' # use local config file
+ #Neo4j::Config.default_file = 'config.yml' # use local config file
# override this default config with this machine configuration
+ Neo4j.config['ha.db'] = true
Neo4j.config['ha.server_id'] = machine_id
Neo4j.config['ha.server'] = "localhost:600#{machine_id}"
- Neo4j.config['enable_remote_shell'] = "port=933#{machine_id}"
- # since we are all running on localhost we can only enable one online back up server
- # since they all use the port 6362
- Neo4j.config[:online_backup_enabled] = true if machine_id == 1
+ Neo4j.config['ha.pull_interval'] = '500ms'
+ Neo4j.config['ha.discovery.enabled'] = false
+ other_machines = [1,2,3].map{|id| "localhost:500#{id}"}.join(',')
+ puts "ha.initial_hosts: #{other_machines}"
+ Neo4j.config['ha.initial_hosts'] = other_machines
+ Neo4j.config['ha.cluster_server'] = "localhost:500#{machine_id}"
Neo4j.config[:storage_path] = "db/neo#{machine_id}"
Neo4j.start
@@ -1,19 +0,0 @@
-#!/bin/sh
-
-# Only follow symlinks if readlink supports it
-if readlink -f "$0" > /dev/null 2>&1
-then
- ZOOBIN=`readlink -f "$0"`
-else
- ZOOBIN="$0"
-fi
-
-ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"
-
-ZOOBINDIR=`dirname "$ZOOBIN"`
-JARDIR=`cd "${ZOOBINDIR}/lib"; pwd`
-
-CLASSPATH=$JARDIR/log4j-1.2.16.jar:$JARDIR/zookeeper-3.3.2.jar
-java -cp "$CLASSPATH" $ZOOMAIN conf/server1.cfg &
-java -cp "$CLASSPATH" $ZOOMAIN conf/server2.cfg &
-java -cp "$CLASSPATH" $ZOOMAIN conf/server3.cfg &
Oops, something went wrong.

0 comments on commit 5895b86

Please sign in to comment.