Neo4j on Kubernetes

This repository contains a set of scripts that you can use to deploy a Neo4j cluster on Kubernetes. It will work for Neo4j versions 3.1 and higher.

You can either setup Kubernetes locally using minikube or on one of the cloud providers e.g. Google Compute Engine

Once you’ve got that setup you can run the following command to see where Kubernetes is running. e.g. on my machine

$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://192.168.99.100:8443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

All good so far. Now we’re ready to install Neo4j.

Core Servers

First let’s install the Core servers. We’d typically have a small number of Core servers in a cluster and they can handle both read and write traffic.

Run the following commands to create the Core servers:

$ kubectl apply -f cores
service "neo4j" configured
statefulset "neo4j-core" created

We can check that those servers have spun up by running:

$ kubectl get pods
NAME           READY     STATUS    RESTARTS   AGE
neo4j-core-0   1/1       Running   0          23s
neo4j-core-1   1/1       Running   0          19s
neo4j-core-2   1/1       Running   0          16s

And we can check that Neo4j is up and running by tailing the logs of one of the servers until we see the following:

$ kubectl logs -f neo4j-core-2
Starting Neo4j.
2017-09-13 09:39:31.406+0000 INFO  ======== Neo4j 3.2.3 ========
2017-09-13 09:39:31.824+0000 INFO  Starting...
2017-09-13 09:39:55.741+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2017-09-13 09:39:56.050+0000 INFO  Initiating metrics...
2017-09-13 09:39:58.842+0000 INFO  Resolved initial host 'neo4j.default.svc.cluster.local:5000' to [172.17.0.4:5000, 172.17.0.5:5000, 172.17.0.6:5000]
2017-09-13 09:39:59.976+0000 INFO  My connection info: [
	Discovery:   listen=0.0.0.0:5000, advertised=neo4j-core-0.neo4j.default.svc.cluster.local:5000,
	Transaction: listen=0.0.0.0:6000, advertised=neo4j-core-0.neo4j.default.svc.cluster.local:6000,
	Raft:        listen=0.0.0.0:7000, advertised=neo4j-core-0.neo4j.default.svc.cluster.local:7000,
	Client Connector Addresses: bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687,http://neo4j-core-0.neo4j.default.svc.cluster.local:7474,https://neo4j-core-0.neo4j.default.svc.cluster.local:7473
]
2017-09-13 09:39:59.977+0000 INFO  Discovering cluster with initial members: [neo4j.default.svc.cluster.local:5000]
2017-09-13 09:39:59.977+0000 INFO  Attempting to connect to the other cluster members before continuing...
2017-09-13 09:41:28.430+0000 INFO  Started.
2017-09-13 09:41:29.911+0000 INFO  Mounted REST API at: /db/manage
2017-09-13 09:41:39.562+0000 INFO  Remote interface available at http://neo4j-core-2.neo4j.default.svc.cluster.local:7474/

Neo4j also exposes the cluster topology via a procedure, which we can call like this:

$ kubectl exec neo4j-core-0 -- bin/cypher-shell --format verbose "CALL dbms.cluster.overview()"
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                                                                                                                                                             | role       | groups |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "719fa587-68e4-4194-bc61-8a35476a0af5" | ["bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-0.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-0.neo4j.default.svc.cluster.local:7473"] | "LEADER"   | []     |
| "bb057924-f304-4f6d-b726-b6368c8ac0f1" | ["bolt://neo4j-core-1.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-1.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-1.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER" | []     |
| "f84e7e0d-de6c-480e-8981-dad114de08cf" | ["bolt://neo4j-core-2.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-2.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-2.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER" | []     |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

3 rows available after 3 ms, consumed after another 4 ms

Read Replicas

Now that we’ve got the Core Servers up and running let’s add some Read Replicas. Read Replicas are used to scale out reads and they don’t accept any write operations.

Run the following command to create a single Read Replica:

$ kubectl apply -f read-replicas
deployment "neo4j-replica" created

If we query for the list of pods now we’ll see that it’s been added to the list:

$ kubectl get pods
NAME                             READY     STATUS    RESTARTS   AGE
neo4j-core-0                     1/1       Running   0          5m
neo4j-core-1                     1/1       Running   0          5m
neo4j-core-2                     1/1       Running   0          5m
neo4j-replica-3562563584-sckr8   1/1       Running   0          3s

Let’s check if the replica has joined the cluster and is ready to go:

$ kubectl logs -f neo4j-replica-3562563584-sckr8
2017-09-13 09:44:31.804+0000 INFO  ======== Neo4j 3.2.3 ========
2017-09-13 09:44:32.042+0000 INFO  Starting...
2017-09-13 09:44:41.555+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2017-09-13 09:44:41.675+0000 INFO  Initiating metrics...
2017-09-13 09:44:42.402+0000 INFO  Resolved initial host 'neo4j.default.svc.cluster.local:5000' to [172.17.0.4:5000, 172.17.0.5:5000, 172.17.0.6:5000]
2017-09-13 09:45:23.073+0000 INFO  Started.
2017-09-13 09:45:24.440+0000 INFO  Mounted REST API at: /db/manage
2017-09-13 09:45:30.631+0000 INFO  Remote interface available at http://neo4j-replica-3562563584-sckr8:7474/

Yep, looks good!

Let’s check that Neo4j knows about our new server:

$ kubectl exec neo4j-core-0 -- bin/cypher-shell --format verbose "CALL dbms.cluster.overview()"
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                                                                                                                                                             | role           | groups |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "719fa587-68e4-4194-bc61-8a35476a0af5" | ["bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-0.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-0.neo4j.default.svc.cluster.local:7473"] | "LEADER"       | []     |
| "bb057924-f304-4f6d-b726-b6368c8ac0f1" | ["bolt://neo4j-core-1.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-1.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-1.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "f84e7e0d-de6c-480e-8981-dad114de08cf" | ["bolt://neo4j-core-2.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-2.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-2.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "df1f9885-3197-4546-a163-0e6a357df0c5" | ["bolt://neo4j-replica-3562563584-sckr8:7687", "http://neo4j-replica-3562563584-sckr8:7474", "https://neo4j-replica-3562563584-sckr8:7473"]                                           | "READ_REPLICA" | []     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

4 rows available after 2 ms, consumed after another 4 ms

It does indeed.

Scaling the cluster

Now let’s scale up to 3 read replicas.

Run the following command:

$ kubectl scale deployment neo4j-replica --replicas=3
deployment "neo4j-replica" scaled

And give it a few seconds and Neo4j will know about those servers as well:

$ kubectl exec neo4j-core-0 -- bin/cypher-shell --format verbose "CALL dbms.cluster.overview()"
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                                                                                                                                                             | role           | groups |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "719fa587-68e4-4194-bc61-8a35476a0af5" | ["bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-0.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-0.neo4j.default.svc.cluster.local:7473"] | "LEADER"       | []     |
| "bb057924-f304-4f6d-b726-b6368c8ac0f1" | ["bolt://neo4j-core-1.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-1.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-1.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "f84e7e0d-de6c-480e-8981-dad114de08cf" | ["bolt://neo4j-core-2.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-2.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-2.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "d8c92a76-82c9-4146-9742-ce422e66cd08" | ["bolt://neo4j-replica-3562563584-7qs3x:7687", "http://neo4j-replica-3562563584-7qs3x:7474", "https://neo4j-replica-3562563584-7qs3x:7473"]                                           | "READ_REPLICA" | []     |
| "dac490ed-3ad7-4c86-b22f-41e0a429dce2" | ["bolt://neo4j-replica-3562563584-c8sqd:7687", "http://neo4j-replica-3562563584-c8sqd:7474", "https://neo4j-replica-3562563584-c8sqd:7473"]                                           | "READ_REPLICA" | []     |
| "df1f9885-3197-4546-a163-0e6a357df0c5" | ["bolt://neo4j-replica-3562563584-sckr8:7687", "http://neo4j-replica-3562563584-sckr8:7474", "https://neo4j-replica-3562563584-sckr8:7473"]                                           | "READ_REPLICA" | []     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

6 rows available after 19 ms, consumed after another 2 ms

Great, that worked.

We can also scale the core servers by running the following command:

$ kubectl scale statefulsets neo4j-core --replicas 5
statefulset "neo4j-core" scaled

And we can check that those were added:

$ kubectl exec neo4j-core-0 -- bin/cypher-shell --format verbose "CALL dbms.cluster.overview()"
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                                                                                                                                                             | role           | groups |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "719fa587-68e4-4194-bc61-8a35476a0af5" | ["bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-0.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-0.neo4j.default.svc.cluster.local:7473"] | "LEADER"       | []     |
| "bb057924-f304-4f6d-b726-b6368c8ac0f1" | ["bolt://neo4j-core-1.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-1.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-1.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "f84e7e0d-de6c-480e-8981-dad114de08cf" | ["bolt://neo4j-core-2.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-2.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-2.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "c4252e52-b4a4-4e88-8e1f-4773989c555a" | ["bolt://neo4j-core-3.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-3.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-3.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "fca4057a-bd8b-4b16-a439-0cbce12ffe53" | ["bolt://neo4j-core-4.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-4.neo4j.default.svc.cluster.local:7474", "https://neo4j-core-4.neo4j.default.svc.cluster.local:7473"] | "FOLLOWER"     | []     |
| "d8c92a76-82c9-4146-9742-ce422e66cd08" | ["bolt://neo4j-replica-3562563584-7qs3x:7687", "http://neo4j-replica-3562563584-7qs3x:7474", "https://neo4j-replica-3562563584-7qs3x:7473"]                                           | "READ_REPLICA" | []     |
| "dac490ed-3ad7-4c86-b22f-41e0a429dce2" | ["bolt://neo4j-replica-3562563584-c8sqd:7687", "http://neo4j-replica-3562563584-c8sqd:7474", "https://neo4j-replica-3562563584-c8sqd:7473"]                                           | "READ_REPLICA" | []     |
| "df1f9885-3197-4546-a163-0e6a357df0c5" | ["bolt://neo4j-replica-3562563584-sckr8:7687", "http://neo4j-replica-3562563584-sckr8:7474", "https://neo4j-replica-3562563584-sckr8:7473"]                                           | "READ_REPLICA" | []     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

8 rows available after 6 ms, consumed after another 3 ms

Success!

Neo4j Browser

If we want to interact with the machines in the cluster using the Neo4j browser one way to do this is to use port forwarding. Run the following command:

kubectl port-forward neo4j-core-0 8474:7447 8687:7687

This will forward ports on the neo4j-core-0 pod:

• 7474 → 8474 on your machine

• 7687 → 8687 on your machine

Open http://localhost:8474 and type :server connect if it doesn’t do it automatically. We then need to update the Host field to be bolt://localhost:8687 and fill in a username and password if applicable.

Security

By default we’ve disabled security to make it easier to play around with the cluster. Once you’re ready to deploy to production you’ll want to protect your servers, which you can do by removing the NEO4J_dbms_security_auth__enabled environment variable in cores/statefulset.yaml and read-replicas/deployment.yaml

Running Neo4j on Helm

Helm is a tool that streamlines installing and managing Kubernetes applications. You can think of it as an App Store for Kubernetes.

We created a Neo4j Helm package and it was recently accepted into the Helm incubator.

You can deploy Neo4j on Kubernetes using the Helm package by running the following commands:

helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
helm install incubator/neo4j --name neo-helm

This will create a 3 core cluster. You can read more about this in our blog post.