Neo4j Kafka Integrations
Switch branches/tags
Nothing to show
Clone or download
omarlarus Producer - CRUD NODE and Kafka routing configuration (#40)
* streams packages

* StreamsEventMetaBuilder implemented

* Add unit tests for beforeCommit

* streams packages

* StreamsEventMetaBuilder implemented

* Add unit tests for beforeCommit

* PreviousTransactionDataBuilder

* fix #17

* fixes #29 and #30

* #8 #9 #10 CRUD NODE

* fixes #27

* Filled transaction data into meta

* Refactoring

* Optimization

* moved RoutingConfigurationConstants into KafkaConfiguration

* Update relationship.created.json

* Update relationship.deleted.json

* Update relationship.updated.json

* Meta.source.hostname setted

* Fix before properties collecting
Latest commit 6599e5d Oct 17, 2018

readme.adoc

Neo4j Streaming Data Integrations

Description

The project is composed by 2 parts:

  • Neo4j Streams Producer: a transaction event handler events that sends data to a Kafka topic

  • Neo4j Streams Consumer: a Neo4j application that ingest data from Kafka topics into Neo4j via templated Cypher Statements

Installation

Build locally

mvn clean install
  1. Copy target/neo4j-kafka-*.jar into $NEO4J_HOME/plugins

  2. Restart Neo4j

Quick Configuration

You can set the following configuration values in your neo4j.conf, here are the defaults.

neo4j.conf
kafka.zookeeper.connect=localhost:2181
kafka.bootstrap.servers=localhost:9092
kafka.acks=1
kafka.num.partitions=1
kafka.retries=2
kafka.batch.size=16384
kafka.buffer.memory=33554432
kafka.reindex.batch.size=1000
kafka.session.timeout.ms=15000
kafka.connection.timeout.ms=10000
kafka.replication=1

kafka.routing.nodes.<TOPIC_NAME>=<PATTERN>

See the Apache Kafka documentation for details on these settings.

Patterns

To control which nodes are sent to Kafka, and which of their properties you can define node-patterns in the config.

You can chose Labels and properties for inclusion or exclusion, with * meaning all.

Patterns are separated by semicolons ;.

The basic syntax is:

Label{*};Label1{prop1, prop2};Label3{-prop1,-prop2}
pattern meaning

Label{*}

all nodes with this label with all their properties go to the related topic

Label1:Label2

nodes with these two labels are sent to the related topic

Label{prop1,prop2}

the prop1, prop2 of all nodes with this label are sent to the related topic

Label{-prop1,-prop2}

properties of the node with prop1, prop2 excluded, with the label are sent to the related topic

Testing

Following these instructions.

  1. Download & Unzip Kafka

  2. Run Zookeeper, server and a test consumer.

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic neo4j [--from-beginning]

Kafka Connect Neo4j Integration (by Confluent)

Source, Documentation and Support is currently not available from Confluent.

We haven’t checked configuration/scalability yet.

Download Link is a bit hard to find, here you go: Download

confluent-hub install confluentinc/kafka-connect-neo4j:1.0.0-preview