A messaging system implementation which enables realtime Apache Solr indexing of stream data generated from incoming updates to a number of data sources inlcuding Labkey, Apache OODT and ISATools.
Data integration is processed and managed using Apache Kafka. This enables realtime updates, data consistency and integration across the above applications and servers.
Run mvn clean package from the top level directory. Inside dia-kafka-solr/target you will the following executables:
- Producers
- labkey-producer: java -jar ./target/labkey-producer.jar [--url ] [--user <user/email>] [--pass ] [--project ] [--wait ] [--kafka-topic <topic_name>] [--kafka-url]
- isatools-producer java -jar ./target/isatools-producer.jar [--tikaRESTURL ] [--isaToolsDir ] [--wait ] [--kafka-topic <topic_name>] [--kafka-url]
- Consumers
- solr-consumer java -jar ./target/solr-consumer.jar [--solr-url ] [--sorl-collection ] [--zoo-url ] [--project ] [--kafka-topic <topic_name>]
dia-kafka-solr is licensed under the Apache License v2.0