Kafka ingestion

[sirsean]

Backported support for Docker Compose and Kafka.

Using Kafka to read lines and save them into a MySQL database. (This was stolen directly from the Onyx template. Obviously we need to change how/where it gets stored.)

Using a syslog-ng relay that receives syslog lines and passes them along to Kafka.

docker-compose up will start the whole thing.

Before you can do anything else, you need to create the Kafka topic:

docker run --rm -it --link desdemona_kafka_1:kafka1 kafka bash -c "\$KAFKA_HOME/bin/kafka-console-producer.sh --topic test1 --broker-list=kafka1:9092"

(And then type anything. Or you could just send syslog to your Docker host on port 601.)

You also need to create the database table:

docker run -it --link desdemona_db_1:mysql --rm mysql sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'

use logs;
CREATE TABLE logLines (id int primary key auto_increment, line text);

And then you can submit the job!

ZOOKEEPER=$(echo $DOCKER_HOST|cut -d ':' -f 2|sed "s/\/\///g") lein run -m desdemona.jobs.sample-submit-job

And then whatever you send to syslog on that host should go through Kafka and appear in MySQL.

[reaperhulk]

cljfmt is beside itself with anger. Appease the cljfmt gods!

[lvh]

We should probably just get rid of this :)

[lvh]

master now makes kibit non-gating for the threading (#16) issue

[codecov-io]

Current coverage is 52.50%

Merging #21 into master will decrease coverage by -23.43% as of b3ba040

@@            master    #21   diff @@
=====================================
  Files           15     17     +2
  Stmts          399    280   -119
  Branches        26      5    -21
  Methods          0      0       
=====================================
- Hit            303    147   -156
+ Partial         26      5    -21
- Missed          70    128    +58

Review entire Coverage Diff as of b3ba040

Powered by Codecov. Updated on successful CI builds.

[lvh]

This PR has tabs all over it.

[reaperhulk]

classic golang party foul

[sirsean]

What's the Clojure-preferred indentation style? (And why doesn't cljfmt fix it.)

[lvh]

Well, obviously whatever clojure-mode does ;) I'll check into why cljfmt doesn't fix it. Typically it's 4 spaces roughly, except for defs and hanging indents, i.e.

(defn run-query
  ([query events]
   (run-query 1 query events))
  ([n-answers query events]
   (eval (gen-query n-answers query events))))

Lines 3 and 5 "hang" where 2 and 4 do.

[lvh]

cljfmt allegedly does fix indentation: https://github.com/weavejester/cljfmt#configuration

Perhaps it only fixes local style?

[lvh]

#23, #25 should add enough EditorConfig to fix this.

[lvh]

I'm not sure why both this and run_peers.sh exists. What's the difference?

[sirsean]

I think the cluster requires both of them in order to run. (But it seems like we don't need run-container.sh any more.)

[lvh]

Style nitpick: no superfluous commas

[lvh]

I am merging this despite the codecov failure; the code we care about is tested, the untested code comes straight from the template; we'll be addressing individual coverage improvements after this PR.