202 - schema registry validation

[mmacai]

Description

• fixed schema registry validation for kafka consumer
• updated avro docs
• fixed changelog
• RIG will now raise an error when kafka event is binary and schema registry is not set - right now defaults to nil
• updated Avro docs to use docker instead of mix

Closes #202

Test

• integration and unit tests
• example from Avro docs
• sending binary message using only KAFKA_BROKERS config should raise an error, but not restart the consumer process

[kevinbader]

Integration tests fail on my machine. Tried multiple runs; this came up every time:

  1) test Given a started RigKafka client, messages can be produced and consumed. (RigKafkaTest)
     test/rig_kafka_test.exs:42
     No message matching :test_message_received after 10000ms.
     The process mailbox is empty.
     code: assert_receive :test_message_received, 10_000
     stacktrace:
       test/rig_kafka_test.exs:62: (test)

and this only during the first run:

 1) test Given response_from is set to Kafka, the http response is taken from the Kafka response topic instead of forwarding the backend's original response. (RigTests.Proxy.ResponseFrom.KafkaTest)
     test/proxy/response_from/kafka_test.exs:47
     ** (HTTPoison.Error) :timeout
     code: %HTTPoison.Response{status_code: res_status, body: res_body} = HTTPoison.get!(request_url)
     stacktrace:
       (httpoison) lib/httpoison.ex:128: HTTPoison.request!/5
       test/proxy/response_from/kafka_test.exs:109: (test)

[kevinbader]

works okay, however, below there's this:

In terminal you should see something like below

and I don't see this, partly because docker runs the container in detached mode, but even with docker logs -f rig I don't see the line as described. Does this mean it doesn't work as expected?

[mmacai]

Are you sure? Just tested it and I can see:

11:05:48.880 [debug] Decoded Avro message="{\"example\":\"test\"}"
application=rig_kafka module=RigKafka.Avro function=decode/2 file=lib/rig_kafka/avro.ex line=29 pid=<0.1935.0>

11:05:48.882 [debug] [:start_object, {:string, "contenttype"}, :colon, {:string, "avro/binary"}, :comma, {:string, "data"}, :colon, :start_object, {:string, "example"}, :colon, {:string, "test"}, :end_object, :comma, {:string, "id"}, :colon, {:string, "069711bf-3946-4661-984f-c667657b8d85"}, :comma, {:string, "rig"}, :colon, :start_object, {:string, "correlation"}, :colon, {:string, "g2dkAA1yaWdAMTI3LjAuMC4xAAAI7AAAAAAB"}, :comma, {:string, "headers"}, :colon, :start_array, :start_array, {:string, "accept"}, :comma, {:string, "*/*"}, :end_array, :comma, :start_array, {:string, "content-length"}, :comma, {:string, "221"}, :end_array, :comma, :start_array, {:string, "content-type"}, :comma, {:string, "application/json"}, :end_array, :comma, :start_array, {:string, "host"}, :comma, {:string, ...}, :end_array, ...]
application=rig module=Rig.EventStream.KafkaToFilter function=kafka_handler/1 file=lib/rig/event_stream/kafka_to_filter.ex line=20 pid=<0.1935.0>

[mmacai]

Integration tests fail on my machine. Tried multiple runs; this came up every time:

  1) test Given a started RigKafka client, messages can be produced and consumed. (RigKafkaTest)
     test/rig_kafka_test.exs:42
     No message matching :test_message_received after 10000ms.
     The process mailbox is empty.
     code: assert_receive :test_message_received, 10_000
     stacktrace:
       test/rig_kafka_test.exs:62: (test)

and this only during the first run:

 1) test Given response_from is set to Kafka, the http response is taken from the Kafka response topic instead of forwarding the backend's original response. (RigTests.Proxy.ResponseFrom.KafkaTest)
     test/proxy/response_from/kafka_test.exs:47
     ** (HTTPoison.Error) :timeout
     code: %HTTPoison.Response{status_code: res_status, body: res_body} = HTTPoison.get!(request_url)
     stacktrace:
       (httpoison) lib/httpoison.ex:128: HTTPoison.request!/5
       test/proxy/response_from/kafka_test.exs:109: (test)

First one fails even on master, it's tracked in #201. Second one can't confirm, but maybe caused by the same thing as first one.