Kafka FaaS Connector

The Kafka FaaS Connector is used by MICO to route messages from Kafka in the CloudEvents format to an OpenFaaS function.

Requirements

Kafka FaaS Connector requires Kafka and OpenFaaS.

Kubernetes

Run Kafka, ZooKeeper and OpenFaaS in your Kubernetes cluster. You find some notes about Kafka in our documentation.

Modify kafka-faas-connector.yml to set the correct URLs of the OpenFaaS gateway (default: http://gateway.openfaas:8080) and the Kafka servers (default: bootstrap.kafka:9092).

Local execution

Launch Kafka and ZooKeeper locally. Either start OpenFaaS also locally or establish a connection to an already running instance (e.g. via port forwarding).

docker-compose:

docker-compose up kafka zookeeper

Port forwarding to OpenFaaS:

kubectl port-forward svc/gateway -n openfaas 31112:8080

Usage

Build and push the Docker image:

docker build -t ustmico/kafka-faas-connector . && docker push ustmico/kafka-faas-connector

Kubernetes:

Deployment:

kubectl apply -f kafka-faas-connector.yml

Local execution:

docker-compose:

docker-compose up --build kafka-faas-connector

Misc

Actuator get configuration properties:

kubectl port-forward svc/kafka-faas-connector -n mico-workspace 8080
curl localhost:8080/actuator/configprops | jq . > configmaps.json

Get the Kubernetes logs:

kubectl -n $NAMESPACE logs -f $(kubectl get pods -n $NAMESPACE --selector=run=kafka-faas-connector --output=jsonpath={.items..metadata.name})

Restart it by deleting the Kubernetes pod:

kubectl -n $NAMESPACE delete pod $(kubectl get pods -n $NAMESPACE --selector=run=kafka-faas-connector --output=jsonpath={.items..metadata.name})

Produce message to topic 'transform-request' locally:

docker exec -it kafka /bin/bash
/opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic transform-request

Consume topic 'transform-result' locally:

docker exec -it kafka /bin/bash
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --group mico --topic transform-result

Environment Variables

Variable	Description	Default Value
APPLICATION_NAME	The spring application name	kafka-faas-connector
APPLICATION_PORT	The server port	8080
LOGGING_LEVEL_KAFKAFAASCONNECTOR	Logging level of the Kafka-FaaS-Connector	DEBUG
LOGGING_LEVEL_KAFKA_CONSUMER_INFO	Logging level of the kafka consumer	INFO
KAFKA_BOOTSTRAP_SERVERS	The URLs of the bootstrap servers	localhost:9092
KAFKA_GROUP_ID	The group id for kafka	mico
KAFKA_TOPIC_INPUT	The topic, on which the Kafka-Faas-Connector receives messages	transform-request
KAFKA_TOPIC_OUTPUT	The topic, to which the kafka-faas-connector forwards the processed messages	transform-result
KAFKA_TOPIC_INVALID_MESSAGE	The topic, to which invalid messages are forwarded	InvalidMessage
KAFKA_TOPIC_DEAD_LETTER	The topic, to which messages are forwarded that can't/shouldn't be delivered	DeadLetter
KAFKA_TOPIC_TEST_MESSAGE_OUTPUT	The topic, to which test messages are forwarded	TestMessagesOutput
OPENFAAS_GATEWAY	The URL of the OpenFaaS gateway	http://127.0.0.1:8080
OPENFAAS_FUNCTION_NAME	The name of the OpenFaaS function that shall be used for processing the messages	faas-message-transformer