Skip to content
Permalink
Browse files
[FLINK-21865] Add a Docker Compose greeter example
This closes #4.
  • Loading branch information
tzulitai committed Mar 22, 2021
1 parent ebe2bdb commit dce2fa7db176039a43612f966a3536a8a72edc32
Showing 15 changed files with 882 additions and 0 deletions.
@@ -0,0 +1,36 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Build the functions code ...
FROM maven:3.6.3-jdk-11 AS builder
COPY src /usr/src/app/src
COPY pom.xml /usr/src/app
# TODO remove these commented lines and the jar; this is needed now only because we don't have the latest Java SDK published to Maven central yet
# COPY statefun-sdk-java-2.3-SNAPSHOT.jar /usr/src/app
# RUN mvn install:install-file \
# -Dfile=/usr/src/app/statefun-sdk-java-2.3-SNAPSHOT.jar \
# -DgroupId=org.apache.flink \
# -DartifactId=statefun-sdk-java \
# -Dversion=2.3-SNAPSHOT \
# -Dpackaging=jar \
# -DgeneratePom=true
RUN mvn -f /usr/src/app/pom.xml clean package

# ... and run the web server!
FROM openjdk:8
WORKDIR /
COPY --from=builder /usr/src/app/target/greeter-functions-app*jar-with-dependencies.jar greeter-functions-app.jar
EXPOSE 1108
CMD java -jar greeter-functions-app.jar
@@ -0,0 +1,74 @@
# Greeter Example with Docker Compose

This example is intended as a follow-up after completion of the [Java SDK Showcase Tutorial](../showcase). If you're
already familiar with the Java SDK fundamentals and would like to get a better understanding of how a realistic StateFun
application looks like, then you're in the right place! Otherwise, we highly suggest taking a look at the Showcase
tutorial first.

This example works with Docker Compose, and runs a few services that build up an end-to-end StateFun application:
- Functions service that runs your functions and expose them through an HTTP endpoint.
- StateFun runtime processes (a manager plus workers) that will handle ingress, egress, and inter-function messages as
well as function state storage in a consistent and fault-tolerant manner.
- Apache Kafka broker for the application ingress and egress. StateFun currently natively supports AWS Kinesis as well,
and you can also extend to connect with other systems.

To motivate this example, we'll implement a simple user greeter application, which has two functions - a `UserFn` that
expects `UserLogin` JSON events from an ingress and keeps in state storage information about users, and a `GreetingsFn`
that accepts user information to generate personalized greeting messages that are sent to users via an egress.

## Directory structure

- `src/`, `pom.xml` and `Dockerfile`: These files and directories are the contents of a Java Maven project which builds
our functions service, hosting the `UserFn` and `UserLogin` behind a HTTP endpoint. Check out the source code under
`src/main/java`. The `Dockerfile` is used to build a Docker image for our functions service.
- `user-logins.txt`: A file with multiple JSON objects per line; this is used as test events produced to our application ingress.
- `module.yaml`: The [Module Specification]() file to be mounted to the StateFun runtime process containers. This
configures a few things for a StateFun application, such as the service endpoints of the application's functions, as
well as definitions of [Ingresses and Egresses]() which the application will use.
- `docker-compose.yml`: Docker Compose file to spin up everything.

## Prerequisites

- Docker
- Docker Compose

## Running the example

First, lets build the example. From this directory, execute:

```
$ docker-compose build
```

This pulls all the necessary Docker images (StateFun and Kafka), and also builds the functions service image. This can
take a few minutes as it also needs to build the function's Java project.

Afterward the build completes, start running all the services:

```
$ docker-compose run
```

## Play around!

You can take a look at what messages are being sent to the Kafka egress:

```
$ docker-compose exec kafka kafka-console-consumer \
--bootstrap-server kafka:9092 \
--topic greetings \
--from-beginning
```

You can also try modifying the function code in the `src/main/java` directory, and do a zero-downtime upgrade of the
functions. Some ideas you can try out:
- Add some more state to be persisted by the `UserFn`. For example, let it additionally keep track of the user's previous login location.
- Or maybe just simply changing the greetings message generated by the `GreetingsFn`!

After you've finished changing the function code, you can do a hot redeploy of your functions service:

```
$ docker-compose up -d --build greeter-functions
```

This rebuilds the functions service image with the updated code, and restarts the service with the new image.
@@ -0,0 +1,110 @@
################################################################################
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
version: "2.1"

services:

###############################################################
# Functions service
###############################################################

greeter-functions:
build:
dockerfile: Dockerfile
context: .
expose:
- "1108"

###############################################################
# StateFun runtime
###############################################################

statefun-manager:
image: flink-statefun:2.3-SNAPSHOT
expose:
- "6123"
ports:
- "8081:8081"
environment:
ROLE: master
MASTER_HOST: statefun-manager
volumes:
- ./module.yaml:/opt/statefun/modules/greeter/module.yaml

statefun-worker:
image: flink-statefun:2.3-SNAPSHOT
expose:
- "6121"
- "6122"
depends_on:
- statefun-manager
- kafka
- greeter-functions
links:
- "statefun-manager:statefun-manager"
- "kafka:kafka"
- "greeter-functions:greeter-functions"
environment:
ROLE: worker
MASTER_HOST: statefun-manager
volumes:
- ./module.yaml:/opt/statefun/modules/greeter/module.yaml

###############################################################
# Kafka for ingress and egress
###############################################################

zookeeper:
image: confluentinc/cp-zookeeper:5.4.3
environment:
ZOOKEEPER_CLIENT_PORT: "2181"
ports:
- "2181:2181"

kafka:
image: confluentinc/cp-kafka:5.4.3
ports:
- "9092:9092"
depends_on:
- zookeeper
links:
- "zookeeper:zookeeper"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

###############################################################
# Simple Kafka JSON producer to simulate ingress events
###############################################################

user-logins-producer:
image: ververica/statefun-playground-producer:latest
depends_on:
- kafka
- statefun-worker
links:
- "kafka:kafka"
environment:
APP_PATH: /mnt/user-logins.txt
APP_KAFKA_HOST: kafka:9092
APP_KAFKA_TOPIC: user-logins
APP_JSON_PATH: user_id
volumes:
- ./user-logins.txt:/mnt/user-logins.txt
@@ -0,0 +1,53 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

version: "3.0"

module:
meta:
type: remote
spec:
endpoints:
- endpoint:
meta:
kind: http
spec:
typename:
namespace: greeter.fns
urlPathTemplate: http://greeter-functions:1108/
ingresses:
- ingress:
meta:
type: statefun.kafka.io/routable-protobuf-ingress
id: greeter.io/user-logins
spec:
address: kafka:9092
consumerGroupId: greeter
startupPosition:
type: earliest
topics:
- topic: user-logins
typeUrl: greeter.types/org.apache.flink.statefun.playground.java.greeter.types.UserLogin
targets:
- greeter.fns/user
egresses:
- egress:
meta:
type: statefun.kafka.io/generic-egress
id: greeter.io/user-greetings
spec:
address: kafka:9092
deliverySemantic:
type: at-least-once

0 comments on commit dce2fa7

Please sign in to comment.