Permalink
Show file tree
Hide file tree
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
[FLINK-21865] Add a Docker Compose greeter example
This closes #4.
- Loading branch information
Showing
15 changed files
with
882 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -0,0 +1,36 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
# Build the functions code ... | ||
FROM maven:3.6.3-jdk-11 AS builder | ||
COPY src /usr/src/app/src | ||
COPY pom.xml /usr/src/app | ||
# TODO remove these commented lines and the jar; this is needed now only because we don't have the latest Java SDK published to Maven central yet | ||
# COPY statefun-sdk-java-2.3-SNAPSHOT.jar /usr/src/app | ||
# RUN mvn install:install-file \ | ||
# -Dfile=/usr/src/app/statefun-sdk-java-2.3-SNAPSHOT.jar \ | ||
# -DgroupId=org.apache.flink \ | ||
# -DartifactId=statefun-sdk-java \ | ||
# -Dversion=2.3-SNAPSHOT \ | ||
# -Dpackaging=jar \ | ||
# -DgeneratePom=true | ||
RUN mvn -f /usr/src/app/pom.xml clean package | ||
|
||
# ... and run the web server! | ||
FROM openjdk:8 | ||
WORKDIR / | ||
COPY --from=builder /usr/src/app/target/greeter-functions-app*jar-with-dependencies.jar greeter-functions-app.jar | ||
EXPOSE 1108 | ||
CMD java -jar greeter-functions-app.jar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -0,0 +1,74 @@ | ||
# Greeter Example with Docker Compose | ||
|
||
This example is intended as a follow-up after completion of the [Java SDK Showcase Tutorial](../showcase). If you're | ||
already familiar with the Java SDK fundamentals and would like to get a better understanding of how a realistic StateFun | ||
application looks like, then you're in the right place! Otherwise, we highly suggest taking a look at the Showcase | ||
tutorial first. | ||
|
||
This example works with Docker Compose, and runs a few services that build up an end-to-end StateFun application: | ||
- Functions service that runs your functions and expose them through an HTTP endpoint. | ||
- StateFun runtime processes (a manager plus workers) that will handle ingress, egress, and inter-function messages as | ||
well as function state storage in a consistent and fault-tolerant manner. | ||
- Apache Kafka broker for the application ingress and egress. StateFun currently natively supports AWS Kinesis as well, | ||
and you can also extend to connect with other systems. | ||
|
||
To motivate this example, we'll implement a simple user greeter application, which has two functions - a `UserFn` that | ||
expects `UserLogin` JSON events from an ingress and keeps in state storage information about users, and a `GreetingsFn` | ||
that accepts user information to generate personalized greeting messages that are sent to users via an egress. | ||
|
||
## Directory structure | ||
|
||
- `src/`, `pom.xml` and `Dockerfile`: These files and directories are the contents of a Java Maven project which builds | ||
our functions service, hosting the `UserFn` and `UserLogin` behind a HTTP endpoint. Check out the source code under | ||
`src/main/java`. The `Dockerfile` is used to build a Docker image for our functions service. | ||
- `user-logins.txt`: A file with multiple JSON objects per line; this is used as test events produced to our application ingress. | ||
- `module.yaml`: The [Module Specification]() file to be mounted to the StateFun runtime process containers. This | ||
configures a few things for a StateFun application, such as the service endpoints of the application's functions, as | ||
well as definitions of [Ingresses and Egresses]() which the application will use. | ||
- `docker-compose.yml`: Docker Compose file to spin up everything. | ||
|
||
## Prerequisites | ||
|
||
- Docker | ||
- Docker Compose | ||
|
||
## Running the example | ||
|
||
First, lets build the example. From this directory, execute: | ||
|
||
``` | ||
$ docker-compose build | ||
``` | ||
|
||
This pulls all the necessary Docker images (StateFun and Kafka), and also builds the functions service image. This can | ||
take a few minutes as it also needs to build the function's Java project. | ||
|
||
Afterward the build completes, start running all the services: | ||
|
||
``` | ||
$ docker-compose run | ||
``` | ||
|
||
## Play around! | ||
|
||
You can take a look at what messages are being sent to the Kafka egress: | ||
|
||
``` | ||
$ docker-compose exec kafka kafka-console-consumer \ | ||
--bootstrap-server kafka:9092 \ | ||
--topic greetings \ | ||
--from-beginning | ||
``` | ||
|
||
You can also try modifying the function code in the `src/main/java` directory, and do a zero-downtime upgrade of the | ||
functions. Some ideas you can try out: | ||
- Add some more state to be persisted by the `UserFn`. For example, let it additionally keep track of the user's previous login location. | ||
- Or maybe just simply changing the greetings message generated by the `GreetingsFn`! | ||
|
||
After you've finished changing the function code, you can do a hot redeploy of your functions service: | ||
|
||
``` | ||
$ docker-compose up -d --build greeter-functions | ||
``` | ||
|
||
This rebuilds the functions service image with the updated code, and restarts the service with the new image. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -0,0 +1,110 @@ | ||
################################################################################ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
################################################################################ | ||
version: "2.1" | ||
|
||
services: | ||
|
||
############################################################### | ||
# Functions service | ||
############################################################### | ||
|
||
greeter-functions: | ||
build: | ||
dockerfile: Dockerfile | ||
context: . | ||
expose: | ||
- "1108" | ||
|
||
############################################################### | ||
# StateFun runtime | ||
############################################################### | ||
|
||
statefun-manager: | ||
image: flink-statefun:2.3-SNAPSHOT | ||
expose: | ||
- "6123" | ||
ports: | ||
- "8081:8081" | ||
environment: | ||
ROLE: master | ||
MASTER_HOST: statefun-manager | ||
volumes: | ||
- ./module.yaml:/opt/statefun/modules/greeter/module.yaml | ||
|
||
statefun-worker: | ||
image: flink-statefun:2.3-SNAPSHOT | ||
expose: | ||
- "6121" | ||
- "6122" | ||
depends_on: | ||
- statefun-manager | ||
- kafka | ||
- greeter-functions | ||
links: | ||
- "statefun-manager:statefun-manager" | ||
- "kafka:kafka" | ||
- "greeter-functions:greeter-functions" | ||
environment: | ||
ROLE: worker | ||
MASTER_HOST: statefun-manager | ||
volumes: | ||
- ./module.yaml:/opt/statefun/modules/greeter/module.yaml | ||
|
||
############################################################### | ||
# Kafka for ingress and egress | ||
############################################################### | ||
|
||
zookeeper: | ||
image: confluentinc/cp-zookeeper:5.4.3 | ||
environment: | ||
ZOOKEEPER_CLIENT_PORT: "2181" | ||
ports: | ||
- "2181:2181" | ||
|
||
kafka: | ||
image: confluentinc/cp-kafka:5.4.3 | ||
ports: | ||
- "9092:9092" | ||
depends_on: | ||
- zookeeper | ||
links: | ||
- "zookeeper:zookeeper" | ||
environment: | ||
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 | ||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 | ||
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" | ||
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 | ||
|
||
############################################################### | ||
# Simple Kafka JSON producer to simulate ingress events | ||
############################################################### | ||
|
||
user-logins-producer: | ||
image: ververica/statefun-playground-producer:latest | ||
depends_on: | ||
- kafka | ||
- statefun-worker | ||
links: | ||
- "kafka:kafka" | ||
environment: | ||
APP_PATH: /mnt/user-logins.txt | ||
APP_KAFKA_HOST: kafka:9092 | ||
APP_KAFKA_TOPIC: user-logins | ||
APP_JSON_PATH: user_id | ||
volumes: | ||
- ./user-logins.txt:/mnt/user-logins.txt |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -0,0 +1,53 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one or more | ||
# contributor license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright ownership. | ||
# The ASF licenses this file to You under the Apache License, Version 2.0 | ||
# (the "License"); you may not use this file except in compliance with | ||
# the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
version: "3.0" | ||
|
||
module: | ||
meta: | ||
type: remote | ||
spec: | ||
endpoints: | ||
- endpoint: | ||
meta: | ||
kind: http | ||
spec: | ||
typename: | ||
namespace: greeter.fns | ||
urlPathTemplate: http://greeter-functions:1108/ | ||
ingresses: | ||
- ingress: | ||
meta: | ||
type: statefun.kafka.io/routable-protobuf-ingress | ||
id: greeter.io/user-logins | ||
spec: | ||
address: kafka:9092 | ||
consumerGroupId: greeter | ||
startupPosition: | ||
type: earliest | ||
topics: | ||
- topic: user-logins | ||
typeUrl: greeter.types/org.apache.flink.statefun.playground.java.greeter.types.UserLogin | ||
targets: | ||
- greeter.fns/user | ||
egresses: | ||
- egress: | ||
meta: | ||
type: statefun.kafka.io/generic-egress | ||
id: greeter.io/user-greetings | ||
spec: | ||
address: kafka:9092 | ||
deliverySemantic: | ||
type: at-least-once |
Oops, something went wrong.