[FLINK-16685] Add a k8s example for the Python SDK

[igalshilman]

This PR adds a Kubernetes example

This example demonstrates how to deploy a stateful function application
written in Python to Kubernetes.

Prerequisites

• Helm
• Kubernetes cluster
• Kafka
• StateFun distribution
• StateFun Python SDK

Overview

This examples create a stateful function application,
that consumes LoginEvents from a logins Kafka topic,
and produces seen count per user, into the seen Kafka topic.

The main example components contains:

• main.py - A StateFun python function that implements the main logic
• module.yaml - defines the ingress, egress and the remote function specification.
• resources - a Helm chart, templates to deploy StateFun cluster and the remote python worker to k8s.
• build-example.sh - Builds StateFun Docker images and k8s resources to deploy it.

Setup

Create Kafka Topics:

This example consumes LoginEvents from the logins topic, and produces SeenCount to
the seen topic

 ./kafka-topics.sh --create --topic logins --zookeeper <zookeeper address>:2181 --partitions 1 --replication-factor 1
 ./kafka-topics.sh --create --topic seen --zookeeper <zookeeper address>:2181 --partitions 1 --replication-factor 1

update module.yaml

Make sure that your module.yaml ingress/and egress sections point to your
Kafka cluster.

ingresses:
      - ingress:
            ...
          spec:
            address: kafka-service:9092
            ...
    egresses:
      - egress:
            ...
          spec:
            address: kafka-service:9092

Build the Docker images and the k8s resource yamls.

This examples creates two different Docker images, one for the Python remote
worker (k8s-demo-python-worker) and one for the statefun cluster (k8s-demo-statefun).

• If you have a remote docker registry (i.e. gcr.io/<project-name>) make sure
  to update resources/values.yaml relevant image: sections.

• Modify resources/values.yaml and set the value of checkpoint.directory
  to a filesystem / object store. For example

checkpoint:
  dir: gcs://my-project/my-bucket

Assuming the all prerequisites where completed run:

build-example.sh

This should create the Docker images and generate a k8s-demo.yaml file.

Deploy

kubectl create -f k8s-demo.yaml

Generate events

Run:

pip3 install kafka-python 
python3 event_generator.py --address <kafka address> --events 1000

This would generate 1,000 login events into the logins topic

[sjwiesman]

Can you document somewhere as a follow up to remove this as part of the release.

[igalshilman]

This will not be removed as part of the release since the examples would be built on master.
There is an idea of creating independent examples repo, that uses the latest.

[sjwiesman]

https://github.com/apache/flink-playgrounds/

We could use flink playgrounds

[igalshilman]

Sounds good.

[sjwiesman]

Maybe off topics but should we add k8s deployment docs?

[tzulitai]

Thanks for adding the example @igalshilman, LGTM.
Merging ...

[igalshilman]

@tzulitai Thanks for the review and merging.

@sjwiesman I will follow up with FLINK-16064, and we can base any documentation on that.