Skip to content
Kubernetes operator that provides control plane for managing Apache Flink applications
Go Shell Java Other
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
artifacts initial commit May 30, 2019
boilerplate/lyft initial commit May 30, 2019
cmd/flinkk8soperator Initialize Klog explicitly using InitFlags (#40) Jul 8, 2019
config [STRMCMP-590] Update CRD version to v1beta1 (#67) Aug 6, 2019
deploy Bumping to version 0.3.0 (#120) Oct 15, 2019
docs [#97] [STRMCMP-648] [BREAKING] Fix rendering of flink memory configs (#… Sep 27, 2019
examples Fix example Dockerfile (#116) Oct 9, 2019
integ [#97] [STRMCMP-648] [BREAKING] Fix rendering of flink memory configs (#… Sep 27, 2019
pkg [STRMCMP-639] Trigger immediate rollback when job submission fails (#117 Oct 9, 2019
script initial commit May 30, 2019
tmp [STRMCMP-590] Update CRD version to v1beta1 (#67) Aug 6, 2019
version Update version to 0.3 (#118) Oct 10, 2019
.gitignore Adding and fixing files which got missed out May 30, 2019
.golangci.yml Adding and fixing files which got missed out May 30, 2019
.travis.yml [STRMCMP-616] Improvements to integration test stability (#110) Sep 27, 2019
CODEOWNERS Add Lakshmi as codeowner (#14) Jun 3, 2019
CODE_OF_CONDUCT.md initial commit May 30, 2019
Dockerfile Updated Go to 1.13, controller-runtime to 0.2.2, added Lightbend to w… Sep 20, 2019
Gopkg.lock Updated Go to 1.13, controller-runtime to 0.2.2, added Lightbend to w… Sep 20, 2019
Gopkg.toml Updated Go to 1.13, controller-runtime to 0.2.2, added Lightbend to w… Sep 20, 2019
LICENSE initial commit May 30, 2019
Makefile initial commit May 30, 2019
NOTICE initial commit May 30, 2019
README.md Update slack link (#114) Oct 4, 2019
local_config.yaml Improve staleness logic by improving error handling and retrying a su… Jul 10, 2019

README.md

Current Release Build Status GoDoc License CodeCoverage Go Report Card Commit activity Commit since last release Slack

Flinkk8soperator

FlinkK8sOperator is a Kubernetes operator that manages Flink applications on Kubernetes. The operator acts as control plane to manage the complete deployment lifecycle of the application.

Project Status

Beta

The operator is in use for some less-crtical jobs at Lyft. At this point the focus is on testing and stability While in Beta, we will attempt to limit the number of backwards-incompatible changes, but they may still occur as necessary.

Prerequisites

  • Version >= 1.9 of Kubernetes.
  • Version >= 1.7 of Apache Flink.

Overview

Flink operator overview

The goal of running Flink on Kubernetes is to enable more flexible, lighter-weight deployment of streaming applications, without needing to manage infrastructure. The Flink operator aims to abstract out the complexity of hosting, configuring, managing and operating Flink clusters from application developers. It achieves this by extending any kubernetes cluster using a custom resources.

The Operator creates flink clusters dynamically using the specified custom resource. Flink clusters in kubernetes consist of the following:

Deploying and managing Flink applications in Kubernetes involves two steps:

  • Building Flink application packaged as a docker image: A docker image is built containing the application source code with the necessary dependencies built in. This is required to bootstrap the Jobmanager and Taskmanager pods. At Lyft we use Source-To-Image S2I as the image build tool that provides a common builder image with Apache Flink pre-installed. The docker image could be built using any pre-existing workflows at an organization.

  • Creating the Flink application custom resource: The custom resource for Flink application provides the spec for configuring and managing flink clusters in Kubernetes. The FlinkK8sOperator, deployed on Kubernetes, continuously monitors the resource and the corresponding flink cluster, and performs action based on the diff.

Documentation

Community

We welcome you to contribute and make the operator better! For questions, and changes please create an issue or submit a pull request.

You can’t perform that action at this time.