[Merged by Bors] - New Druid landing page

[fhennig]

Description

I've made a diagram and added it to the overview page, this is just a draft, I made it in the break week.

It roughly belongs to this ticket: stackabletech/documentation#282

Netlify Preview 💻

Review Checklist

• Code contains useful comments
• CRD change approved (or not applicable)
• (Integration-)Test cases added (or not applicable)
• Documentation added (or not applicable)
• Changelog updated (or not applicable)
• Cargo.toml only contains references to git tags (not specific commits or branches)
• Helm chart can be installed and deployed operator works (or not applicable)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

[stackfab]

Title: Stackable Operator for Apache Druid: A Kubernetes Operator for Managing Druid Clusters

Meta Description: The Stackable Operator for Apache Druid is a Kubernetes operator that can manage Apache Druid clusters. Learn about its features, resources, dependencies, and demos, and see the list of supported Druid versions.

Keywords: Stackable Operator, Apache Druid, Kubernetes, operator, DevOps, engineer, CRD, StatefulSet, ConfigMap, Service, ZooKeeper, HDFS, S3, Kafka, Trino, OPA, demo, version.

Introduction: Stackable Operator for Apache Druid

The Stackable Operator for Apache Druid is a Kubernetes operator that can manage Apache Druid clusters. With this operator, DevOps engineers can easily deploy and manage Druid clusters on Kubernetes. This operator provides several resources and features to manage Druid clusters efficiently.

Getting Started with the Operator

To get started with the Stackable Operator for Apache Druid, follow the Getting Started guide. The Operator is installed along with the DruidCluster CustomResourceDefinition, which supports five roles: Router, Coordinator, Broker, MiddleManager, and Historical. These roles correspond to Druid processes.

Kubernetes Resources Created by the Operator

The Stackable Operator for Apache Druid watches DruidCluster objects and creates multiple Kubernetes resources for each DruidCluster based on its configuration. The operator creates a StatefulSet for every RoleGroup, where each StatefulSet can contain multiple replicas (Pods). In each Pod, there are two containers: the main Druid container and a preparation container (TODO what for exactly?).

The operator also creates a ConfigMap for each RoleGroup containing three files: a generated jvm.config and runtime.properties file based on (TODO), and a log4j2.properties file used for log aggregation. The operator creates a Service for every Role and RoleGroup. Finally, the operator creates a discovery ConfigMap for the whole DruidCluster that contains information on how to connect to the Druid cluster.

Dependencies

The Stackable Operator for Apache Druid has the following dependencies:

• The Stackable Commons Operator: provides common CRDs such as S3 resources CRDs.
• The Stackable Secret Operator: required for things like S3 access credentials or LDAP integration.
• Apache ZooKeeper via the Stackable Operator for Apache ZooKeeper: used by Druid for internal communication between processes.
• Storage backend: The operator requires a storage backend such as HDFS with the Stackable Operator for Apache HDFS or S3. (TODO: link to S3 deep storage config usage).
• SQL database: The operator requires an SQL database to store metadata.

Other Operators to connect to

This Operator can natively connect to these other Stackable Operators:

• Stackable Operator for Apache Kafka: This is used for ingesting data from Kafka.
• Stackable Operator for Trino: Trino can read from Druid with this operator.
• Stackable Operator for OPA (OpenPolicyAgent) for Authorization: This operator is used for authorization.

Demos for Stackable Operator for Apache Druid

stackablectl provides an easy way to install demos that showcase multiple components of the Stackable platform working together. Two demos that include Druid as part of the data pipeline are:

Waterlevel Demo: The nifi-kafka-druid-water-level-data demo uses data from PEGELONLINE to visualize water levels in rivers and coastal regions of Germany from historic and real-time data.

Earthquake Demo: The nifi-kafka-druid-earthquake-data demo ...

Supported versions

The Stackable Operator for Apache Druid currently supports the following versions of Druid:

0.23.0
24.0.0

[adwk67]

Nice job! LGTM

[fhennig]

bors merge

[bors]

Pull request successfully merged into main.

Build succeeded:

• All tests passed