Hadoop Operator

Overview

Hadoop is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, providing a highly-available service on top of a cluster of computers, each of which may be prone to failures

The hadoop Operator manages hadoop clusters deployed to Kubernetes, and provides a cloud-native way to submit Hadoop Yarn jobs.

Install the Operator

Cert Manager

hadoop operator relies on cert-manager, you must have cert-manager installed and configured in your Kubernetes cluster. cert-manager is a Kubernetes add-on that automates the management of certificates and is a requirement for this Operator to function correctly.

Installing cert-manager:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.3/cert-manager.yaml

📖 Read more about installing cert-manager using kubectl apply and static manifests.

Master Branch

kubectl apply -k github.com/chriskery/hadoop-operator/manifests/default

Quick Start

Please refer to the quick-start.md for more information.

Features

• Create and destroy a hadoop-cluster
• Hdfs DataNode HPA
• Yarn NodeManager HPA
• Submit Hadoop Job

Contributing and Community

We thrive to build a welcoming and open community for anyone who wants to use the operator or contribute to it.