Skip to content

zwzhang0107/yarn-copilot

 
 

Repository files navigation

Koordinator YARN Copilot

Koordinator

License GitHub release CI Go Report Card codecov PRs Welcome Slack

English | 简体中文

Introduction

Koordinator has supported hybrid orchestration workloads on Kubernetes, so that batch jobs can use the requested but unused resource as koord-batch priority and BE QoS class to improve the cluster utilization. However, there are still lots of applications running beyond K8s such as Apache Haddop YARN. As a resource management platform in BigData ecosystem, YARN has supported numbers of computing engines including MapReduce, Spark, Flink, Presto, etc.

In order to extend the co-location scenario of Koordinator, now the community has provided Hadoop YARN extended suits Koordinator YARN Copilot in BigData ecosystem, supporting running Hadoop YARN jobs by koord-batch resources with other K8s pods. The Koordinator YARN Copilot has following characteristics:

  • Open-Source native: implement against open-sourced version of Hadoop YARN; so there is no hack inside YARN modules.
  • Unifed resource priority and QoS strategy: the suits aims to the koord-batch priority of Koordinator, and also managed by QoS strategies of koordlet.
  • Resource sharing on node level: node resources of koord-batch priority can be requested by tasks of YARN or Batch pods both.
  • Adaptive for multiple environments: the suits can be run under any environment, including public cloud or IDC.

Quick Start

You can view the full documentation from the Koordinator website.

  • Install or upgrade Koordinator with the latest version.
  • Referring to best practices, there will be detailed instructions for running Hadoop YARN jobs with Koordinator batch resources in K8s.

Code of conduct

The Koordinator community is guided by our Code of Conduct, which we encourage everybody to read before participating.

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Contributing

You are warmly welcome to hack on Koordinator. We have prepared a detailed guide CONTRIBUTING.md.

Community

The koordinator-sh/community repository hosts all information about the community, membership and how to become them, developing inspection, who to contact about what, etc.

We encourage all contributors to become members. We aim to grow an active, healthy community of contributors, reviewers, and code owners. Learn more about requirements and responsibilities of membership in the community membership page.

Active communication channels:

Dingtalk QRCode

License

Koordinator is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

About

An extended suite for running Hadoop YARN jobs in K8s with Koordinator.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 89.8%
  • Makefile 4.4%
  • Shell 4.3%
  • Dockerfile 1.5%