No description, website, or topics provided.
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Godeps
api
build Merge tag 'v0.9.2' of https://github.com/GoogleCloudPlatform/kubernetes Jan 31, 2015
cluster Merge tag 'v0.9.2' of https://github.com/GoogleCloudPlatform/kubernetes Jan 31, 2015
cmd Implement volumes as plugins. Jan 20, 2015
contrib Put a version tag on kubernetes/serve_hostname Jan 21, 2015
docs Add a kubectl stop command Jan 23, 2015
examples Merge tag 'v0.9.2' of https://github.com/GoogleCloudPlatform/kubernetes Jan 31, 2015
hack Merge tag 'v0.9.2' of https://github.com/GoogleCloudPlatform/kubernetes Jan 31, 2015
hooks Fix errant error message from boilerplate check Jan 19, 2015
pkg upgrading to a newer version of gohadoop Feb 21, 2015
plugin Merge tag 'v0.9.2' of https://github.com/GoogleCloudPlatform/kubernetes Jan 31, 2015
test
third_party Swagger UI: Updating swagger-ui.js to list the resources and operations Jan 13, 2015
www Moving swagger-ui files from www/swagger-ui to third-party/swagger-ui Jan 6, 2015
.gitignore Clean up how client is passed to Kubelet in preparation for reading pods Jan 7, 2015
.travis.yml
CHANGELOG.md Update CHANGELOG.md Dec 23, 2014
CONTRIB.md Move CONTRIB{,UTING}.md so GitHub shows it Jul 28, 2014
CONTRIBUTING.md
DESIGN.md Removing TOC links to no-longer-extant sections Oct 29, 2014
LICENSE First commit Jun 6, 2014
MAINTAINERS.md ubuntu getting started guide Dec 15, 2014
Makefile
README.md adding a note about ansible to README.md Jan 24, 2015
README.original.md Merge tag 'v0.9.2' of https://github.com/GoogleCloudPlatform/kubernetes Jan 31, 2015
Vagrantfile Merge tag 'v0.9.2' of https://github.com/GoogleCloudPlatform/kubernetes Jan 31, 2015
clean_cluster.sh adding a couple of helper scripts Oct 15, 2014
logo.pdf change logo blue slightly Aug 15, 2014
logo.png
logo.svg
logo_usage_guidelines.md Add logo usage guidelines doc. Aug 26, 2014
provision_guestbook_app.sh

README.md

Kubernetes-YARN

A version of Kubernetes using Apache Hadoop YARN as the scheduler. Integrating Kubernetes with YARN lets users run Docker containers packaged as pods (using Kubernetes) and YARN applications (using YARN), while ensuring common resource management across these (PaaS and data) workloads.

Kubernetes-YARN is currently in the protoype/alpha phase

This integration is under development. Please expect bugs and significant changes as we work towards making things more stable and adding additional features.

Getting started

Dev Environment

Kubernetes and Kubernetes-YARN are written in Go. Currently, vagrant and ansible based setup mechanims are supported. The instructions below are for creating a vagrant based cluster. For ansible instructions, see here.

Please ensure you have boot2docker, Go (at least 1.3), Vagrant (at least 1.6), VirtualBox (at least 4.3.x) and git installed. Run boot2docker to bring up a VM with a running docker daemon (this is used for building release binaries for Kubernetes).

$ $(boot2docker shellinit) #sets up docker env vars
$ echo $GOPATH
/home/user/goproj
$ mkdir -p $GOPATH/src/github.com/hortonworks/
$ cd $GOPATH/src/github.com/hortonworks/
$ git clone git@github.com:hortonworks/kubernetes-yarn.git
$ cd kubernetes-yarn
$ build/release.sh #builds kubernetes release binaries 
$ hack/build-go.sh #builds kubernetes client binaries
$ cluster/kube-up.sh #brings up kubernetes cluster

Following these steps will bring up a multi-VM cluster (1 master and 3 minions, by default) running Kubernetes and YARN. Please note that, depending on your local hardware and available bandwidth, bringing the cluster up could take a while to complete.

YARN Dashboard

By default, the kubernetes master is assigned the IP 10.245.1.2. The YARN resource manager runs on the name host. Once the vagrant cluster is running, the YARN dashboard accessible at http://10.245.1.2:8088/

HDFS Dashboard

The HDFS dashboard is accessible at http://10.245.1.2:50070/

Interacting with the Kubernetes-YARN cluster

Creating pods/running Docker containters

For instructions on creating pods, running containers and other interactions with the cluster, please see Kubernetes' vagrant instructions here

Running a test map-reduce job

In order to run a test map-reduce job, log into the cluster (ensure that you are in the kubernetes-yarn directory) and run the included test script.

$ vagrant ssh master
[vagrant@kubernetes-master ~]$ cd hadoop/install/hadoop-2.6.0-SNAPSHOT/
[vagrant@kubernetes-master hadoop-2.6.0-SNAPSHOT]$ sudo su #you need to be root to run this test script
[root@kubernetes-master hadoop-2.6.0-SNAPSHOT]# ./test-pi-yarn.sh