Skip to content

myHadoop is a simple system for end-users to provision Hadoop instances on traditional supercomputing resources, without requiring any root privileges. Users may use myHadoop to configure and instantiate Hadoop on the fly via regular batch scripts.

License

OpenTopography/myHadoop

main
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
bin
 
 
 
 
etc
 
 
 
 
 
 
 
 

NSF-0844530

UPDATE myHadoop is no longer being updated here. Originally written by Sriram Krishnan at SDSC, it is currently being maintained at https://github.com/glennklockwood/myhadoop/

myHadoop

myHadoop enables the use of Hadoop in a non-dedicated cluster environment, being administered by typical batch scheduler. We are currently supporting the PBS (Moab) and the Sun Grid Engine (SGE) schedulers, although a port to a scheduler such as Condor would be very trivial.

The pbs-example.sh and sge-example.sh provide an example of how to use myHadoop with PBS and SGE respectively. For more details, please read the documentation in the "docs" directory.

Pre-requisites

myHadoop needs Apache Hadoop 0.20.2 and a batch scheduler such as PBS.

Other

Work was funded by a grant from the NSF Cluster Exploratory (CluE) program (Award# IIS-0844530, PI: Baru, Co-PI Krishnan) More info: http://nsf.gov/awardsearch/showAward?AWD_ID=0844530

About

myHadoop is a simple system for end-users to provision Hadoop instances on traditional supercomputing resources, without requiring any root privileges. Users may use myHadoop to configure and instantiate Hadoop on the fly via regular batch scripts.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages