Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Hadoop Interactive Shell
Java Groovy
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
src/main/java
README.md
build.gradle

README.md

HadooSh

Ever wanted to grep a file on HDFS and store the results locally? Now you can, with one simple command in HadooSh:

cbeavers > cd books/
books > cat Ulysses-part-00000 | grep Mulligan >l localOut.txt

HadooSh is an interactive shell for HDFS built on top of JLine to offer tab completion of commands and paths under Hadoop. It supports piping to local system commands, and both local and remote output. There are many bugs to be found, so please play nicely with it.

Currently supported operations:

  • cd [dir]
  • pwd
  • head [numLines]
  • all FsShell commands (hadoop fs), such as
    • cat [files]
    • mv <src> <dst>
    • rm [files]
  • avrocat (prints the first ten records from avro file)
  • local (to execute one of the above commands on the local FS)
  • support for piping to local commands
  • use ">" to run command output to HDFS filesystem
  • use ">l" to run command output to local filesystem
  • all JobClient commands (hadoop job) are accessible to by typing "job" after the prompt first
  • runjar <localjar> ... (hadoop jar)
  • tlog (-job <jobid> |-dir <jobOutputDir>) [-taskpattern taskglob] [-hostpattern hostglob]

     # Show all logs for a job:
     tlog -job job_201306131712_0004
    
     # Show all mapper logs for a teragen job:
     tlog -dir /user/gera/tgen -taskpattern *_m_*
    
     # grep logs for job tasks run on certain nodes
     tlog -job job_201306131712_0004 -hostpattern *.rack.company.com | grep needle

Planned future actions:

  • should probably put some limits on file sizes

Known bugs:

  • piped commands that contain quotes such as
cat file | cut -d" " -f 1

Build HadooSh:

git clone https://github.com/gerashegalov/hadoosh.git
cd hadoosh
mvn package

This builds HadooSh against Apache dependencies

You can also use build HadooSh against MapR artifacts:

mvn package -Pmapr

Executing HadooSh:

To use HadooSh, just copy the included jar to your Hadoop cluster's gateway, make sure you've kinit'd if necessary, and run the following:

wget https://github.com/cbeav/hadoosh/raw/master/HadooSh.jar
hadoop jar HadooSh.jar HadooSh

To run HadooSh with Maven:

# symlink hadoop conf
ln -s ${HADOOP_HOME}/conf
mvn clean compile exec:exec -Dexec.executable=java -Dexec.args="-classpath %classpath HadooSh"

Enjoy.

Authors: Chris Beavers, Paul Hobbs, and Gera Shegalov

Something went wrong with that request. Please try again.