A number of bash helper functions to make life with Hadoop more pleasant.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
README
hadoop-funcs

README

hadoop-funcs

A number of bash helper functions to make life with Hadoop more pleasant.
It's grown organically so some aspects may need a bit of work.

Requires perl5.

File Listing ("ls")
===================

hls   - list only file names (supports -r for recursion)
hlsh  - list only file sizes (in human readable format) and file names
hlsm  - list only file sizes (in MB) and file names
        (supports -t for sort by modification time old->new)
hlsg  - list only file sizes (in GB) and file names
        (supports -t for sort by modification time old->new)
hlsr  - recursively list only filenames
hlsl  - full directory listing in human readable format
        (supports -r for recursion)
hlslr - recursive full directory listing in human readable format

File Listing Sorted by Modification Time
========================================

hlst  - hls output sorted by time old->new
hlsht - hlsh output sorted by time old->new
hlsmt - hlsm output sorted by time old->new
hlsgt - hlsg output sorted by time old->new
hlslt - hlsl output sorted by time old->new

Directory Sizing ("du")
=======================

hduhs - directory summary in human readable form (like du -hs)
hduks - directory summary in KB (like du -ks)
hdf   - filesystem size in human readable form

Convenience
===========

hcat   - hadoop fs -cat
htext  - hadoop fs -text
hcat   - hadoop fs -cat
hhead  - file head
htail  - file tail
hless  - file piped to less
hcp    - hadoop fs -cp
hmv    - hadoop fs -mv
hget   - hadoop fs -get
hput   - hadoop fs -put (supports -f for force ... use with caution)
hrm    - hadoop fs -rm (supports -r for recursion)
hrmr   - hadoop fs -rmr
hmkdir - hadoop fs -mkdir
hchown - hadoop fs -chown
hchgrp - hadoop fs -chgrp
htouch - hadoop fs -touchz
hmerge - hadoop fs -getmerge
hrep   - sudo -u hdfs hadoop dfsadmin -report | less
hjl    - hadoop job -list
hkill  - hadoop job -kill