Export Redis data to HDFS
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
redis2hdfs
tests
.gitignore
.travis.yml
CHANGES.rst
MANIFEST.in
README.rst
setup.py
tests-req.txt

README.rst

redis2hdfs

Build Status Latest Version Supported Python versions Development Status License

redis2hdfs is a command line tool to help you export Redis data to HDFS. It use SCAN related commands to iterate elements (no KEYS), so you can use in production safely.

NOTE: ensure WebHDFS is enabled.

Installation

$ pip install redis2hdfs

Usage

$ redis2hdfs --redis-key myzset --namenode-host namenode.example.com --hdfs-username hdfs --hdfs-path /tmp/myzset.lzo --compress-format lzo

redis2hdfs could compress file before copy to HDFS, through --compress-format option. Currently supported compress formats are: LZO.

If you want to use LZO format, you need install lzop first.

redis2hdfs supports Redis global-style key pattern, so you can specify --redis-key like *abc*. redis2hdfs will copy all matched keys' data to HDFS, at this time the --hdfs-path option will be the parent directory to store data.

For more information, just run redis2hdfs --help.

Development

$ mkvirtualenv redis2hdfs
$ python setup.py develop
$ pip install -r tests-req.txt
$ nosetests -v