Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.rst

redis2hdfs

Build Status Latest Version Supported Python versions Development Status License

redis2hdfs is a command line tool to help you export Redis data to HDFS. It use SCAN related commands to iterate elements (no KEYS), so you can use in production safely.

NOTE: ensure WebHDFS is enabled.

Installation

$ pip install redis2hdfs

Usage

$ redis2hdfs --redis-key myzset --namenode-host namenode.example.com --hdfs-username hdfs --hdfs-path /tmp/myzset.lzo --compress-format lzo

redis2hdfs could compress file before copy to HDFS, through --compress-format option. Currently supported compress formats are: LZO.

If you want to use LZO format, you need install lzop first.

redis2hdfs supports Redis global-style key pattern, so you can specify --redis-key like *abc*. redis2hdfs will copy all matched keys' data to HDFS, at this time the --hdfs-path option will be the parent directory to store data.

For more information, just run redis2hdfs --help.

Development

$ mkvirtualenv redis2hdfs
$ python setup.py develop
$ pip install -r tests-req.txt
$ nosetests -v

About

Export Redis data to HDFS

Resources

Packages

No packages published

Languages

You can’t perform that action at this time.