HDFS client for Python
Python Shell
Switch branches/tags
Nothing to show
Clone or download
Travis Crawford
Latest commit 0ad9421 Feb 2, 2011


Python HDFS Client

Use the Hadoop distributed filesystem directly from Python! Implemented as a file-like object, working with HDFS files feels similar to how you'd expect.

Quick Start

from hdfs.hfile import Hfile

hfile = Hfile(hostname, port, path, mode='w')

hfile = Hfile(hostname, port, path)
data = hfile.read()

See example.py for more help getting started.

Java Classpath

As libhdfs uses JNI, we must set the classpath environment variable for python-hdfs to function properly. The exact classpath will vary between installations, but basically you need the Hadoop conf directory and all Hadoop jar's. For example:

export CLASSPATH=${HADOOP_CONF}:$(find ${HADOOP_HOME} -name *.jar | tr '\n' ':')

After exporting the classpath simply start your process using python-hdfs as usual.

Development Status

The library is under active development and should not be used in production at this time. The ctypes layer wrapping the native client works correctly, as do critical methods like read and write. Areas under development include implementing more Hfilesystem methods, and general polish.


Contributions are extremely welcome, as are suggestions about how to structure, package, and test the library. Feel free to send me an email or pull request and I'll get back ASAP.