Added way for users to create Writables and specify the Java names to save in the SequenceFile, overriding the reflection-based determination. #4

Merged
merged 1 commit into from Jul 3, 2012

Conversation

Projects
None yet
2 participants
Contributor

calmofthestorm commented Jul 2, 2012

hadoop.util.ReflectionUtils.hadoopClassName now checks for the existence of hadoop_module_name and hadoop_class_name class attributes before pulling the hadoop class name from the Python module and class name.

This is useful because it lets you subclass AbstractValueWritable and implement other Hadoop (such as BytesWritable in my case) writables in your code, rather than needing to modify the python-hadoop code. This also provides an easy way to implement arbitrary Writables that are not part of Hadoop, provided an equivalent Java class is used with Hadoop jobs.

Added way for users to create Writables and specify the Java names to…
… save in the SequenceFile, overriding the reflection-based determination.

hadoop.util.ReflectionUtils.hadoopClassName now checks for the existence of hadoop_module_name and hadoop_class_name class attributes before pulling the hadoop class name from the Python module and class name.

This is useful because it lets you subclass AbstractValueWritable and implement other Hadoop (such as BytesWritable in my case) writables in your code, rather than needing to modify the python-hadoop code. This also provides an easy way to implement arbitrary Writables that are not part of Hadoop, provided an equivalent Java class is used with Hadoop jobs.

matteobertozzi added a commit that referenced this pull request Jul 3, 2012

Merge pull request #4 from calmofthestorm/master
Added way for users to create Writables and specify the Java names to save in the SequenceFile, overriding the reflection-based determination.

@matteobertozzi matteobertozzi merged commit 6dad178 into matteobertozzi:master Jul 3, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment