Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Added way for users to create Writables and specify the Java names to save in the SequenceFile, overriding the reflection-based determination. #4

Merged
merged 1 commit into from

2 participants

@calmofthestorm

hadoop.util.ReflectionUtils.hadoopClassName now checks for the existence of hadoop_module_name and hadoop_class_name class attributes before pulling the hadoop class name from the Python module and class name.

This is useful because it lets you subclass AbstractValueWritable and implement other Hadoop (such as BytesWritable in my case) writables in your code, rather than needing to modify the python-hadoop code. This also provides an easy way to implement arbitrary Writables that are not part of Hadoop, provided an equivalent Java class is used with Hadoop jobs.

@calmofthestorm calmofthestorm Added way for users to create Writables and specify the Java names to…
… save in the SequenceFile, overriding the reflection-based determination.

hadoop.util.ReflectionUtils.hadoopClassName now checks for the existence of hadoop_module_name and hadoop_class_name class attributes before pulling the hadoop class name from the Python module and class name.

This is useful because it lets you subclass AbstractValueWritable and implement other Hadoop (such as BytesWritable in my case) writables in your code, rather than needing to modify the python-hadoop code. This also provides an easy way to implement arbitrary Writables that are not part of Hadoop, provided an equivalent Java class is used with Hadoop jobs.
73ca1d5
@matteobertozzi matteobertozzi merged commit 6dad178 into matteobertozzi:master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jul 2, 2012
  1. @calmofthestorm

    Added way for users to create Writables and specify the Java names to…

    calmofthestorm authored
    … save in the SequenceFile, overriding the reflection-based determination.
    
    hadoop.util.ReflectionUtils.hadoopClassName now checks for the existence of hadoop_module_name and hadoop_class_name class attributes before pulling the hadoop class name from the Python module and class name.
    
    This is useful because it lets you subclass AbstractValueWritable and implement other Hadoop (such as BytesWritable in my case) writables in your code, rather than needing to modify the python-hadoop code. This also provides an easy way to implement arbitrary Writables that are not part of Hadoop, provided an equivalent Java class is used with Hadoop jobs.
This page is out of date. Refresh to see the latest.
Showing with 7 additions and 2 deletions.
  1. +7 −2 python-hadoop/hadoop/util/ReflectionUtils.py
View
9 python-hadoop/hadoop/util/ReflectionUtils.py
@@ -24,8 +24,13 @@ def hadoopClassFromName(class_path):
return classFromName(class_path)
def hadoopClassName(class_type):
- module_name = class_type.__module__
- class_name = class_type.__name__
+ if hasattr(class_type, "hadoop_module_name") and \
+ hasattr(class_type, "hadoop_class_name"):
+ module_name = class_type.hadoop_module_name
+ class_name = class_type.hadoop_class_name
+ else:
+ module_name = class_type.__module__
+ class_name = class_type.__name__
if module_name.startswith('hadoop.io.'):
module_name, _, file_name = module_name.rpartition('.')
return 'org.apache.%s.%s' % (module_name, class_name)
Something went wrong with that request. Please try again.