Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapreduce decompress s3 file #6

Open
tachu opened this issue Nov 26, 2015 · 0 comments
Open

Mapreduce decompress s3 file #6

tachu opened this issue Nov 26, 2015 · 0 comments

Comments

@tachu
Copy link

tachu commented Nov 26, 2015

Having trouble getting a file from s3 decompressed on the fly. here is the command we are trying

/usr/bin/hadoop jar /usr/lib/hadoop/hadoop-streaming.jar -libjars /home/hadoop/hadoop-xz-1.0.jar -Dmapred.output.compress=false -Dmapred.compress.map.out=false -Dmapred.input.compression.codec=io.sensesecure.hadoop.xz.XZCodec -Dmapred.map.tasks=1 -Dmapred.reduce.tasks=0 -input s3n://XXX/2013-05-02.txt.xz -output s3n://XXX/20130501_resultt -mapper /bin/cat -reducer /bin/cat

We basically are trying to decompress a file on the fly. this example would just write it back to s3 so far it just writes back the compressed data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant