GitHub - rajeshbalamohan/hadoop-aws: Hadoop aws with fixes for S3 access

rajeshbalamohan / hadoop-aws Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

Hadoop aws with fixes for S3 access - Reduces the number of connections

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
src/main/java/org/apache/hadoop/fs/s3r		src/main/java/org/apache/hadoop/fs/s3r
README.txt		README.txt
hadoop-aws-s3r-2.7.1.jar		hadoop-aws-s3r-2.7.1.jar
pom.xml		pom.xml

Repository files navigation

Build:
=====
1. "mvn clean package".  This should create a hadoop-aws-s3r jar file in target folder.

Running:
========
1. Place target/hadoop-aws-sr3r*.jar in the classpath
2. Start cli as 'hive --hiveconf fs.s3a.impl="org.apache.hadoop.fs.s3r.S3RFileSystem" '
3. In hive cli, "ADD JAR file:///PATH_TO_YOUR_JAR/hadoop-aws-s3r-2.7.1.jar;"
    e.g "ADD JAR file:///home/rajesh/hadoop-aws-s3r-2.7.1.jar;" (in case the jar in home folder).
4. Run your queries as normal.

OOTB:
===
1. Attached here the pre-built hadoop-aws-s3r-2.7.1.jar (in case of checking quickly without having to recompile)