Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Infinispan Hadoop Build Status

Integrations with Apache Hadoop and related frameworks.

Compatibility

Version Infinispan Hadoop Java
0.1 8.0.x 2.x 8
0.2 8.2.x 2.x 8
0.3 9.4.x 2.x 3.x 8
0.4 9.4.x 2.x 3.x 8

InfinispanInputFormat and InfinispanOutputFormat

Implementation of Hadoop InputFormat and OutputFormat that allows reading and writing data to Infinispan Server with best data locality. Partitions are generated based on segment ownership and allows processing of data in a cache using multiple splits in parallel.

Maven Coordinates

 <dependency>  
    <groupId>org.infinispan.hadoop</groupId>  
    <artifactId>infinispan-hadoop-core</artifactId>  
    <version>0.4</version>
 </dependency>  

Sample usage with Hadoop YARN mapreduce application:

import org.infinispan.hadoop.*;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.mapreduce.Job;

Configuration configuration = new Configuration();
String hosts = "172.17.0.2:11222;172.17.0.3:11222";

// Configures input/output caches
configuration.set(InfinispanConfiguration.INPUT_REMOTE_CACHE_SERVER_LIST, hosts);
configuration.set(InfinispanConfiguration.OUTPUT_REMOTE_CACHE_SERVER_LIST, hosts);

configuration.set(InfinispanConfiguration.INPUT_REMOTE_CACHE_NAME, "map-reduce-in");
configuration.set(InfinispanConfiguration.OUTPUT_REMOTE_CACHE_NAME, "map-reduce-out");

Job job = Job.getInstance(configuration, "Infinispan job");

// Map and Reduce implementation
job.setMapperClass(MapClass.class);
job.setReducerClass(ReduceClass.class);

job.setInputFormatClass(InfinispanInputFormat.class);
job.setOutputFormatClass(InfinispanOutputFormat.class);

Supported Configurations:

Name Description Default
hadoop.ispn.input.filter.factory The name of the filter factory deployed on the server to pre-filter data before reading null (no filtering)
hadoop.ispn.input.cache.name The name of cache where data will be read from "default"
hadoop.ispn.input.read.batch Batch size when reading from the cache 5000
hadoop.ispn.output.write.batch Batch size when writing to the cache 500
hadoop.ispn.input.remote.cache.servers List of servers of the input cache, in the format host1:port1;host2:port2 localhost:11222
hadoop.ispn.output.cache.name The name of cache where job results will be written to "default"
hadoop.ispn.output.remote.cache.servers List of servers of the output cache, in the format host1:port1;host2:port2
hadoop.ispn.input.converter Class name with an implementation of org.infinispan.hadoop.KeyValueConverter, applied after reading from the cache null (no converting)
hadoop.ispn.output.converter Class name with an implementation of org.infinispan.hadoop.KeyValueConverter, applied before writing null (no converting)

Demos

Refer to https://github.com/infinispan/infinispan-hadoop/tree/master/samples/

Releasing

The $MAVEN_HOME/conf/settings.xml must contain credentials for the release repository. Add the following section in <servers>:

<server>
   <id>jboss-snapshots-repository</id>
   <username>RELEASE_USER</username>
   <password>RELEASE_PASS</password>
</server>
<server>
   <id>jboss-releases-repository</id>
   <username>RELEASE_USER</username>
   <password>RELEASE_PASS</password>
</server>

To release:

mvn release:prepare release:perform -B

About

Infinispan Hadoop integration

Resources

License

Packages

No packages published

Languages

You can’t perform that action at this time.