StreamSets

StreamSets (STS)

Setting up on Cloudera Hadoop

Cloudera Manager -> HDFS -> Configuration
Than in the "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml"

add:

<property>   
<name>hadoop.proxyuser.sdc.hosts</name>   
<value>*</value>   
</property>    
<property>   
<name>hadoop.proxyuser.sdc.groups</name>   
<value>*</value>   
</property> 

Then restart the cluster

Connecting to Kafka

The Hadoop need to have the Kafka IPs info in /etc/hosts

Add to /etc/hosts

x.x.x.x     <host1>.domain_name.com     <host1>
y.y.y.y     <host2> <host2>.domain_name.com
z.z.z.z     <host3> <host3>.domain_name.com

Data Collector Environment Configuration

In Cloudera CDH type deployment :
$SDC_DIST = /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR/