StreamSets

StreamSets (STS)

Setting up on Cloudera Hadoop

Cloudera Manager -> HDFS -> Configuration
Than in the "Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml"

add:

<property>   
<name>hadoop.proxyuser.sdc.hosts</name>   
<value>*</value>   
</property>    
<property>   
<name>hadoop.proxyuser.sdc.groups</name>   
<value>*</value>   
</property> 

Then restart the cluster

Connecting to Kafka

The Hadoop need to have the Kafka IPs info in /etc/hosts

Add to /etc/hosts

x.x.x.x     <host1>.domain_name.com     <host1>
y.y.y.y     <host2> <host2>.domain_name.com
z.z.z.z     <host3> <host3>.domain_name.com

Data Collector Environment Configuration

In Cloudera CDH type deployment :
$SDC_DIST = /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR/

Clearing pipeline Cache

In some situations SDC pipelines might not function well due to cache issues related to Kafka.
To clear the cache follow the steps:

• Stop the pipeline
• Delete StreamSets spark cache dir for specific consumer group of specific Kafka topic.
  This directory is located in HDFS:
  /user/sdc/.streamsets-spark-streaming/<StreamSets internal id>/<Kafka topic>/<consumer group>/<pipeline name>
  where <StreamSets internal id> is the same for all pipelines.

Installing mysql driver in docker SDC

There is a security issue that needs to be changed in /etc/sdc/sdc-security.policy
After getting the SDC and running it
Need to change:

// Groovy will parse files in a different context, so we need to grant it additional privileges
grant codebase "file:/groovy/script" {
  permission java.lang.RuntimePermission "getClassLoader";
};

to

// Groovy will parse files in a different context, so we need to grant it additional privileges
grant codebase "file:/groovy/script" {
  permission java.security.AllPermission;
};

• Use docker cp to copy the file into your local machine file system
• Edit the file with your favorite text editor
• Copy back into docker using docker cp
  Next need to set proper permission on the file
• Log as root $ docker exec -it --user=root docker_id_first_few_digits
• Change to /etc/sdc/
• run chown sdc:sdc sdc-security.policy
  Next
• run the SDC pipeline
• install the the groovy processor (put in in the pipeline and follow instruction if need any installation)
• use the package manager to add external package of the jar file, such as mysql-connector-java-5.1.45-bin.jar
  to the groovy lib

You might need to restart the pipeline several times
Note the Docker use Alpine linux, so if you want to install python or other things
you might need to use apk --add-no cashe add ... when logged as root