Skip to content

Commit

Permalink
EmrEtlRunner: added hadoop_elasticsearch to config.yml.sample (closes #…
Browse files Browse the repository at this point in the history
  • Loading branch information
fblundun authored and alexanderdean committed Jan 14, 2016
1 parent d28db34 commit 8258493
Showing 1 changed file with 16 additions and 2 deletions.
18 changes: 16 additions & 2 deletions 3-enrich/emr-etl-runner/config/config.yml.sample
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,9 @@ collectors:
enrich:
job_name: Snowplow ETL # Give your job a name
versions:
hadoop_enrich: 1.2.0 # Version of the Hadoop Enrichment process
hadoop_shred: 0.5.0 # Version of the Hadoop Shredding process
hadoop_enrich: 1.3.0 # Version of the Hadoop Enrichment process
hadoop_shred: 0.6.0 # Version of the Hadoop Shredding process
hadoop_elasticsearch: 0.1.0 # Version of the Hadoop to Elasticsearch copying process
continue_on_unexpected_error: false # Set to 'true' (and set :out_errors: above) if you don't want any exceptions thrown from ETL
output_compression: NONE # Compression only supported with Redshift, set to NONE if you have Postgres targets. Allowed formats: NONE, GZIP
storage:
Expand Down Expand Up @@ -80,6 +81,19 @@ storage:
password: ADD HERE
maxerror: # Not required for Postgres
comprows: # Not required for Postgres
- name: "My Elasticsearch database"
type: elasticsearch
host: ADD HERE # The Elasticsearch endpoint
database: ADD HERE # Name of index
port: 9200 # Default Elasticsearch port - change to 80 if using Amazon Elasticsearch Service
sources: # Leave blank to write the bad rows created in this run to Elasticsearch, or explicitly provide an array of bad row buckets like ["s3://my-enriched-bucket/bad/run=2015-10-06-15-25-53"]
ssl_mode: # Not required for Elasticsearch
table: ADD HERE # Name of type
username: # Not required for Elasticsearch
password: # Not required for Elasticsearch
es_nodes_wan_only: false # Set to true if using Amazon Elasticsearch Service
maxerror: # Not required for Elasticsearch
comprows: # Not required for Elasticsearch
monitoring:
tags: {} # Name-value pairs describing this job
logging:
Expand Down

0 comments on commit 8258493

Please sign in to comment.