Browse files

no more need for wu-hadoop-es

  • Loading branch information...
Dhruv Bansal
Dhruv Bansal committed Nov 29, 2012
1 parent c102e8c commit 21a7636ce11ca54330242334f729eb8cba8e91bb
Showing with 0 additions and 61 deletions.
  1. +0 −61 bin/wu-hadoop-es
@@ -1,61 +0,0 @@
-#!/usr/bin/env ruby
-require 'wonderdog'
-settings = Wukong::Elasticsearch::Configuration
-def settings.usage()
- "usage: #{File.basename($0)} [ --param=value | -p value | --param | -p] PROCESSOR|FLOW [PROCESSOR|FLOW]"
-settings.description = <<EOF
-wu-hadoop-es is a tool that works in combination with wu-hadoop to run
-Hadoop jobs powered by Wukong using data read from or written to
-The details of how wu-hadoop-es interacts with Hadoop are the same as
-for wu-hadoop, so see that tool's reference for more details.
-wu-hadoop-es is just a layer over wu-hadoop that makes it easier to
-connect to ElasticSearch for reading or writing data. It does this by
-providing a simple way to specify ElasticSearch as an output in
- $ wu-hadoop-es mapper.rb reducer.rb --input=/hdfs/input/path --output=es://my_index/my_type
-with the es:// scheme-prefix implying that the --output path should be
-interpreted as an ElasticSearch index ("my_index") and type
-("my_type"). You can have more fine-grained control over how data is
-written, see the various "field" options.
-You can also easily read data:
- $ wu-hadoop-es mapper.rb reducer.rb --input=es://my_index/my_type --output=/hdfs/output/path
-and even specify a query
- $ wu-hadoop-es mapper.rb reducer.rb --input=es://my_index/my_type --output=/hdfs/output/path --query='{"query_string": {"query": "hi +there"}}'
-There are several tuning options available. ElasticSearch
-configuration (and how to specify which cluster to find and join) are
-handled using ElasticSearch's usual configuration file based approach,
-specified by the --es_config option.
-# These options we want to pass on to wu-hadoop but sometimes we want
-# to mess with one first so we'll have to reconstruct them later
-# ourselves.
-settings.define(:input, :description => "Comma-separated list of input paths: local, HDFS, or ElasticSearch", :wonderdog => true)
-settings.define(:output, :description => "Output path: local, HDFS, or ElasticSearch", :wonderdog => true)
-require 'wukong/boot' ; Wukong.boot!(settings)
- settings.dump_help
- exit(1)

0 comments on commit 21a7636

Please sign in to comment.