Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

no more need for wu-hadoop-es

  • Loading branch information...
commit 21a7636ce11ca54330242334f729eb8cba8e91bb 1 parent c102e8c
@dhruvbansal dhruvbansal authored
Showing with 0 additions and 61 deletions.
  1. +0 −61 bin/wu-hadoop-es
View
61 bin/wu-hadoop-es
@@ -1,61 +0,0 @@
-#!/usr/bin/env ruby
-
-require 'wonderdog'
-
-settings = Wukong::Elasticsearch::Configuration
-
-settings.use(:commandline)
-
-def settings.usage()
- "usage: #{File.basename($0)} [ --param=value | -p value | --param | -p] PROCESSOR|FLOW [PROCESSOR|FLOW]"
-end
-
-settings.description = <<EOF
-wu-hadoop-es is a tool that works in combination with wu-hadoop to run
-Hadoop jobs powered by Wukong using data read from or written to
-ElasticSearch.
-
-The details of how wu-hadoop-es interacts with Hadoop are the same as
-for wu-hadoop, so see that tool's reference for more details.
-
-wu-hadoop-es is just a layer over wu-hadoop that makes it easier to
-connect to ElasticSearch for reading or writing data. It does this by
-providing a simple way to specify ElasticSearch as an output in
-Hadoop:
-
- $ wu-hadoop-es mapper.rb reducer.rb --input=/hdfs/input/path --output=es://my_index/my_type
-
-with the es:// scheme-prefix implying that the --output path should be
-interpreted as an ElasticSearch index ("my_index") and type
-("my_type"). You can have more fine-grained control over how data is
-written, see the various "field" options.
-
-You can also easily read data:
-
- $ wu-hadoop-es mapper.rb reducer.rb --input=es://my_index/my_type --output=/hdfs/output/path
-
-and even specify a query
-
- $ wu-hadoop-es mapper.rb reducer.rb --input=es://my_index/my_type --output=/hdfs/output/path --query='{"query_string": {"query": "hi +there"}}'
-
-There are several tuning options available. ElasticSearch
-configuration (and how to specify which cluster to find and join) are
-handled using ElasticSearch's usual configuration file based approach,
-specified by the --es_config option.
-EOF
-
-# These options we want to pass on to wu-hadoop but sometimes we want
-# to mess with one first so we'll have to reconstruct them later
-# ourselves.
-settings.define(:input, :description => "Comma-separated list of input paths: local, HDFS, or ElasticSearch", :wonderdog => true)
-settings.define(:output, :description => "Output path: local, HDFS, or ElasticSearch", :wonderdog => true)
-
-
-require 'wukong/boot' ; Wukong.boot!(settings)
-
-if settings.rest.empty?
- settings.dump_help
- exit(1)
-end
-
-Wukong::Elasticsearch::Driver.run(settings, settings.rest)
Please sign in to comment.
Something went wrong with that request. Please try again.