Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Revert "removed ElasticSearch recipe for refactoring"

This reverts commit b59816e.
  • Loading branch information...
commit ba4f7eb947bcc9965c79eb9d435c008e35b24a5c 1 parent b59816e
Ines Sombra authored
View
80 cookbooks/elasticsearch/README.md
@@ -0,0 +1,80 @@
+Elasticsearch Cookbook for AppCloud
+---------------
+
+You know, for Search
+
+So, we build a web site or an application and want to add search to it, and then it hits us: getting search working is hard. We want our search solution to be fast, we want a painless setup and a completely free search schema, we want to be able to index data simply using JSON over HTTP, we want our search server to be always available, we want to be able to start with one machine and scale to hundreds, we want real-time search, we want simple multi-tenancy, and we want a solution that is built for the cloud.
+
+"This should be easier", we declared, "and cool, bonsai cool".
+
+[elasticsearch][2] aims to solve all these problems and more. It is an Open Source (Apache 2), Distributed, RESTful, Search Engine built on top of [Lucene][1].
+
+Dependencies
+--------
+
+ * Your application should use gems(s) such as [tire][5],[rubberband][3],[elastic_searchable][6] or lastly for JRuby users there is [jruby-elasticsearch][4].
+
+Using it
+--------
+
+ * There is two ways to run this recipe. By default you can use the 'default' recipe and use this in an clustered configuration that requires utility instances. Alternatively you can use the alternate recipe called 'non_util' which will configure your app_master/solo instance to have elasticsearch. You would add to main/recipes/default.rb the following,
+
+``require_recipe "elasticsearch::non_util"``
+
+ * Otheriwse you would do the following
+
+``require_recipe "elasticsearch"``
+
+ * Now you should upload the recipes to your environment,
+
+``ey recipes upload -e <environment>``
+
+ * If you picked the non_util recipe you can ignore naming your utility instances. Upload the recipe, click apply and you should find the neccesary things done;otherwise name your utility instances like below.
+
+ * Add an utility instance with the following naming scheme(s)
+ * elasticsearch_0
+ * elasticsearch_1
+ * elasticsearch_2
+ * ...
+
+ * Produce /data/<appname>/shared/config/elasticsearch.yml on all instances so you can easily parse/configure elasticsearch for your usage.
+
+Plugins
+--------
+
+Rudamentary plugin support is there in a definition. You will need to update the template for configuration options for said plugin; if you wish to improve this functionality please submit a pull request.
+
+Examples:
+
+``es_plugin "cloud-aws" do``
+``action :install``
+``end``
+
+``es_plugin "transport-memcached" do``
+``action :remove``
+``end``
+
+
+Caveats
+--------
+
+plugin support is still not complete/automated. CouchDB and Memcached plugins may be worth investigtating, pull requests to improve this.
+
+Backups
+--------
+
+None automated, regular snapshot should work. If you have a large cluster this may complicate things, please consult the [elasticsearch][2] documentation regarding that.
+
+Warranty
+--------
+
+This cookbook is provided as is, there is no offer of support for this
+recipe by Engine Yard in any capacity. If you find bugs, please open an
+issue and submit a pull request.
+
+[1]: http://lucene.apache.org/
+[2]: http://www.elasticsearch.org/
+[3]: https://github.com/grantr/rubberband
+[4]: https://github.com/jordansissel/jruby-elasticsearch/
+[5]: https://github.com/karmi/tire
+[6]: https://github.com/wireframe/elastic_searchable/
View
9 cookbooks/elasticsearch/attributes/elasticsearch.rb
@@ -0,0 +1,9 @@
+elasticsearch_version("0.18.2")
+elasticsearch_checksum("ce1bad39c66b2eb1ec0286aa4c75a03e6d7ac076")
+elasticsearch_clustername("#{@attribute[:environment][:name]}")
+elasticsearch_home("/data/elasticsearch")
+elasticsearch_s3_gateway_bucket("elasticsearch_#{@attribute[:environment][:name]}")
+elasticsearch_heap_size(1000)
+elasticsearch_fdulimit(nil) # nofiles limit (make this something like 32768, see /etc/security/limits.conf)
+elasticsearch_defaultreplicas(1) # replicas are in addition to the original, so 1 replica means 2 copies of each shard
+elasticsearch_defaultshards(6) # 6*2 shards per index distributes evenly across 3, 4, or 6 nodes
View
23 cookbooks/elasticsearch/definitions/plugin.rb
@@ -0,0 +1,23 @@
+define :es_plugin, :name => nil do
+ name = params[:name]
+
+ case params[:action].to_sym
+ when :install
+ Chef::Log.info "attempting to install ElasticSearch plugin #{name}"
+
+ execute "plugin install #{name}" do
+ cwd "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}"
+ command "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/bin/plugin -install #{name}"
+ not_if { File.directory?("/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/plugins/#{name}") }
+ end
+
+ when :remove
+ Chef::Log.info "attempting to remove ElasticSearch plugin #{name}"
+
+ execute "plugin remove #{name}" do
+ cwd "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}"
+ command "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/bin/plugin -remove #{name}"
+ not_if { File.directory?("/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/plugins/#{name}") }
+ end
+ end
+end
View
6 cookbooks/elasticsearch/libraries/engineyard.rb
@@ -0,0 +1,6 @@
+require 'dnapi'
+class Chef::Node
+ def engineyard
+ @engineyard ||= DNApi.from(File.read("/etc/chef/dna.json"))
+ end
+end
View
176 cookbooks/elasticsearch/recipes/default.rb
@@ -0,0 +1,176 @@
+#
+# Cookbook Name:: elasticsearch
+# Recipe:: default
+#
+# Credit goes to GoTime for their original recipe ( http://cookbooks.opscode.com/cookbooks/elasticsearch )
+
+if ['util'].include?(node[:instance_role])
+ if node['utility_instances'].empty?
+ Chef::Log.info "No utility instances found"
+ else
+ elasticsearch_instances = []
+ elasticsearch_expected = 0
+ node['utility_instances'].each do |elasticsearch|
+ if elasticsearch['name'].include?("elasticsearch_")
+ elasticsearch_expected = elasticsearch_expected + 1 unless node['fqdn'] == elasticsearch['hostname']
+ elasticsearch_instances << "#{elasticsearch['hostname']}:9300" unless node['fqdn'] == elasticsearch['hostname']
+ end
+ end
+ end
+
+ Chef::Log.info "Downloading Elasticsearch v#{node[:elasticsearch_version]} checksum #{node[:elasticsearch_checksum]}"
+ remote_file "/tmp/elasticsearch-#{node[:elasticsearch_version]}.zip" do
+ source "https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-#{node[:elasticsearch_version]}.zip"
+ mode "0644"
+ checksum node[:elasticsearch_checksum]
+ not_if { File.exists?("/tmp/elasticsearch-#{node[:elasticsearch_version]}.zip") }
+ end
+
+ user "elasticsearch" do
+ uid 61021
+ gid "nogroup"
+ end
+
+ # Update JAVA as the Java on the AMI can sometimes crash
+ #
+ Chef::Log.info "Updating Sun JDK"
+ package "dev-java/sun-jdk" do
+ version "1.6.0.26"
+ action :upgrade
+ end
+
+ directory "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}" do
+ owner "root"
+ group "root"
+ mode 0755
+ end
+
+ ["/var/log/elasticsearch", "/var/lib/elasticsearch", "/var/run/elasticsearch"].each do |dir|
+ directory dir do
+ owner "root"
+ group "root"
+ mode 0755
+ end
+ end
+
+ bash "unzip elasticsearch" do
+ user "root"
+ cwd "/tmp"
+ code %(unzip /tmp/elasticsearch-#{node[:elasticsearch_version]}.zip)
+ not_if { File.exists? "/tmp/elasticsearch-#{node[:elasticsearch_version]}" }
+ end
+
+ bash "copy elasticsearch root" do
+ user "root"
+ cwd "/tmp"
+ code %(cp -r /tmp/elasticsearch-#{node[:elasticsearch_version]}/* /usr/lib/elasticsearch-#{node[:elasticsearch_version]})
+ not_if { File.exists? "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/lib" }
+ end
+
+ directory "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/plugins" do
+ owner "root"
+ group "root"
+ mode 0755
+ end
+
+ link "/usr/lib/elasticsearch" do
+ to "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}"
+ end
+
+ directory "#{node[:elasticsearch_home]}" do
+ owner "elasticsearch"
+ group "nogroup"
+ mode 0755
+ end
+
+ directory "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/data" do
+ owner "root"
+ group "root"
+ mode 0755
+ action :create
+ recursive true
+ end
+
+ mount "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/data" do
+ device "#{node[:elasticsearch_home]}"
+ fstype "none"
+ options "bind,rw"
+ action :mount
+ end
+
+ template "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/config/logging.yml" do
+ source "logging.yml.erb"
+ mode 0644
+ end
+
+ directory "/usr/share/elasticsearch" do
+ group "elasticsearch"
+ group "nogroup"
+ mode 0755
+ end
+
+ max_mem = ((node[:memory][:total].to_i / 1024 * 0.75)).to_i.to_s + "m"
+ template "/usr/share/elasticsearch/elasticsearch.in.sh" do
+ source "elasticsearch.in.sh.erb"
+ mode 0644
+ backup 0
+ variables(
+ :es_max_mem => ((node[:memory][:total].to_i / 1024 * 0.75)).to_i.to_s + "m"
+ )
+ end
+
+ # include_recipe "elasticsearch::s3_bucket"
+ template "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/config/elasticsearch.yml" do
+ source "elasticsearch.yml.erb"
+ owner "elasticsearch"
+ group "nogroup"
+ variables(
+ :aws_access_key => node[:aws_secret_key],
+ :aws_access_id => node[:aws_secret_id],
+ :elasticsearch_s3_gateway_bucket => node[:elasticsearch_s3_gateway_bucket],
+ :elasticsearch_instances => elasticsearch_instances.join('", "'),
+ :elasticsearch_defaultreplicas => node[:elasticsearch_defaultreplicas],
+ :elasticsearch_expected => elasticsearch_expected,
+ :elasticsearch_defaultshards => node[:elasticsearch_defaultshards],
+ :elasticsearch_clustername => node[:elasticsearch_clustername]
+ )
+ mode 0600
+ backup 0
+ end
+
+ template "/etc/monit.d/elasticsearch_#{node[:elasticsearch_clustername]}.monitrc" do
+ source "elasticsearch.monitrc.erb"
+ owner "elasticsearch"
+ group "nogroup"
+ backup 0
+ mode 0644
+ end
+
+ # Tell monit to just reload, if elasticsearch is not running start it. If it is monit will do nothing.
+ execute "monit reload" do
+ command "monit reload"
+ end
+end
+
+# This portion of the recipe should run on all instances in your environment. We are going to drop elasticsearch.yml for you so you can parse it and provide the instances to your application.
+if ['solo','app_master','app','util'].include?(node[:instance_role])
+ elasticsearch_hosts = []
+ node['utility_instances'].each do |elasticsearch|
+ if elasticsearch['name'].include?("elasticsearch_")
+ elasticsearch_hosts << "#{elasticsearch['hostname']}:9200"
+ end
+
+ node.engineyard.apps.each do |app|
+ template "/data/#{app.name}/shared/config/elasticsearch.yml" do
+ owner node[:owner_name]
+ group node[:owner_name]
+ mode 0660
+ source "es.yml.erb"
+ backup 0
+ variables(:yaml_file => {
+ node.engineyard.environment.framework_env => {
+ :hosts => elasticsearch_hosts} })
+ end
+ end
+ end
+end
View
127 cookbooks/elasticsearch/recipes/non_util.rb
@@ -0,0 +1,127 @@
+#
+# Cookbook Name:: elasticsearch
+# Recipe:: default
+#
+# Credit goes to GoTime for their original recipe ( http://cookbooks.opscode.com/cookbooks/elasticsearch )
+
+if ['solo','app_master'].include?(node[:instance_role])
+ Chef::Log.info "Downloading Elasticsearch v#{node[:elasticsearch_version]} checksum #{node[:elasticsearch_checksum]}"
+ remote_file "/tmp/elasticsearch-#{node[:elasticsearch_version]}.zip" do
+ source "https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-#{node[:elasticsearch_version]}.zip"
+ mode "0644"
+ checksum node[:elasticsearch_checksum]
+ not_if { File.exists?("/tmp/elasticsearch-#{node[:elasticsearch_version]}.zip") }
+ end
+
+ user "elasticsearch" do
+ uid 61021
+ gid "nogroup"
+ end
+
+ # Update JAVA as the Java on the AMI can sometimes crash
+ #
+ Chef::Log.info "Updating Sun JDK"
+ package "dev-java/sun-jdk" do
+ version "1.6.0.26"
+ action :upgrade
+ end
+
+ directory "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}" do
+ owner "root"
+ group "root"
+ mode 0755
+ end
+
+ ["/var/log/elasticsearch", "/var/lib/elasticsearch", "/var/run/elasticsearch"].each do |dir|
+ directory dir do
+ owner "root"
+ group "root"
+ mode 0755
+ end
+ end
+
+ bash "unzip elasticsearch" do
+ user "root"
+ cwd "/tmp"
+ code %(unzip /tmp/elasticsearch-#{node[:elasticsearch_version]}.zip)
+ not_if { File.exists? "/tmp/elasticsearch-#{node[:elasticsearch_version]}" }
+ end
+
+ bash "copy elasticsearch root" do
+ user "root"
+ cwd "/tmp"
+ code %(cp -r /tmp/elasticsearch-#{node[:elasticsearch_version]}/* /usr/lib/elasticsearch-#{node[:elasticsearch_version]})
+ not_if { File.exists? "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/lib" }
+ end
+
+ directory "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/plugins" do
+ owner "root"
+ group "root"
+ mode 0755
+ end
+
+ link "/usr/lib/elasticsearch" do
+ to "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}"
+ end
+
+ directory "#{node[:elasticsearch_home]}" do
+ owner "elasticsearch"
+ group "nogroup"
+ mode 0755
+ end
+
+ directory "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/data" do
+ owner "root"
+ group "root"
+ mode 0755
+ action :create
+ recursive true
+ end
+
+ mount "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/data" do
+ device "#{node[:elasticsearch_home]}"
+ fstype "none"
+ options "bind,rw"
+ action :mount
+ end
+
+ template "/usr/lib/elasticsearch-#{node[:elasticsearch_version]}/config/logging.yml" do
+ source "logging.yml.erb"
+ mode 0644
+ end
+
+ directory "/usr/share/elasticsearch" do
+ group "elasticsearch"
+ group "nogroup"
+ mode 0755
+ end
+
+ template "/etc/monit.d/elasticsearch_#{node[:elasticsearch_clustername]}.monitrc" do
+ source "elasticsearch.monitrc.erb"
+ owner "elasticsearch"
+ group "nogroup"
+ backup 0
+ mode 0644
+ end
+
+ # Tell monit to just reload, if elasticsearch is not running start it. If it is monit will do nothing.
+ execute "monit reload" do
+ command "monit reload"
+ end
+end
+
+solo = node[:instance_role] == 'solo'
+if ['solo','app_master','app','util'].include?(node[:instance_role])
+ node.engineyard.apps.each do |app|
+ template "/data/#{app.name}/shared/config/elasticsearch.yml" do
+ owner node[:owner_name]
+ group node[:owner_name]
+ mode 0660
+ source "es.yml.erb"
+ backup 0
+ variables(:yaml_file => {
+ node.engineyard.environment.framework_env => {
+ :hosts => solo ? "127.0.0.1:9200" : "#{node[:master_app_server][:public_ip]}:9200" }})
+ end
+ end
+end
View
16 cookbooks/elasticsearch/recipes/s3_bucket.rb
@@ -0,0 +1,16 @@
+ruby_block "create_bucket" do
+ block do
+ require 'fog'
+
+ connection = Fog::Storage.new(
+ :provider => 'AWS',
+ :aws_secret_access_key => node.engineyard.environment.aws_secret_key,
+ :aws_access_key_id => node.engineyard.environment.aws_secret_id
+ )
+
+ bucket = connection.directories.create(
+ :key => "elasticsearch_#{node['environment']['name']}",
+ :public => false
+ )
+ end
+end
View
52 cookbooks/elasticsearch/templates/default/elasticsearch.in.sh.erb
@@ -0,0 +1,52 @@
+ES_CLASSPATH=$ES_CLASSPATH:$ES_HOME/lib/elasticsearch-0.16.3.jar:$ES_HOME/lib/*:$ES_HOME/lib/sigar/*
+
+if [ "x$ES_MIN_MEM" = "x" ]; then
+ ES_MIN_MEM=256m
+fi
+if [ "x$ES_MAX_MEM" = "x" ]; then
+ ES_MAX_MEM=1300m
+fi
+
+# min and max heap sizes should be set to the same value to avoid
+# stop-the-world GC pauses during resize, and so that we can lock the
+# heap in memory on startup to prevent any of it from being swapped
+# out.
+JAVA_OPTS="$JAVA_OPTS -Xms${ES_MIN_MEM}"
+JAVA_OPTS="$JAVA_OPTS -Xmx${ES_MAX_MEM}"
+
+# reduce the per-thread stack size
+JAVA_OPTS="$JAVA_OPTS -Xss128k"
+
+JAVA_OPTS="$JAVA_OPTS -Djline.enabled=true"
+
+# Enable aggressive optimizations in the JVM
+# - Disabled by default as it might cause the JVM to crash
+# JAVA_OPTS="$JAVA_OPTS -XX:+AggressiveOpts"
+
+# Enable reference compression, reducing memory overhead on 64bit JVMs
+# - Disabled by default as it is not stable for Sun JVM before 6u19
+#JAVA_OPTS="$JAVA_OPTS -XX:+UseCompressedOops"
+
+# This stops an annoying warning
+JAVA_OPTS="$JAVA_OPTS -server"
+JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
+JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
+JAVA_OPTS="$JAVA_OPTS -XX:+CMSParallelRemarkEnabled"
+JAVA_OPTS="$JAVA_OPTS -XX:SurvivorRatio=8"
+JAVA_OPTS="$JAVA_OPTS -XX:MaxTenuringThreshold=1"
+JAVA_OPTS="$JAVA_OPTS -XX:CMSInitiatingOccupancyFraction=75"
+JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
+
+# GC logging options -- uncomment to enable
+# JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
+# JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
+# JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
+# JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
+# JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
+# JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
+
+# Causes the JVM to dump its heap on OutOfMemory.
+JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
+# The path to the heap dump location, note directory must exists and have enough
+# space for a full heap dump.
+#JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=$ES_HOME/logs/heapdump.hprof"
View
5 cookbooks/elasticsearch/templates/default/elasticsearch.monitrc.erb
@@ -0,0 +1,5 @@
+check process elasticsearch_9080
+with pidfile /var/run/elasticsearch.pid
+ start program = "/usr/lib/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch.pid" with timeout 90 seconds
+ stop program = "/bin/bash -c '/bin/kill `cat /var/run/elasticsearch.pid`'" with timeout 90 seconds
+ group elasticsearch
View
20 cookbooks/elasticsearch/templates/default/elasticsearch.yml.erb
@@ -0,0 +1,20 @@
+path:
+ work: /var/lib/elasticsearch/work
+ logs: /var/log/elasticsearch
+ data: /data/elasticsearch
+
+
+cluster:
+ name: <%= @elasticsearch_clustername %>
+
+discovery:
+ zen:
+ minimum_master_nodes: 1
+ ping.timeout: 3s
+ ping.multicast.enabled: false
+ ping.unicast.hosts: ["<%= @elasticsearch_instances %>"]
+
+gateway:
+ recover_after_nodes: 1
+ recover_after_time: 5m
+ expected_nodes: <%= @elasticsearch_expected %>
View
6 cookbooks/elasticsearch/templates/default/es.monitrc
@@ -0,0 +1,6 @@
+check process elasticsearch_9080
+with pidfile /var/run/solr/solr.pid
+ start program = "/engineyard/bin/esr start" as uid <%= @user %> and gid <%= @group %>
+ stop program = "/engineyard/bin/es stop" as uid <%= @user %> and gid <%= @group %>
+ if totalmem is greater than 512 MB for 2 cycles then restart # eating up memory?
+ group solr
View
2  cookbooks/elasticsearch/templates/default/es.yml.erb
@@ -0,0 +1,2 @@
+<% require 'yaml' -%>
+<%=@yaml_file.to_yaml%>
View
21 cookbooks/elasticsearch/templates/default/logging.yml.erb
@@ -0,0 +1,21 @@
+rootLogger: INFO, console, file
+logger:
+ jgroups: WARN
+ # log action execution errors for easier debugging
+ action : INFO
+
+appender:
+ console:
+ type: console
+ layout:
+ type: consolePattern
+ conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n"
+
+ file:
+ type: dailyRollingFile
+ file: ${path.logs}/${cluster.name}.log
+ datePattern: "'.'yyyy-MM-dd"
+ layout:
+ type: pattern
+ conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n"
+
Please sign in to comment.
Something went wrong with that request. Please try again.