- Add Riak and Riaksearch cookbooks from github.com/damm/ey-riak and

    github.com/damm/ey-riaksearch.  They are still experimental and not
    supported, however they are free for you to test, use and help vet.

Signed-off-by: Scott M. Likens <slikens@engineyard.com>

cookbooks/riak/README.md

@@ -0,0 +1,82 @@
+Riak Cookbook for EngineYard AppCloud
+=========
+
+[Riak][1] is a Dynamo-inspired key/value store that scales predictably and easily. Riak also simplifies development by giving developers the ability to quickly prototype, test, and deploy their applications.
+
+A truly fault-tolerant system, Riak has no single point of failure. No machines are special or central in Riak, so developers and operations professionals can decide exactly how fault-tolerant they want and need their applications to be.
+
+Overview
+--------
+
+This cookbook once complete will attempt to provide one method of "Hosting" a Riak Ring on AppCloud.  It will not run inside your regular environment as this cookbook will attempt to achieve a scaleable stable Riak configuration with the least disruption of automation possible.
+
+Design
+--------
+
+* 2-3+ utility instances (m1.large or larger)
+
+* Riak 0.14 with Bitcask
+* Erlang R13B04
+* haproxy on each app instance listening on port 8098 directing to the utility instances.
+
+Notes
+--------
+
+This Cookbook automates the creation (join) action of a Riak 'Ring' on AppCloud.  As your needs may vary it is suggested to fork this recipe and make any customization you do on the fork.  You can omit the main cookbook it is only there for my testing purposes.
+
+Specifics of Usage
+--------
+
+Currently this Cookbook provides the following methods of using Riak:
+
+1. Riak K/V only
+
+  * Add an utility instance with the following naming scheme,
+
+  * riak_0
+  * riak_1
+  ...
+
+  * Note you must always start with _0 as that is the 'ring master'.  
+
+* Lastly, Words of Wisdom from Basho themselves.
+
+> you should look at the ring ready command and make it returns 0 before adding the next node
+> if you try and do more than 4 or 5 nodes the gossip is a little heavy for ec2 right now
+> and sometimes it takes a minute or two to converge the ring
+> changing the gossip interval in the conf alleviates this somewhat
+
+Depdencies
+--------
+
+This cookbook depends on the dnapi|emerge cookbook, you can add it as a
+submodule as follows,
+
+``git submodule update --init``  
+``git submodule add git://github.com/damm/ey-dnapi.git cookbooks/dnapi`` 
+``git submodule add git://github.com/damm/ey-emerge.git cookbooks/emerge``  
+
+Installation
+--------
+
+This cookbook can be added as a submodule, provided you have the proper
+dependencies you can add it as such,
+
+``git submodule add git://github.com/damm/ey-riak.git cookbooks/``  
+
+Add the following to your main/recipes/default.rb
+
+``require_recipe "riak"``  
+
+How to get Support
+--------
+
+* irc://irc.freenode.net/#riak
+* This Github repository.
+* This Cookbook provides a technology that is not listed in the Engine Yard [Technology Stack][2]
+
+* Additionally because of that there is *NO SUPPORT* for this recipe by EngineYard at this time.  If you have any problems with this reciple please open an issue, add a comment.  If you open a ticket regarding this cookbook you will be directed to this Github repository to open an issue.
+
+
+[1]: http://wiki.basho.com/display/RIAK/Riak
+[2]: http://www.engineyard.com/products/technology/stack

cookbooks/riak/attributes/riak.rb

@@ -0,0 +1,4 @@
+riak :version => "0.14.2", :js_vm_count => "8", :js_max_vm_mem => "8"
+luwak :enabled => "true"
+bitcask :max_file_size => "16#80000000", :open_timeout => 4, :sync_strategy => "none", :merge_window => "always", :frag_merge_trigger => "60", :dead_bytes_merge_trigger => "536870912", :frag_threshold => "40", :dead_bytes_threshold => "134217728", :small_file_threshold => "10485760", :expiry_secs => "-1", :data_root => "data/bitcask"
+

cookbooks/riak/definitions/s3_forceinstall.rb

@@ -0,0 +1,36 @@
+define :s3_forceinstall, :atom => nil do
+  version = params[:version]
+  name = params[:name]
+  atom = `echo #{name} | cut -d '/' -f 1`.chomp
+  full_name = name + '-' + version
+
+  directory "/engineyard/portage/packages" do
+    action :create
+    mode 0755
+    owner "root"
+    group "root"
+  end
+
+  directory "/engineyard/portage/packages/#{atom}" do
+    action :create
+    mode 0755
+    owner "root"
+    group "root"
+  end
+
+  execute "remove old package file" do
+    command "rm /engineyard/portage/packages/#{full_name}.tbz2"
+    only_if { File.exists?("/engineyard/portage/packages/#{full_name}.tbz2") }
+  end
+
+  remote_file "/engineyard/portage/packages/#{full_name}.tbz2" do
+    source "http://ey-portage.s3.amazonaws.com/#{node[:kernel][:machine]}/#{full_name}.tbz2"
+    not_if { FileTest.exists?("/engineyard/portage/packages/#{full_name}.tbz2") }
+    backup false
+  end
+
+  execute "emerge --ignore-default-opts /engineyard/portage/packages/#{full_name}.tbz2" do
+    action :run
+  end
+
+end

cookbooks/riak/definitions/s3_install.rb

@@ -0,0 +1,32 @@
+define :s3_install, :atom => nil do
+  version = params[:version]
+  name = params[:name]
+  atom = `echo #{name} | cut -d '/' -f 1`.chomp
+  full_name = name + '-' + version
+
+  directory "/engineyard/portage/packages" do
+    action :create
+    mode 0755
+    owner "root"
+    group "root"
+  end
+
+  directory "/engineyard/portage/packages/#{atom}" do
+    action :create
+    mode 0755
+    owner "root"
+    group "root"
+  end
+
+  remote_file "/engineyard/portage/packages/#{full_name}.tbz2" do
+    source "http://ey-portage.s3.amazonaws.com/#{node[:kernel][:machine]}/#{full_name}.tbz2"
+    not_if { FileTest.exists?("/engineyard/portage/packages/#{full_name}.tbz2") }
+    backup false
+  end
+
+  execute "emerge /engineyard/portage/packages/#{full_name}.tbz2" do
+    action :run
+  not_if { FileTest.directory?("/var/db/pkg/#{full_name}") }
+  end
+
+end

cookbooks/riak/libraries/ruby_block.rb

@@ -0,0 +1,40 @@
+
+class Chef
+  class Resource
+    class RubyBlock < Chef::Resource
+      def initialize(name, collection=nil, node=nil)
+        super(name, collection, node)
+        @resource_name = :ruby_block
+        @action = :create
+        @allowed_actions.push(:create)
+      end
+
+      def block(&block)
+        if block
+          @block = block
+        else
+          @block
+        end
+      end
+    end
+  end
+end
+
+
+class Chef
+  class Provider
+    class RubyBlock < Chef::Provider
+      def load_current_resource
+        Chef::Log.debug(@new_resource.inspect)
+        true
+      end
+
+      def action_create
+        @new_resource.block.call
+      end
+    end
+  end
+end
+
+Chef::Platform.platforms[:default].merge! :ruby_block => Chef::Provider::RubyBlock
+

cookbooks/riak/libraries/run_for_app.rb

@@ -0,0 +1,12 @@
+class Chef
+  class Recipe
+    def run_for_app(*apps, &block)
+      apps.map! {|a| a.to_s }
+      node[:applications].map{|k,v| [k,v] }.sort_by {|a,b| a }.each do |name, app_data|
+        if apps.include?(name)
+          block.call(name, app_data)
+        end
+      end
+    end
+  end
+end

cookbooks/riak/recipes/app_config.rb

@@ -0,0 +1,12 @@
+node[:applications].each do |app,data|
+  template "/data/#{app}/shared/config/ripple.yml" do
+    source "ripple.yml.erb"
+    owner node[:owner_name]
+    group node[:owner_name]
+    mode 0655
+    backup 0
+    variables({
+      :hostname => "localhost"
+    })
+  end
+end

cookbooks/riak/recipes/basho_bench.rb

@@ -0,0 +1,57 @@
+# AppCloud's GCC does not include Fortran by default, so let's install GCC binaries with fortran.
+
+if File.exists?("/usr/bin/gfortran")
+  # noop
+else
+  s3_forceinstall "sys-devel/gcc" do
+    version "4.1.2"
+  end
+end
+
+directory "/mnt/src" do
+  owner "root"
+  group "root"
+  action :create
+  mode 0755
+end
+remote_file "/mnt/src/R-2.13.0.tar.gz" do
+  source "http://cran.opensourceresources.org/src/base/R-2/R-2.13.0.tar.gz"
+  backup 0
+  owner "root"
+  group "root"
+  not_if { File.exists?("/mnt/src/R-2.13.0.tar.gz") }
+end
+
+package "x11-libs/libXt" do
+  version "1.0.5"
+  action :install
+end
+
+package "x11-libs/libICE" do
+  version "1.0.4"
+  action :install
+end
+
+execute "un-archive R" do
+  cwd "/mnt/src"
+  command "tar zxfv R-2.13.0.tar.gz"
+  not_if { File.exists?("/usr/bin/Rscript") }
+end
+
+execute "configure R" do
+  cwd "/mnt/src/R-2.13.0"
+  command "./configure --prefix=/usr"
+  not_if { File.exists?("/usr/bin/Rscript") }
+end
+
+execute "make R" do
+  cwd "/mnt/src/R-2.13.0"
+  command "make -j2"
+  not_if { File.exists?("/usr/bin/Rscript") }
+end
+
+execute "make install R" do
+  cwd "/mnt/src/R-2.13.0"
+  command "make install"
+  not_if { File.exists?("/usr/bin/Rscript") }
+end

cookbooks/riak/recipes/configure_riak.rb

@@ -0,0 +1,34 @@
+# Configure stuff goes here
+#
+
+template "/data/riak/etc/vm.args" do
+  source "vm.args.erb"
+  owner node[:owner_name]
+  group node[:owner_name]
+  mode 0655
+  backup 0
+end
+
+template "/data/riak/etc/app.config" do
+  source "app.config.erb"
+  backup 0
+  mode 0655
+  owner node[:owner_name]
+  group node[:owner_name]
+  variables(
+    :js_vm_count => node[:riak][:js_vm_count],
+    :js_max_vm_mem => node[:riak][:js_max_vm_mem],
+    :max_file_size => node[:bitcask][:max_file_size],
+    :open_timeout => node[:bitcask][:open_timeout],
+    :sync_strategy => node[:bitcask][:sync_strategy],
+    :merge_window => node[:bitcask][:merge_window], 
+    :frag_merge_trigger => node[:bitcask][:frag_merge_trigger],
+    :dead_bytes_merge_trigger => node[:bitcask][:dead_bytes_merge_trigger],
+    :frag_threshold => node[:bitcask][:frag_threshold],
+    :dead_bytes_threshold => node[:bitcask][:dead_bytes_threshold],
+    :small_file_threshold => node[:bitcask][:small_file_threshold],
+    :expiry_secs => node[:bitcask][:expiry_secs],
+    :data_root => node[:bitcask][:data_root],
+    :luwak => node[:luwak][:enabled]
+  )
+end

cookbooks/riak/recipes/default.rb

@@ -0,0 +1,53 @@
+# Pardon the dust in here. <-- sml 10-10-10..nil?
+
+# This portion of the recipe runs on all instances, except the utility instances.  You don't want haproxy on the utility instances but you do on solo|app|app_master
+if ['solo','app','app_master'].include?(node[:instance_role])
+  require_recipe "riak::haproxy"
+  #require_recipe "riak::basho_bench"
+  require_recipe "riak::app_config"
+  # Remove old version of Erlang that is NOT new enough
+  package "dev-lang/erlang" do
+    version "erlang-12.2.5-r1"
+    action :remove
+    only_if { FileTest.directory?("/var/db/pkg/dev-lang/erlang-12.2.5-r1") }
+  end
+
+# Install a version of Erlang (R130B4) provided by me.  I built the package, if you would like to see the ebuild sources make an issue.
+  s3_install "dev-lang/erlang" do
+    version "13.2.4"
+  end
+end
+
+# This portion of the recipe runs on only utility instances
+if ['util'].include?(node[:instance_role])
+  if node['name'].include?('riak_')
+# Remove old version of Erlang that is NOT new enough
+    package "dev-lang/erlang" do
+      version "erlang-12.2.5-r1"
+      action :remove
+      only_if { FileTest.directory?("/var/db/pkg/dev-lang/erlang-12.2.5-r1") }
+    end
+
+# Install a version of Erlang (R130B4) provided by me.  I built the package, if you would like to see the ebuild sources make an issue.
+    s3_install "dev-lang/erlang" do
+      version "13.2.4"
+    end
+
+# Install python_setuptools as the version in portage is too old.
+
+    include_recipe "riak::setuptools"
+
+# Install Mercurial as it's required for most things with Webmachine / rebar.
+
+    execute "easy_install mercurial" do
+      action :run
+      not_if { FileTest.exists?("/usr/bin/hg") }
+    end
+
+    include_recipe "riak::install_riak"
+    include_recipe "riak::configure_riak"
+    include_recipe "riak::start_riak"
+    include_recipe "riak::join_ring"
+    #include_recipe "riak::basho_bench"
+  end
+end