Change how attributes and resource parameters work

- Node attributes and the `elasticsearch::default` recipe:

  1. We're now using `node['elasticsearch'][<resource>][<param>]` and we automatically post any of those values to the corresponding resource in `elasticsearch::default`. For example, if you set `node['elasticsearch']['user']['shell']='/bin/sh'`, we will automatically call `elasticsearch_user_resource.shell '/bin/sh'`. This allows people to set as much as they want in the node attributes and still use the ::default recipe, and we'll pass it to the appropriate places.

  2. We now use strings for the type of install (e.g. 'package' or 'tarball', not :package or :tarball). Also, 'tar' has been dropped as a supported value.

- Installing elasticsearch

  1. We now install the tarball and package files into the same locations. There's no more `/usr/local`; it's all in one place. This allows us to drop a huge part of the default values for non-package installs. You may still override the values to be `/usr/local`, we just don't give you the defaults for it.

  2. We dropped the fancy logic for figuring out the requested version of Elasticsearch to be installed. You should pass it on the resource or in the recipe, but we no longer do a bunch of logic to figure out what you meant -- we favor being explicit now.

  3. We delete the sample configs directory when doing a tarball installation. It was overriding the stuff in /etc and /lib.

- Configuring elasticsearch

  1. We've dropped the `gc_options` parameter of elasticsearch_configure, and now have `jvm.options`. We've also dropped thread_stack_size and env_options, as they aren't used in the upstream packaging as defaults anymore. We added `max_map_count` since that was added in the packaging as well.

  2. Systemd support required we create `/usr/lib/systemd/system` if it didn't exist, but `/usr/lib/systemd` did exist.

- Running elasticsearch

  1. We now start the service by default, instead of only `:enable` but not `:start`.

  2. The systemd packaging scripts don't allow the override of ES_USER or ES_GROUP env vars to take effect. We've filed an upstream bug about this.

- Cookbook testing

  1. Due to the new attribute handling in the recipe, we're using `::default` a lot more in our tests.

Fixes #497.
Fixes #512.

.kitchen.yml

@@ -52,7 +52,8 @@ suites:
       - recipe[elasticsearch_test::default_with_plugins]
     attributes:
       elasticsearch:
-        install_type: package
+        install:
+          type: package
 
   - name: tarball # install by tarball
     require_chef_omnibus: 12.15.19
@@ -61,7 +62,8 @@ suites:
       - recipe[elasticsearch_test::default_with_plugins]
     attributes:
       elasticsearch:
-        install_type: tarball
+        install:
+          type: tarball
 
   - name: override_package # the override-everything use case
     driver_config:

README.md

@@ -25,16 +25,16 @@ the version parameter as a string into your download_url.
 |Name|Default|Other values|
 |----|-------|------------|
 |`default['elasticsearch']['version']`|`'5.0.0'`|[See list](attributes/default.rb).|
-|`default['elasticsearch']['install_type']`|`:package`|`:tarball`|
+TODO: |`default['elasticsearch']['install_type']`|`:package`|`:tarball`|
 |`default['elasticsearch']['download_urls']['debian']`|[See values](attributes/default.rb).|`%s` will be replaced with the version attribute above|
 |`default['elasticsearch']['download_urls']['rhel']`|[See values](attributes/default.rb).|`%s` will be replaced with the version attribute above|
-|`default['elasticsearch']['download_urls']['tar']`|[See values](attributes/default.rb).|`%s` will be replaced with the version attribute above|
+|`default['elasticsearch']['download_urls']['tarball']`|[See values](attributes/default.rb).|`%s` will be replaced with the version attribute above|
 
 ## Recipes
 
 Resources are the intended way to consume this cookbook, however we have
 provided a single recipe that configures Elasticsearch by downloading an archive
-containing a distribution of Elasticsearch, and extracting that into /usr/local.
+containing a distribution of Elasticsearch, and extracting that into `/usr/share`.
 
 See the attributes section above to for what defaults you can adjust.
 
@@ -220,28 +220,26 @@ Very complicated -
 ```ruby
 elasticsearch_configure 'my_elasticsearch' do
   # if you override one of these, you probably want to override all
-  path_home     tarball: "/opt/elasticsearch"
-  path_conf     tarball: "/etc/opt/elasticsearch"
-  path_data     tarball: "/var/opt/elasticsearch"
-  path_logs     tarball: "/var/log/elasticsearch"
-  path_pid      tarball: "/var/run/elasticsearch"
-  path_plugins  tarball: "/opt/elasticsearch/plugins"
-  path_bin      tarball: "/opt/elasticsearch/bin"
+  path_home     "/opt/elasticsearch"
+  path_conf     "/etc/opt/elasticsearch"
+  path_data     "/var/opt/elasticsearch"
+  path_logs     "/var/log/elasticsearch"
+  path_pid      "/var/run/elasticsearch"
+  path_plugins  "/opt/elasticsearch/plugins"
+  path_bin      "/opt/elasticsearch/bin"
 
   logging({:"action" => 'INFO'})
 
   allocated_memory '123m'
-  thread_stack_size '512k'
 
-  env_options '-DFOO=BAR'
-  gc_settings <<-CONFIG
+  jvm_options %w(
                 -XX:+UseParNewGC
                 -XX:+UseConcMarkSweepGC
                 -XX:CMSInitiatingOccupancyFraction=75
                 -XX:+UseCMSInitiatingOccupancyOnly
                 -XX:+HeapDumpOnOutOfMemoryError
                 -XX:+PrintGCDetails
-              CONFIG
+              )
 
   configuration ({
     'node.name' => 'crazy'

attributes/default.rb

@@ -1,15 +1,19 @@
-# elasticsearch version & install type
-default['elasticsearch']['version'] = '5.0.0'
-default['elasticsearch']['install_type'] = :package
+# empty settings (populate these for the elasticsearch::default recipe)
+# see the resources or README.md to see what you can pass here.
+default['elasticsearch']['user'] = {}
+default['elasticsearch']['install'] = {}
+default['elasticsearch']['configure'] = {}
+default['elasticsearch']['service'] = {}
+default['elasticsearch']['plugin'] = {}
 
 # platform_family keyed download URLs
 default['elasticsearch']['download_urls'] = {
   'debian' => 'https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-%s.deb',
   'rhel' => 'https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-%s.rpm',
-  'tar' => 'https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-%s.tar.gz'
+  'tarball' => 'https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-%s.tar.gz'
 }
 
 # platform_family keyed download sha256 checksums
 default['elasticsearch']['checksums']['5.0.0']['debian'] = '0a4f8842a1f7d7bd3015118298284383efcd4c25f9591c46bb5ab68189815268'
 default['elasticsearch']['checksums']['5.0.0']['rhel'] = 'fb502a9754f2162f27590125aecc6b68fa999738051d8f230c4da4ed6deb8d62'
-default['elasticsearch']['checksums']['5.0.0']['tar'] = 'a866534f0fa7428e980c985d712024feef1dee04709add6e360fc7b73bb1e7ae'
+default['elasticsearch']['checksums']['5.0.0']['tarball'] = 'a866534f0fa7428e980c985d712024feef1dee04709add6e360fc7b73bb1e7ae'

libraries/helpers.rb

@@ -61,61 +61,39 @@ def find_instance_name_resource(run_context, resource_type, instance_name)
       nil # falsey
     end
 
-    def determine_version(new_resource, node)
-      if new_resource.version
-        new_resource.version.to_s
-      elsif node['elasticsearch'] && node['elasticsearch']['version']
-        node['elasticsearch']['version'].to_s
-      else
-        raise 'could not determine version of elasticsearch to install'
-      end
-    end
-
-    def determine_install_type(new_resource, node)
-      if new_resource.type
-        new_resource.type.to_s
-      elsif node['elasticsearch'] && node['elasticsearch']['install_type']
-        node['elasticsearch']['install_type'].to_s
-      else
-        raise 'could not determine how to install elasticsearch (package? tarball?)'
-      end
-    end
-
     def determine_download_url(new_resource, node)
       platform_family = node['platform_family']
-      install_type = determine_install_type(new_resource, node)
-      version = determine_version(new_resource, node)
 
       url_string = nil
       if new_resource.download_url
         url_string = new_resource.download_url
-      elsif install_type.to_s == 'tar' || install_type.to_s == 'tarball'
-        url_string = node['elasticsearch']['download_urls']['tar']
-      elsif install_type.to_s == 'package' && node['elasticsearch']['download_urls'][platform_family]
+      elsif new_resource.type == 'tarball'
+        url_string = node['elasticsearch']['download_urls']['tarball']
+      elsif new_resource.type == 'package' && node['elasticsearch']['download_urls'][platform_family]
         url_string = node['elasticsearch']['download_urls'][platform_family]
       end
 
-      if url_string && version
-        return format(url_string, version)
+      if url_string && new_resource.version
+        return format(url_string, new_resource.version)
       elsif url_string
         return url_string
       end
     end
 
     def determine_download_checksum(new_resource, node)
       platform_family = node['platform_family']
-      install_type = determine_install_type(new_resource, node)
-      version = determine_version(new_resource, node)
+      install_type = new_resource.type
+      version = new_resource.version
 
       if new_resource.download_checksum
         new_resource.download_checksum
-      elsif install_type.to_s == 'tar' || install_type.to_s == 'tarball'
+      elsif install_type == 'tarball'
         node && version &&
           node['elasticsearch'] &&
           node['elasticsearch']['checksums'] &&
           node['elasticsearch']['checksums'][version] &&
-          node['elasticsearch']['checksums'][version]['tar']
-      elsif install_type.to_s == 'package' && node['elasticsearch']['checksums'][version] && node['elasticsearch']['checksums'][version][platform_family]
+          node['elasticsearch']['checksums'][version]['tarball']
+      elsif install_type == 'package' && node['elasticsearch']['checksums'][version] && node['elasticsearch']['checksums'][version][platform_family]
         node && version && platform_family &&
           node['elasticsearch'] &&
           node['elasticsearch']['checksums'] &&

libraries/provider_configure.rb

@@ -17,16 +17,16 @@ def whyrun_supported?
     default_configuration = new_resource.default_configuration.dup
     # if a subdir parameter is missing but dir is set, infer the subdir name
     # then go and be sure it's also set in the YML hash if it wasn't given there
-    if new_resource.path_conf[es_install.type] && default_configuration['path.conf'].nil?
-      default_configuration['path.conf'] = new_resource.path_conf[es_install.type]
+    if new_resource.path_conf && default_configuration['path.conf'].nil?
+      default_configuration['path.conf'] = new_resource.path_conf
     end
 
-    if new_resource.path_data[es_install.type] && default_configuration['path.data'].nil?
-      default_configuration['path.data'] = new_resource.path_data[es_install.type]
+    if new_resource.path_data && default_configuration['path.data'].nil?
+      default_configuration['path.data'] = new_resource.path_data
     end
 
-    if new_resource.path_logs[es_install.type] && default_configuration['path.logs'].nil?
-      default_configuration['path.logs'] = new_resource.path_logs[es_install.type]
+    if new_resource.path_logs && default_configuration['path.logs'].nil?
+      default_configuration['path.logs'] = new_resource.path_logs
     end
 
     # calculation for memory allocation; 50% or 31g, whatever is smaller
@@ -38,7 +38,7 @@ def whyrun_supported?
 
     # Create ES directories
     #
-    [new_resource.path_conf[es_install.type], "#{new_resource.path_conf[es_install.type]}/scripts", new_resource.path_logs[es_install.type]].each do |path|
+    [new_resource.path_conf, "#{new_resource.path_conf}/scripts", new_resource.path_logs].each do |path|
       d = directory path do
         owner es_user.username
         group es_user.groupname
@@ -52,7 +52,7 @@ def whyrun_supported?
 
     # Create data path directories
     #
-    data_paths = new_resource.path_data[es_install.type].is_a?(Array) ? new_resource.path_data[es_install.type] : new_resource.path_data[es_install.type].split(',')
+    data_paths = new_resource.path_data.is_a?(Array) ? new_resource.path_data : new_resource.path_data.split(',')
 
     data_paths.each do |path|
       d = directory path.strip do
@@ -76,19 +76,19 @@ def whyrun_supported?
     # We provide these values as resource attributes/parameters directly
 
     params = {}
-    params[:ES_HOME] = new_resource.path_home[es_install.type]
+    params[:ES_HOME] = new_resource.path_home
     params[:JAVA_HOME] = new_resource.java_home
-    params[:CONF_DIR] = new_resource.path_conf[es_install.type]
-    params[:DATA_DIR] = new_resource.path_data[es_install.type]
-    params[:LOG_DIR] = new_resource.path_logs[es_install.type]
-    params[:PID_DIR] = new_resource.path_pid[es_install.type]
+    params[:CONF_DIR] = new_resource.path_conf
+    params[:DATA_DIR] = new_resource.path_data
+    params[:LOG_DIR] = new_resource.path_logs
+    params[:PID_DIR] = new_resource.path_pid
     params[:RESTART_ON_UPGRADE] = new_resource.restart_on_upgrade
     params[:ES_USER] = es_user.username
     params[:ES_GROUP] = es_user.groupname
     params[:ES_STARTUP_SLEEP_TIME] = new_resource.startup_sleep_seconds.to_s
     params[:MAX_OPEN_FILES] = new_resource.nofile_limit
     params[:MAX_LOCKED_MEMORY] = new_resource.memlock_limit
-    params[:MAX_MAP_COUNT] = new_resource.memlock_limit
+    params[:MAX_MAP_COUNT] = new_resource.max_map_count
 
     default_config_name = es_svc.service_name || es_svc.instance_name || new_resource.instance_name || 'elasticsearch'
 
@@ -106,7 +106,7 @@ def whyrun_supported?
     # Create jvm.options file
     #
     jvm_options_template = template 'jvm_options' do
-      path   "#{new_resource.path_conf[es_install.type]}/jvm.options"
+      path   "#{new_resource.path_conf}/jvm.options"
       source new_resource.template_jvm_options
       cookbook new_resource.cookbook_jvm_options
       owner es_user.username
@@ -125,7 +125,7 @@ def whyrun_supported?
     # Create ES logging file
     #
     logging_template = template 'log4j2_properties' do
-      path   "#{new_resource.path_conf[es_install.type]}/log4j2.properties"
+      path   "#{new_resource.path_conf}/log4j2.properties"
       source new_resource.template_log4j2_properties
       cookbook new_resource.cookbook_log4j2_properties
       owner es_user.username
@@ -148,7 +148,7 @@ def whyrun_supported?
     end
 
     yml_template = template 'elasticsearch.yml' do
-      path "#{new_resource.path_conf[es_install.type]}/elasticsearch.yml"
+      path "#{new_resource.path_conf}/elasticsearch.yml"
       source new_resource.template_elasticsearch_yml
       cookbook new_resource.cookbook_elasticsearch_yml
       owner es_user.username

libraries/provider_install.rb

@@ -11,26 +11,19 @@ def whyrun_supported?
   end
 
   action :install do
-    install_type = determine_install_type(new_resource, node)
-    unless new_resource.version
-      new_resource.version determine_version(new_resource, node)
-    end
-
-    if install_type == 'tarball' || install_type == 'tar'
+    if new_resource.type == 'tarball'
       install_tarball_wrapper_action
-    elsif install_type == 'package'
+    elsif new_resource.type == 'package'
       install_package_wrapper_action
     else
       raise "#{install_type} is not a valid install type"
     end
   end
 
   action :remove do
-    install_type = determine_install_type(new_resource, node)
-
-    if install_type == 'tarball' || install_type == 'tar'
+    if new_resource.type == 'tarball'
       remove_tarball_wrapper_action
-    elsif install_type == 'package'
+    elsif new_resource.type == 'package'
       remove_package_wrapper_action
     else
       raise "#{install_type} is not a valid install type"
@@ -101,15 +94,15 @@ def install_tarball_wrapper_action
       url   determine_download_url(new_resource, node)
       owner es_user.username
       group es_user.groupname
-      version determine_version(new_resource, node)
+      version new_resource.version
       has_binaries ['bin/elasticsearch', 'bin/elasticsearch-plugin']
       checksum determine_download_checksum(new_resource, node)
-      prefix_root   new_resource.dir[new_resource.type]
-      prefix_home   new_resource.dir[new_resource.type]
+      prefix_root   new_resource.dir
+      prefix_home   new_resource.dir
 
       not_if do
         link   = "#{new_resource.dir}/elasticsearch"
-        target = "#{new_resource.dir}/elasticsearch-#{determine_version(new_resource, node)}"
+        target = "#{new_resource.dir}/elasticsearch-#{new_resource.version}"
         binary = "#{target}/bin/elasticsearch"
 
         ::File.directory?(link) && ::File.symlink?(link) && ::File.readlink(link) == target && ::File.exist?(binary)
@@ -118,14 +111,24 @@ def install_tarball_wrapper_action
     end
     ark_r.run_action(:install)
     new_resource.updated_by_last_action(true) if ark_r.updated_by_last_action?
+
+    # destroy the sample config directory for tarball installs, or it will
+    # take precedence beyond the default stuff in /etc/elasticsearch and within
+    # /etc/sysconfig or /etc/default
+    sample_r = directory "#{new_resource.dir}/elasticsearch/config" do
+      action :nothing
+      recursive true
+    end
+    sample_r.run_action(:delete)
+    new_resource.updated_by_last_action(true) if sample_r.updated_by_last_action?
   end
 
   def remove_tarball_wrapper_action
     # remove the symlink to this version
     link_r = link "#{new_resource.dir}/elasticsearch" do
       only_if do
         link   = "#{new_resource.dir}/elasticsearch"
-        target = "#{new_resource.dir}/elasticsearch-#{determine_version(new_resource, node)}"
+        target = "#{new_resource.dir}/elasticsearch-#{new_resource.version}"
 
         ::File.directory?(link) && ::File.symlink?(link) && ::File.readlink(link) == target
       end
@@ -135,7 +138,7 @@ def remove_tarball_wrapper_action
     new_resource.updated_by_last_action(true) if link_r.updated_by_last_action?
 
     # remove the specific version
-    d_r = directory "#{new_resource.dir}/elasticsearch-#{determine_version(new_resource, node)}" do
+    d_r = directory "#{new_resource.dir}/elasticsearch-#{new_resource.version}" do
       action :nothing
     end
     d_r.run_action(:delete)