Description

This cookbook installs and configures the elasticsearch search engine and database.

It requires a working Java installation on the target node. The cookbook downloads the elasticsearch tarball from Github, unpacks and moves it to the directory you have specified in the node configuration (/usr/local by default).

It also installs a service which enables you to start, stop, restart and check status of elasticsearch.

If your node has the monit recipe available, it will also create a configuration file for Monit, which will check whether elasticsearch is running, reachable by HTTP and the cluster is in the “green” state.

If you include the elasticsearch::plugin_aws recipe, the appropriate plugin will be installed for you, allowing you to use Amazon AWS features: node auto-discovery and S3/EBS persistence. You may set your AWS credentials either in a “elasticsearch/aws” data bag, or directly in the node configuration.

You may want to include the elasticsearch::proxy_nginx recipe, which will configure Nginx as a reverse proxy so you may access elasticsearch remotely with HTTP Authentication. (Be sure to include a nginx cookbook in your node setup as well.)

The cookbook also provides the elasticsearch::test recipe, which populates the test_chef_cookbook index with some sample data to check if the installation, the S3 persistence, etc is working.

Usage

Include the elasticsearch recipe in the run_list of a node. Then, upload the cookbook to the Chef server:

    knife cookbook upload elasticsearch

To enable the Amazon AWS related features, include the elasticsearch::plugin_aws recipe. You will need to configure the AWS credentials, bucket names, etc.

You may do that in the node configuration (with knife node edit MYNODE or at the Chef console), but it is probably more convenient to store the information in a "elasticsearch" data bag:

    mkdir -p ./data_bags/elasticsearch
    echo '{ 
      "id" : "aws",
      "discovery" : { "type": "ec2" },

      "gateway" : {
        "type"    : "s3",
        "s3"      : { "bucket": "YOUR BUCKET NAME" }
      },

      "cloud"   : {
        "aws"     : { "access_key": "YOUR ACCESS KEY", "secret_key": "YOUR SECRET ACCESS KEY" },
        "ec2"     : { "security_group": "elasticsearch" }
      }
    }' >> ./data_bags/elasticsearch/aws.json

Do not forget to upload the data bag to the Chef server:

    knife data bag from file elasticsearch aws.json

Usually, you will restrict the access to elasticsearch. However, it's convenient to be able to connect to the elasticsearch cluster from curl or HTTP client, or to use a management tool such as bigdesk.

To enable authorized access to elasticsearch, you may want to include the elasticsearch::proxy_nginx recipe, which will install, configure and run Nginx as a reverse proxy, allowing users with proper credentials to connect.

As with AWS, you may store the usernames and passwords in the node configuration, but also in a data bag item:

    mkdir -p ./data_bags/elasticsearch
    echo '{
      "id" : "users",
      "users" : [
        {"username" : "USERNAME", "password" : "PASSWORD"},
        {"username" : "USERNAME", "password" : "PASSWORD"}
      ]
    }
    ' >> ./data_bags/elasticsearch/users.json

Again, do not forget to upload the data bag to the Chef server.

After you have configured the node and uploaded all the information to the Chef server, run chef-client on the node(s):

    knife ssh name:elasticsearch* 'sudo su - root -c "chef-client"'

Testing with Vagrant

The cookbook comes with a Vagrantfile, allowing you to test-drive the installation and configuration with Vagrant, the tool for building virtualized development infrastructure.

First, make sure, you have both VirtualBox and Vagrant installed.

Then, clone this repository into elasticsearch, somewhere on your development machine:

    git clone git://github.com/karmi/cookbook-elasticsearch.git elasticsearch

Switch to the cloned repository:

   cd elasticsearch

Download the required cookbooks (unless you already have them in ~/cookbooks):

    curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1184/original/apt.tgz   | tar xz -C tmp/cookbooks
    curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1421/original/java.tgz  | tar xz -C tmp/cookbooks
    curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1098/original/vim.tgz   | tar xz -C tmp/cookbooks
    curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1413/original/nginx.tgz | tar xz -C tmp/cookbooks
    curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/915/original/monit.tgz  | tar xz -C tmp/cookbooks

We will use the Ubuntu Lucid 64 box, but you may want to test-drive this cookbook on a different OS, of course. Check out the available boxes at http://vagrantbox.es.

Now, launch the virtual machine with Vagrant (it will download the box unless you already have it):

    vagrant up

The machine will be started and automatically provisioned with chef-solo. You'll see Chef debug messages flying by in your terminal, installing and configuring Java, Nginx, elasticsearch, etc. The process should take less then 15 minutes.

After the process is done, you may connect to elasticsearch via the Nginx proxy:

    open 'http://USERNAME:PASSWORD@33.33.33.10:8080/test_chef_cookbook/_search?q=*'

Of course, you should connect to the box with SSH and check things out:

    vagrant ssh
    ps aux | grep elasticsearch
    service elasticsearch status --verbose
    curl http://localhost:9200/_cluster/health?pretty

Cookbook Organization

• attributes/default.rb: version, paths, memory and naming settings for the node
• attributes/plugin_aws.rb: Amazon Web Services settings
• attributes/proxy_nginx.rb: Nginx settings
• templates/default/elasticsearch.init.erb: service init script
• templates/default/elasticsearch.yml.erb: main elasticsearch configuration file
• templates/default/elasticsearch-env.sh.erb: environment variables needed by the Java Virtual Machine and elasticsearch
• templates/default/elasticsearch_proxy_nginx.conf.erb: the reverse proxy configuration
• templates/default/elasticsearch.conf.erb: Monit configuration file

License

Author: Karel Minarik (karmi@karmi.cz)

MIT LICENSE