Setup Elasticsearch on Google Cloud

Install

    0  sudo su
    1  sudo apt-get update
    2  sudo apt-get install python-software-properties
    3  sudo add-apt-repository ppa:webupd8team/java
    4  sudo apt-get update
    5  sudo apt-get install tree
    6  sudo apt-get install oracle-java7-installer
    7  java -version
    8  wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.2.deb
    8  sudo dpkg -i elasticsearch-1.7.2.deb

    9  vi /etc/elasticsearch/elasticsearch.yml
       # You will want to restrict outside access to your Elasticsearch instance (port 9200),
       # so outsiders can't read your data or shutdown your Elasticsearch cluster through the HTTP API.
       # Find the line that specifies network.host, uncomment it, and replace its value with "localhost"
       # so it looks like this:
       network.host: localhost

   10  sudo service elasticsearch restart
   11  curl -X GET 'http://localhost:9200'
       {
         "status" : 200,
         "name" : "Myron MacLain",
         "version" : {
           "number" : "1.1.2",
           "build_hash" : "e511f7b28b77c4d99175905fac65bffbf4c80cf7",
           "build_timestamp" : "2014-05-22T12:27:39Z",
           "build_snapshot" : false,
           "lucene_version" : "4.7"
         },
         "tagline" : "You Know, for Search"
       }
   12  find / -name elasticsearch
       /var/log/elasticsearch
       /var/lib/elasticsearch
       /usr/share/lintian/overrides/elasticsearch
       /usr/share/elasticsearch
       /usr/share/elasticsearch/data/elasticsearch
       /usr/share/elasticsearch/bin/elasticsearch
       /usr/share/doc/elasticsearch
       /etc/init.d/elasticsearch
       /etc/default/elasticsearch
       /etc/elasticsearch
       /tmp/elasticsearch

Add plugin

• Default URLs after plugins are installed:
  • myhost.com/_plugin/head/
  • myhost.com/_plugin/marvel/sense/index.html

   36  cd /usr/share/elasticsearch
   37  tree -L 1
       .
       ├── bin
       ├── core-signatures.txt
       ├── data
       ├── lib
       └── plugins
   38  service elasticsearch status
        * elasticsearch is running
   39  service elasticsearch stop
        * Stopping Elasticsearch Server [ OK ]
   40  service elasticsearch status
        * elasticsearch is not running
   41  ./bin/plugin -i mobz/elasticsearch-head
       -> Installing mobz/elasticsearch-head...
       Trying https://github.com/mobz/elasticsearch-head/archive/master.zip...
       Downloading
       ...DONE
       Installed mobz/elasticsearch-head into /usr/share/elasticsearch/plugins/head
       Identified as a _site plugin, moving to _site structure ...
   42  ./bin/plugin -i elasticsearch/marvel/latest
   43  service elasticsearch start
        * Starting Elasticsearch Server [ OK ]
   44  service elasticsearch status
        * elasticsearch is running

Remove Plugin: Marvel

$ cd /usr/share/elasticsearch
$ ./bin/plugin -remove elasticsearch/marvel/latest

Add/Remove Plugin: Shield

$ cd /usr/share/elasticsearch
$ ./bin/plugin -i elasticsearch/license/latest
-> Installing elasticsearch/license/latest...
Trying http://download.elasticsearch.org/elasticsearch/license/license-latest.zip...
Downloading ...............DONE
Installed elasticsearch/license/latest into /usr/share/elasticsearch/plugins/license

$ ./bin/plugin -i elasticsearch/shield/latest
-> Installing elasticsearch/shield/latest...
Trying http://download.elasticsearch.org/elasticsearch/shield/shield-latest.zip...
Downloading ...............DONE
Installed elasticsearch/shield/latest into /usr/share/elasticsearch/plugins/shield

$ service elasticsearch restart
 * Stopping Elasticsearch Server                                                                            [ OK ] 
 * Starting Elasticsearch Server                                                                            [ OK ] 

$ less /var/log/elasticsearch/es_shoppinpal.log
[Spot] version[1.1.2], pid[18013], build[e511f7b/2014-05-22T12:27:39Z]
[INFO ][node] [Spot] initializing ...
[WARN ][plugins] [Spot] failed to load plugin from [jar:file:/usr/share/elasticsearch/plugins/shield/elasticsearch-shield-1.1.1.jar!/es-plugin.properties]
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.shield.ShieldPlugin]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:512)
    ...
    Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/ElasticsearchException$WithRestHeaders
    ... 7 more

###
## Posted on forums:
## https://discuss.elastic.co/t/which-version-of-elasticsearch-does-shield-work-with/27677
###

$  ./bin/plugin -remove elasticsearch/license/latest
$  ./bin/plugin -remove elasticsearch/shield/latest
-> Removing elasticsearch/shield/latest 
Removed elasticsearch/shield/latest

$ service elasticsearch restart
 * Stopping Elasticsearch Server                                                                            [ OK ] 
 * Starting Elasticsearch Server

Administrative Tasks

   ## Elasticsearch on 64 bit Linux now uses mmapfs by default. 
   ## Make sure that you set MAX_MAP_COUNT to a sufficiently high number.
   ## The RPM and Debian packages default this value to 262144.
   11  sysctl vm.max_map_count
   12  curl http://localhost:9200/_nodes/process?pretty
       {
         "cluster_name" : "...",
         "nodes" : {
           "hHZH7A4jS2qxBiMByJBSaA" : {
             "name" : "Myron MacLain",
             "transport_address" : "inet[/xx.xxx.xx.xx:xxxx]",
             "host" : "elasticsearch-1-1.c.xxx-xxx-xxx.internal",
             "ip" : "xx.xxx.xx.xx",
             "version" : "1.1.2",
             "build" : "e511f7b",
             "http_address" : "inet[/xx.xxx.xx.xx:xxxx]",
             "process" : {
               "refresh_interval" : 1000,
               "id" : 3866,
               "max_file_descriptors" : 65535,
               "mlockall" : false
             }
           }
         }
       }

Secure with nginx

1. https://github.com/ShoppinPal/gcp-scripts/wiki/Setup%20Nginx%20on%20Google%20Cloud
2. https://www.digitalocean.com/community/tutorials/how-to-set-up-http-authentication-with-nginx-on-ubuntu-12-10

Add Kibana

Reference: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04

1. Install Kibana

cd ~/
wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

vi ./kibana-4.1.2-linux-x64/config/kibana.yml
# In the Kibana configuration file, find the line that specifies host,
# and replace the IP address ("0.0.0.0" by default) with "localhost":
# host: "localhost"

mkdir -p /opt/kibana
cp -R kibana-4.1.2-linux-x64/* /opt/kibana/

cd /etc/init.d && sudo wget https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/bce61d85643c2dcdfbc2728c55a41dab444dca20/kibana4
chmod +x /etc/init.d/kibana4
update-rc.d kibana4 defaults 96 9
service kibana4 status
service kibana4 start
service kibana4 status

2. Tweak the nginx proxy so that all requests from /kibana are served by the kibana server and not the original elasticsearch server:

3. vi /etc/nginx/sites-available/default

4. Sample:

     location / {
       ...
       proxy_pass          http://localhost:9200;
       ...
     }
   
     # `/kibana/` is used instead of `/kibana` to make this work
     # so don't remove it willy-nilly
     location /kibana/ { 
       ...
       proxy_pass          http://localhost:5601/;
       proxy_redirect      http://localhost:5601 https://YOUR_SUBDOMAIN.YOUR_DOMAIN.com/kibana;
       ...
     }
   

5. test and reload

   /etc/init.d/nginx configtest /etc/nginx/sites-available/default
   service nginx restart
   

6. https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04#connect-to-kibana