Merge branch 'Bg-Tek-update-5613' into v5

Dockerfile

@@ -1,5 +1,5 @@
 # Dockerfile for ELK stack
-# Elasticsearch, Logstash, Kibana 5.6.10
+# Elasticsearch, Logstash, Kibana 5.6.13
 
 # Build with:
 # docker build -t <repo-user>/elk .
@@ -39,7 +39,7 @@ RUN set -x \
  && set +x
 
 
-ENV ELK_VERSION 5.6.10
+ENV ELK_VERSION 5.6.13
 
 ### install Elasticsearch
 

README-short.txt

@@ -1 +1 @@
-Collect, search and visualise log data with ELK (Elasticsearch 5.6.10, Logstash 5.6.10, Kibana 5.6.10).
+Collect, search and visualise log data with ELK (Elasticsearch 5.6.13, Logstash 5.6.13, Kibana 5.6.13).

README.md

@@ -14,7 +14,17 @@ This image is hosted on Docker Hub at [https://hub.docker.com/r/sebp/elk/](https
 
 The following tags are available:
 
-- `latest`, `630`: ELK 6.3.0.
+- `latest`, `642`: ELK 6.4.2.
+
+- `641`: ELK 6.4.1.
+
+- `640`: ELK 6.4.0.
+
+- `632`: ELK 6.3.2.
+
+- `631`: ELK 6.3.1.
+
+- `630`: ELK 6.3.0.
 
 - `624`: ELK 6.2.4.
 
@@ -38,6 +48,8 @@ The following tags are available:
 
 - `600`: ELK 6.0.0.
 
+- `5613`: ELK 5.6.13.
+
 - `5610`: ELK 5.6.10.
 
 - `569`: ELK 5.6.9.

docs/index.md

@@ -37,6 +37,7 @@ This web page documents how to use the [sebp/elk](https://hub.docker.com/r/sebp/
 	- [Elasticsearch is not starting (2): `cat: /var/log/elasticsearch/elasticsearch.log: No such file or directory`](#es-not-starting-not-enough-memory)
 	- [Elasticsearch is not starting (3): bootstrap tests](#es-not-starting-bootstrap-tests)
 	- [Elasticsearch is suddenly stopping after having started properly](#es-suddenly-stopping)
+	- [Miscellaneous](#issues-misc)
 - [Known issues](#known-issues)
 - [Troubleshooting](#troubleshooting)
 	- [If Elasticsearch isn't starting...](#es-not-starting)
@@ -55,6 +56,9 @@ To run a container using this image, you will need the following:
 
 	Install [Docker](https://docker.com/), either using a native package (Linux) or wrapped in a virtual machine (Windows, OS X – e.g. using [Boot2Docker](http://boot2docker.io/) or [Vagrant](https://www.vagrantup.com/)).
 
+	**Note** – As the *sebp/elk* image is based on a Linux image, users of Docker for Windows will need to ensure that [Docker is using Linux containers](https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers).
+	 
+
 - **A minimum of 4GB RAM assigned to Docker**
 
 	Elasticsearch alone needs at least 2GB of RAM to run.
@@ -409,9 +413,15 @@ The next few subsections present some typical use cases.
 
 ### Updating Logstash's configuration <a name="updating-logstash-configuration"></a>
 
-The image contains several configuration files for Logstash (e.g. `01-lumberjack-input.conf`, `02-beats-input.conf`), all located in `/etc/logstash/conf.d`.
+Generally speaking, the directory layout for Logstash is the one described [here](https://www.elastic.co/guide/en/logstash/current/dir-layout.html#zip-targz-layout). 
+
+Logstash's settings are defined by the configuration files (e.g. `logstash.yml`, `jvm.options`, `pipelines.yml`) located in `/opt/logstash/config`.
+
+Out of the box the image's `pipelines.yml` configuration file defines a default pipeline, made of the files (e.g. `01-lumberjack-input.conf`, `02-beats-input.conf`) located in `/etc/logstash/conf.d`.
 
-To modify an existing configuration file, you can bind-mount a local configuration file to a configuration file within the container at runtime. For instance, if you want to replace the image's `30-output.conf` Logstash configuration file with your local file `/path/to/your-30-output.conf`, then you would add the following `-v` option to your `docker` command line:
+**Note** – Somewhat confusingly, the term "configuration file" may be used to refer to the files defining  Logstash's settings or those defining its pipelines (which are probably the ones you want to tweak the most).
+
+To modify an existing configuration file (be it a high-level Logstash configuration file, or a pipeline configuration file), you can bind-mount a local configuration file to a configuration file within the container at runtime. For instance, if you want to replace the image's `30-output.conf` configuration file with your local file `/path/to/your-30-output.conf`, then you would add the following `-v` option to your `docker` command line:
 
 	$ sudo docker run ... \
 		-v /path/to/your-30-output.conf:/etc/logstash/conf.d/30-output.conf \
@@ -442,8 +452,8 @@ A `Dockerfile` like the following will extend the base image and install the [Ge
 	ENV ES_HOME /opt/elasticsearch
 	WORKDIR ${ES_HOME}
 
-	RUN CONF_DIR=/etc/elasticsearch gosu elasticsearch bin/elasticsearch-plugin \
-		install ingest-geoip
+	RUN yes | CONF_DIR=/etc/elasticsearch gosu elasticsearch bin/elasticsearch-plugin \
+		install -b ingest-geoip
 
 You can now build the new image (see the *[Building the image](#building-image)* section above) and run the container in the same way as you did with the base image.
 
@@ -468,14 +478,7 @@ The name of Kibana's home directory in the image is stored in the `KIBANA_HOME`
 
 Kibana runs as the user `kibana`. To avoid issues with permissions, it is therefore recommended to install Kibana plugins as `kibana`, using the `gosu` command (see below for an example, and references for further details).  
 
-The following `Dockerfile` can be used to extend the base image and install the latest version of the [Sense plugin](https://www.elastic.co/guide/en/sense/current/index.html), a handy console for interacting with the REST API of Elasticsearch:
-
-	FROM sebp/elk
-
-	WORKDIR ${KIBANA_HOME}
-	RUN gosu kibana bin/kibana-plugin install elastic/sense
-
-See the *[Building the image](#building-image)* section above for instructions on building the new image. You can then run a container based on this image using the same command line as the one in the *[Usage](#usage)* section. The Sense interface will be accessible at `http://<your-host>:5601/apss/sense` (e.g. [http://localhost:5601/app/sense](http://localhost:5601/app/sense) for a local native instance of Docker).
+A `Dockerfile` similar to the ones in the sections on Elasticsearch and Logstash plugins can be used to extend the base image and install a Kibana plugin.
 
 ## Persisting log data <a name="persisting-log-data"></a>
 
@@ -703,6 +706,15 @@ With the default image, this is usually due to Elasticsearch running out of memo
 
 As a reminder (see [Prerequisites](#prerequisites)), you should use no less than 3GB of memory to run the container... and possibly much more.
 
+### Miscellaneous <a name="issues-misc"></a>
+
+Other known issues include:
+
+- Elasticsearch not having enough time to start up with the default image settings: in that case [set the `ES_CONNECT_RETRY` environment variable](#overriding-variables) to a value larger than 30. (By default Elasticsearch has 30 seconds to start before other services are started, which may not be enough and cause the container to stop.)
+
+- Incorrect proxy settings, e.g. if a proxy is defined for Docker, ensure that connections to `localhost` are not proxied (e.g. by using a `no_proxy` setting).
+
+
 ## Known issues <a name="known-issues"></a>
 
 When using Filebeat, an [index template file](https://www.elastic.co/guide/en/beats/filebeat/6.0/filebeat-template.html) is used to connect to Elasticsearch to define settings and mappings that determine how fields should be analysed.

nginx-filebeat/Dockerfile

@@ -1,5 +1,5 @@
 # Dockerfile to illustrate how Filebeat can be used with nginx
-# Filebeat 5.6.10
+# Filebeat 5.6.13
 
 # Build with:
 # docker build -t filebeat-nginx-example .
@@ -19,7 +19,7 @@ ENV REFRESHED_AT 2016-11-04
 
 ### install Filebeat
 
-ENV FILEBEAT_VERSION 5.6.10
+ENV FILEBEAT_VERSION 5.6.13
 
 RUN apt-get update -qq \
  && apt-get install -qqy curl \
@@ -50,6 +50,9 @@ RUN chmod 644 /etc/filebeat/filebeat.yml
 RUN mkdir -p /etc/pki/tls/certs
 ADD logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt
 
+# create template based on filebeat version (assumption: it is the same version as elasticsearch version)
+RUN filebeat export template --es.version ${FILEBEAT_VERSION} > /etc/filebeat/filebeat.template.json
+
 ###############################################################################
 #                                    DATA
 ###############################################################################

nginx-filebeat/start.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-curl -XPUT 'http://elk:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json
+curl -XPUT -H "Content-Type: application/json" 'http://elk:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json
 /etc/init.d/filebeat start
 nginx
 tail -f /var/log/nginx/access.log -f /var/log/nginx/error.log

spec/Dockerfile_spec.rb

@@ -1,7 +1,7 @@
 require "docker"
 require "serverspec"
 
-ELK_VERSION = "5.6.10"
+ELK_VERSION = "5.6.13"
 ELASTICSEARCH_VERSION = ELK_VERSION
 LOGSTASH_VERSION = ELK_VERSION
 KIBANA_VERSION = ELK_VERSION