update ELK to version OSS 8.0.0

Dockerfile

@@ -1,5 +1,5 @@
 # Dockerfile for ELK stack
-# Elasticsearch, Logstash, Kibana 7.17.1
+# Elasticsearch, Logstash, Kibana OSS 8.0.0
 
 # Build with:
 # docker build -t <repo-user>/elk .
@@ -33,10 +33,10 @@ RUN set -x \
 
 ### set current package version
 
-ARG ELK_VERSION=7.17.1
+ARG ELK_VERSION=oss-8.0.0
 
 # base version (i.e. remove OSS prefix) for Elasticsearch and Kibana (no OSS version since 7.11.0)
-ARG ELK_BASE_VERSION=7.17.1
+ARG ELK_BASE_VERSION=8.0.0
 
 # replace with aarch64 for ARM64 systems
 ARG ARCH=x86_64 

README-short.txt

@@ -1 +1 @@
-Collect, search and visualise log data with ELK (Elasticsearch 7.17.1, Logstash 7.17.1, Kibana 7.17.1).
+Collect, search and visualise log data with ELK (Elasticsearch OSS 8.0.0, Logstash OSS 8.0.0, Kibana OSS 8.0.0).

README.md

@@ -14,9 +14,9 @@ This image is hosted on Docker Hub at [https://hub.docker.com/r/sebp/elk/](https
 
 The following tags are available:
 
-- `latest`, `7.17.1`: ELK 7.17.1.
+- `latest`, `oss-8.0.0`: ELK OSS 8.0.0.
 
-- `oss-7.17.1` (ELK OSS 7.17.1), `7.17.0` (7.17.0), `oss-7.17.0` (OSS 7.17.0), `7.16.3` (7.16.3), `oss-7.16.3` (OSS 7.16.3), `7.16.2` (7.16.2), `oss-7.16.2` (OSS 7.16.2), `7.16.1` (7.16.1), `oss-7.16.1` (OSS 7.16.1), `7.16.0` (7.16.0), `oss-7.16.0` (OSS 7.16.0), `7.15.2` (7.15.2), `oss-7.15.2` (OSS 7.15.2), `7.15.1` (7.15.1), `oss-7.15.1` (OSS 7.15.1), `7.15.0` (7.15.0), `oss-7.15.0` (OSS 7.15.0), `7.14.2` (7.14.2), `oss-7.14.2` (OSS 7.14.2), `7.14.1` (7.14.1), `oss-7.14.1` (OSS 7.14.1), `7.14.0` (7.14.0), `oss-7.14.0` (OSS 7.14.0), `7.13.4` (7.13.4), `oss-7.13.4` (OSS 7.13.4), `7.13.3` (7.13.3), `oss-7.13.3` (OSS 7.13.3), `7.13.2` (7.13.2), `oss-7.13.2` (OSS 7.13.2), `7.13.1` (7.13.1), `oss-7.13.1` (OSS 7.13.1), `7.13.0` (7.13.0), `oss-7.13.0` (OSS 7.13.0), `7.12.1` (7.12.1), `oss-7.12.1` (OSS 7.12.1), `7.12.0` (7.12.0), `oss-7.12.0` (OSS 7.12.0), `7.11.2` (7.11.2), `oss-7.11.2` (OSS 7.11.2), `7.11.1` (7.11.1), `oss-7.11.1` (OSS 7.11.1), `oss-7.11.0` (OSS 7.11.0), `7.11.0` (7.11.0), `oss-7.10.2` (OSS 7.10.2), `7.10.2` (7.10.2), `oss-7.10.1` (OSS 7.10.1), `7.10.1` (7.10.1), `oss-7.10.0` (OSS 7.10.0), `7.10.0` (7.10.0), `oss-793` (OSS 7.9.3), `793` (7.9.3), `oss-792` (OSS 7.9.2), `792` (7.9.2), `oss-791` (OSS 7.9.1), `791` (7.9.1), `oss-790` (OSS 7.9.0), `790` (7.9.0), `oss-781` (OSS 7.8.1), `781` (7.8.1), `oss-780` (OSS 7.8.0), `780` (7.8.0), `771` (7.7.1), `770` (7.7.0), `762` (7.6.2), `761` (7.6.1), `760` (7.6.0), `752` (7.5.2), `751` (7.5.1), `750` (7.5.0), `742` (7.4.2), `741` (7.4.1), `740` (7.4.0), `732` (7.3.2), `731` (7.3.1), `730` (7.3.0), `721` (7.2.1), `720` (7.2.0), `711` (7.1.1), `710` (7.1.0), `701` (7.0.1), `700` (7.0.0).
+- `7.17.1` (ELK 7.17.1), `oss-7.17.1` (OSS 7.17.1), `7.17.0` (7.17.0), `oss-7.17.0` (OSS 7.17.0), `7.16.3` (7.16.3), `oss-7.16.3` (OSS 7.16.3), `7.16.2` (7.16.2), `oss-7.16.2` (OSS 7.16.2), `7.16.1` (7.16.1), `oss-7.16.1` (OSS 7.16.1), `7.16.0` (7.16.0), `oss-7.16.0` (OSS 7.16.0), `7.15.2` (7.15.2), `oss-7.15.2` (OSS 7.15.2), `7.15.1` (7.15.1), `oss-7.15.1` (OSS 7.15.1), `7.15.0` (7.15.0), `oss-7.15.0` (OSS 7.15.0), `7.14.2` (7.14.2), `oss-7.14.2` (OSS 7.14.2), `7.14.1` (7.14.1), `oss-7.14.1` (OSS 7.14.1), `7.14.0` (7.14.0), `oss-7.14.0` (OSS 7.14.0), `7.13.4` (7.13.4), `oss-7.13.4` (OSS 7.13.4), `7.13.3` (7.13.3), `oss-7.13.3` (OSS 7.13.3), `7.13.2` (7.13.2), `oss-7.13.2` (OSS 7.13.2), `7.13.1` (7.13.1), `oss-7.13.1` (OSS 7.13.1), `7.13.0` (7.13.0), `oss-7.13.0` (OSS 7.13.0), `7.12.1` (7.12.1), `oss-7.12.1` (OSS 7.12.1), `7.12.0` (7.12.0), `oss-7.12.0` (OSS 7.12.0), `7.11.2` (7.11.2), `oss-7.11.2` (OSS 7.11.2), `7.11.1` (7.11.1), `oss-7.11.1` (OSS 7.11.1), `oss-7.11.0` (OSS 7.11.0), `7.11.0` (7.11.0), `oss-7.10.2` (OSS 7.10.2), `7.10.2` (7.10.2), `oss-7.10.1` (OSS 7.10.1), `7.10.1` (7.10.1), `oss-7.10.0` (OSS 7.10.0), `7.10.0` (7.10.0), `oss-793` (OSS 7.9.3), `793` (7.9.3), `oss-792` (OSS 7.9.2), `792` (7.9.2), `oss-791` (OSS 7.9.1), `791` (7.9.1), `oss-790` (OSS 7.9.0), `790` (7.9.0), `oss-781` (OSS 7.8.1), `781` (7.8.1), `oss-780` (OSS 7.8.0), `780` (7.8.0), `771` (7.7.1), `770` (7.7.0), `762` (7.6.2), `761` (7.6.1), `760` (7.6.0), `752` (7.5.2), `751` (7.5.1), `750` (7.5.0), `742` (7.4.2), `741` (7.4.1), `740` (7.4.0), `732` (7.3.2), `731` (7.3.1), `730` (7.3.0), `721` (7.2.1), `720` (7.2.0), `711` (7.1.1), `710` (7.1.0), `701` (7.0.1), `700` (7.0.0).
 
 - `6.8.22` (ELK 6.8.22), `683` (6.8.3), `681` (ELK 6.8.2), `681` (ELK 6.8.1), `680` (ELK 6.8.0), `672` (ELK 6.7.2), `671` (ELK 6.7.1), `670` (6.7.0), `662` (6.6.2), `661` (6.6.1), `660` (6.6.0), `651` (6.5.1), `650` (6.5.0), `643` (6.4.3), `642` (6.4.2), `641` (6.4.1), `640` (6.4.0), `632` (6.3.2), `631` (6.3.1), `630` (6.3.0), `624` (6.2.4), `623` (6.2.3), `622` (6.2.2), `621` (6.2.1), `620` (6.2.0), `613` (6.1.3), `612` (6.1.2), `611` (6.1.1), `610` (6.1.0), `601` (6.0.1), `600` (6.0.0).
 

docs/index.md

@@ -47,6 +47,7 @@ This web page documents how to use the [sebp/elk](https://hub.docker.com/r/sebp/
 	- [Additional tips](#general-troubleshooting)
 - [Reporting issues](#reporting-issues)
 - [Breaking changes](#breaking-changes)
+- [Release notes](#release-notes)
 - [References](#references)
 - [About](#about)
 
@@ -647,17 +648,15 @@ An even more optimal way to distribute Elasticsearch, Logstash and Kibana across
 
 As it stands this image is meant for local test use, and as such hasn't been secured: access to the ELK services is unrestricted, and default authentication server certificates and private keys for the Logstash input plugins are bundled with the image.
 
-To harden this image, at the very least you would want to:
+**Note** – In fact, since version 8 of the image, security has been explicitly disabled, see the [Release notes](#release-notes) section.
 
-- Restrict the access to the ELK services to authorised hosts/networks only, as described in e.g. [Elasticsearch Scripting and Security](https://www.elastic.co/blog/scripting-security/) and [Elastic Security: Deploying Logstash, ElasticSearch, Kibana "securely" on the Internet](http://blog.eslimasec.com/2014/05/elastic-security-deploying-logstash.html).
-- Password-protect the access to Kibana and Elasticsearch (see [SSL And Password Protection for Kibana](http://technosophos.com/2014/03/19/ssl-password-protection-for-kibana.html)).
-- Generate a new self-signed authentication certificate for the Logstash input plugins (see [Notes on certificates](#certificates)) or (better) get a proper certificate from a commercial provider (known as a certificate authority), and keep the private key private.
+To harden this image, at the very least you would want to:
 
-X-Pack, which is now bundled with the other ELK services, may be a useful to implement enterprise-grade security to the ELK stack.
+- Configure the services to run with security enabled, see [Start the Elastic Stack with security enabled](https://www.elastic.co/guide/en/elasticsearch/reference/8.0/configuring-stack-security.html) (Elasticsearch),  [Secure your connection to Elasticsearch](https://www.elastic.co/guide/en/logstash/8.0/ls-security.html) (Logstash), and [Configure security in Kibana](https://www.elastic.co/guide/en/kibana/8.0/using-kibana-with-security.html) (Kibana) for version 8 of the ELK services.
 
-Alternatively, to implement authentication in a simple way, a reverse proxy (e.g. as provided by [nginx](https://www.nginx.com/) or [Caddy](https://caddyserver.com/)) could be used in front of the ELK services. 
+- Generate a new self-signed authentication certificate for the Logstash input plugins (see [Notes on certificates](#certificates)) or (better) get a proper certificate from a commercial provider (known as a certificate authority), and keep the private key private.
 
-If on the other hand you want to disable certificate-based server authentication (e.g. in a demo environment), see [Disabling SSL/TLS](#disabling-ssl-tls).
+  If on the other hand you want to disable certificate-based server authentication (e.g. in a demo environment), see [Disabling SSL/TLS](#disabling-ssl-tls).
 
 ### Notes on certificates <a name="certificates"></a>
 
@@ -935,6 +934,20 @@ Here is the list of breaking changes that may have side effects when upgrading t
 
 	- From `es234_l234_k452` to `es241_l240_k461`: add `--auto-reload` to `LS_OPTS`. 
 
+## Release notes <a name="release-notes"></a>
+
+The following information may be helpful when upgrading to later versions of the ELK image:
+
+- **Version 8.0**
+
+  Elasticsearch security is on by default since version 8.0 of the Elastic stack.
+
+  In the opinion of the Docker image’s author, setting up the Elastic stack with security enabled is somewhat fiddly. Security has therefore been disabled in this image to get everything up and running as smoothly as possible out of the box.
+
+  See the [Security considerations](#security-considerations) section for information on setting up security.
+
+
+
 ## References <a name="references"></a>
 
 - [How To Install Elasticsearch, Logstash, and Kibana 4 on Ubuntu 14.04](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04)

elasticsearch.yml

@@ -91,3 +91,9 @@ cluster.initial_master_nodes: ["elk"]
 # Require explicit names when deleting indices:
 #
 #action.destructive_requires_name: true
+#
+# ---------------------------------- Security ----------------------------------
+#
+# Enable/disable security (enabled by default since version 8.0)
+#
+xpack.security.enabled: false

kibana.yml

@@ -1,5 +1,7 @@
-# Default Kibana 5 file from https://github.com/elastic/kibana/blob/master/config/kibana.yml
-# 
+# For more configuration options see the configuration guide for Kibana in
+# https://www.elastic.co/guide/index.html
+
+# =================== System: Kibana Server ===================
 # Kibana is served by a back end server. This setting specifies the port to use.
 #server.port: 5601
 
@@ -8,55 +10,49 @@
 # To allow connections from remote users, set this parameter to a non-loopback address.
 server.host: "0.0.0.0"
 
-# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
-# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
-# to Kibana. This setting cannot end in a slash.
+# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
+# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
+# from requests it receives, and to prevent a deprecation warning at startup.
+# This setting cannot end in a slash.
 #server.basePath: ""
 
-# The maximum payload size in bytes for incoming server requests.
-#server.maxPayloadBytes: 1048576
+# Specifies whether Kibana should rewrite requests that are prefixed with
+# `server.basePath` or require that they are rewritten by your reverse proxy.
+# Defaults to `false`.
+#server.rewriteBasePath: false
 
-# The Kibana server's name.  This is used for display purposes.
-#server.name: "your-hostname"
+# Specifies the public URL at which Kibana is available for end users. If
+# `server.basePath` is configured this URL should end with the same basePath.
+#server.publicBaseUrl: ""
 
-# The URL of the Elasticsearch instance to use for all your queries.
-#elasticsearch.url: "http://localhost:9200"
+# The maximum payload size in bytes for incoming server requests.
+#server.maxPayload: 1048576
 
-# When this setting’s value is true Kibana uses the hostname specified in the server.host
-# setting. When the value of this setting is false, Kibana uses the hostname of the host
-# that connects to this Kibana instance.
-#elasticsearch.preserveHost: true
+# The Kibana server's name. This is used for display purposes.
+#server.name: "your-hostname"
 
-# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
-# dashboards. Kibana creates a new index if the index doesn’t already exist.
-#kibana.index: ".kibana"
+# =================== System: Kibana Server (Optional) ===================
+# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
+# These settings enable SSL for outgoing requests from the Kibana server to the browser.
+#server.ssl.enabled: false
+#server.ssl.certificate: /path/to/your/server.crt
+#server.ssl.key: /path/to/your/server.key
 
-# The default application to load.
-#kibana.defaultAppId: "discover"
+# =================== System: Elasticsearch ===================
+# The URLs of the Elasticsearch instances to use for all your queries.
+#elasticsearch.hosts: ["http://localhost:9200"]
 
 # If your Elasticsearch is protected with basic authentication, these settings provide
 # the username and password that the Kibana server uses to perform maintenance on the Kibana
 # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
 # is proxied through the Kibana server.
-#elasticsearch.username: "user"
+#elasticsearch.username: "kibana_system"
 #elasticsearch.password: "pass"
 
-# Paths to the PEM-format SSL certificate and SSL key files, respectively. These
-# files enable SSL for outgoing requests from the Kibana server to the browser.
-#server.ssl.cert: /path/to/your/server.crt
-#server.ssl.key: /path/to/your/server.key
-
-# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
-# These files validate that your Elasticsearch backend uses the same key files.
-#elasticsearch.ssl.cert: /path/to/your/client.crt
-#elasticsearch.ssl.key: /path/to/your/client.key
-
-# Optional setting that enables you to specify a path to the PEM file for the certificate
-# authority for your Elasticsearch instance.
-#elasticsearch.ssl.ca: /path/to/your/CA.pem
-
-# To disregard the validity of SSL certificates, change this setting’s value to false.
-#elasticsearch.ssl.verify: true
+# Kibana can also authenticate to Elasticsearch via "service account tokens".
+# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
+# Use this token instead of a username/password.
+# elasticsearch.serviceAccountToken: "my_token"
 
 # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
 # the elasticsearch.requestTimeout setting.
@@ -75,27 +71,88 @@ server.host: "0.0.0.0"
 #elasticsearch.customHeaders: {}
 
 # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
-#elasticsearch.shardTimeout: 0
+#elasticsearch.shardTimeout: 30000
 
-# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
-#elasticsearch.startupTimeout: 5000
+# =================== System: Elasticsearch (Optional) ===================
+# These files are used to verify the identity of Kibana to Elasticsearch and are required when
+# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
+#elasticsearch.ssl.certificate: /path/to/your/client.crt
+#elasticsearch.ssl.key: /path/to/your/client.key
 
-# Specifies the path where Kibana creates the process ID file.
-#pid.file: /var/run/kibana.pid
+# Enables you to specify a path to the PEM file for the certificate
+# authority for your Elasticsearch instance.
+#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
+
+# To disregard the validity of SSL certificates, change this setting's value to 'none'.
+#elasticsearch.ssl.verificationMode: full
+
+# =================== System: Logging ===================
+# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'error'
+#logging.root.level: debug
+
+# Enables you to specify a file where Kibana stores log output.
+#logging.appenders.default:
+#  type: file
+#  fileName: /var/logs/kibana.log
+#  layout:
+#    type: json
 
-# Enables you specify a file where Kibana stores log output.
-#logging.dest: stdout
+# Logs queries sent to Elasticsearch.
+#logging.loggers:
+#  - name: elasticsearch.query
+#    level: debug
 
-# Set the value of this setting to true to suppress all logging output.
-#logging.silent: false
+# Logs http responses.
+#logging.loggers:
+#  - name: http.server.response
+#    level: debug
 
-# Set the value of this setting to true to suppress all logging output other than error messages.
-#logging.quiet: false
+# Logs system usage information.
+#logging.loggers:
+#  - name: metrics.ops
+#    level: debug
 
-# Set the value of this setting to true to log all events, including system usage information
-# and all requests.
-#logging.verbose: false
+# =================== System: Other ===================
+# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
+#path.data: data
+
+# Specifies the path where Kibana creates the process ID file.
+#pid.file: /run/kibana/kibana.pid
 
 # Set the interval in milliseconds to sample system and process performance
 # metrics. Minimum is 100ms. Defaults to 5000.
 #ops.interval: 5000
+
+# Specifies locale to be used for all localizable strings, dates and number formats.
+# Supported languages are the following: English - en , by default , Chinese - zh-CN .
+#i18n.locale: "en"
+
+# =================== Frequently used (Optional)===================
+
+# =================== Saved Objects: Migrations ===================
+# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.
+
+# The number of documents migrated at a time.
+# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
+# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000
+# migrations.batchSize: 1000
+
+# The maximum payload size for indexing batches of upgraded saved objects.
+# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
+# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
+# configuration option. Default: 100mb
+# migrations.maxBatchSizeBytes: 100mb
+
+# The number of times to retry temporary migration failures. Increase the setting
+# if migrations fail frequently with a message such as `Unable to complete the [...] step after
+# 15 attempts, terminating`. Defaults to 15
+# migrations.retryAttempts: 15
+
+# =================== Search Autocomplete ===================
+# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
+# This value must be a whole number greater than zero. Defaults to 1000
+# data.autocomplete.valueSuggestions.timeout: 1000
+
+# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
+# This value must be a whole number greater than zero. Defaults to 100000
+# data.autocomplete.valueSuggestions.terminateAfter: 100000

nginx-filebeat/Dockerfile

@@ -1,5 +1,5 @@
 # Dockerfile to illustrate how Filebeat can be used with nginx
-# Filebeat 7.17.1
+# Filebeat OSS 8.0.0
 
 # Build with:
 # docker build -t filebeat-nginx-example .
@@ -19,8 +19,8 @@ ENV REFRESHED_AT 2020-10-02
 
 ### install Filebeat
 
-ENV FILEBEAT_VERSION 7.17.1
-ENV FILEBEAT_BASE_VERSION 7.17.1
+ENV FILEBEAT_VERSION oss-8.0.0
+ENV FILEBEAT_BASE_VERSION 8.0.0
 
 
 RUN apt-get update -qq \