diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/index.md b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/index.md new file mode 100644 index 00000000000..a76e19239ef --- /dev/null +++ b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/index.md @@ -0,0 +1,311 @@ +--- +author: + name: Tyler Langlois + email: ty@tjll.net +description: 'This tutorial will explain how to configure Filebeat and Metricbeat to monitor Docker container logs and metrics to be stored in Elasticsearch and visualized in Kibana.' +og_description: 'Monitor Docker containers using the Elastic Stack.' +keywords: ["elastic", "filebeat", "metricbeat", "elasticsearch", "kibana", "docker", "container", "nginx"] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +modified: 2019-02-10 +modified_by: + name: Linode +published: 2019-02-10 +title: Container Instrumentation with the Elastic Stack +external_resources: +- '[Filebeat Modules](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html)' +- '[Metricbeat Modules](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-modules.html)' +- '[Elastic Container Monitoring](https://www.elastic.co/docker-kubernetes-container-monitoring)' +- '[Docker Command-Line Reference](https://docs.docker.com/reference/)' +--- + +The [Elastic Stack](https://www.elastic.co/products) can monitor a variety of data generated by [Docker](https://www.docker.com/) containers. In this guide, you will set up a Linode to analyze and visualize container logs and metrics using tools like Kibana, Beats, and Elasticsearch. Once finished, you will be able to configure your system to collect data for additional containers automatically. + +## Before you Begin + +1. Familiarize yourself with Linode's [Getting Started](/docs/getting-started/) guide and complete the steps for deploying and setting up a Linode running a recent Linux distribution (such as Ubuntu 18.04 LTS or CentOS 7), including setting the hostname and timezone. + +1. This guide uses `sudo` wherever possible. Complete the sections of our [Securing Your Server](/docs/security/securing-your-server/) guide to create a standard user account, harden SSH access, and remove unnecessary network services. + +1. Follow our [UFW Guide](/docs/security/firewalls/configure-firewall-with-ufw/) in order to install and configure a firewall (UFW) on your Ubuntu or Debian-based system, or our [FirewallD Guide](/docs/security/firewalls/introduction-to-firewalld-on-centos/) for rpm or CentOS-based systems. After configuring the firewall, ensure that the necessary ports are open in order to proceed with connections over SSH for the rest of this guide: + + sudo ufw allow ssh + +1. Ensure your system is up to date. On Debian-based systems, use: + + sudo apt update && sudo apt upgrade + + For rpm-based systems such as CentOS, use: + + sudo yum update + +1. Install Docker on your Linode by following [the installation guide from the Docker project](https://docs.docker.com/). + +{{< note >}} +The services in this guide bind to localhost only, which means they are not accessible outside of the Linode from remote hosts. This ensures that Elasticsearch's REST API remains private to localhost and is not remotely accessible from the internet. If you take steps beyond this guide to configure Elasticsearch and related components further, ensure that your firewall is in place and correctly blocking traffic to the Elasticsearch and Kibana nodes from the internet (ports 9200 and 9300 for Elasticsearch and 5601 for Kibana) to keep them properly secured. +{{< /note >}} + +## Install Elastic Stack Components + +Before configuring your system to monitor running containers, first install the components necessary to collect and ship logs and metrics to Elasticsearch. + +### Debian-Based Distributions + +Configure the Elastic `apt` repository and install the necessary packages and their dependencies. + +1. Install the official Elastic APT package signing key: + + wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - + +2. Install the `apt-transport-https` package, which is required to retrieve `deb` packages served over HTTPS: + + sudo apt-get install apt-transport-https + +3. Add the APT repository information to your server's list of sources: + + echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list + +4. Refresh the list of available packages: + + sudo apt-get update + +5. Before installing Elasticsearch, the Java runtime must be present. On systems such as Ubuntu 18.04 LTS, using the `default-jre-headless` package installs a compatible Java runtime: + + sudo apt-get install default-jre-headless + +6. Install Elasticsearch, Kibana, Filebeat, and Metricbeat: + + sudo apt-get install elasticsearch kibana filebeat metricbeat + +### Redhat-Based Distributions + +Configure the `rpm` repository for `yum` and related packaging tools. + +1. Trust the Elastic signing key: + + sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch + +1. Create a yum repository configuration to use the Elastic yum repository: + + {{< file "/etc/yum.repos.d/elasticsearch.repo" ini >}} +[elasticsearch-6.x] +name=Elastic repository for 6.x packages +baseurl=https://artifacts.elastic.co/packages/6.x/yum +gpgcheck=1 +gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch +enabled=1 +autorefresh=1 +type=rpm-md + {{< /file >}} + +1. Update the `yum` cache to ensure any new packages become available: + + sudo yum update + +1. Before installing Elasticsearch, the Java runtime must be present. On CentOS, for example, a compatible Java runtime can be installed using a headless OpenJDK package: + + sudo yum install java-11-openjdk-headless + +1. Install Elasticsearch, Kibana, Filebeat, and Metricbeat: + + sudo yum install elasticsearch kibana filebeat metricbeat + +## Configure The Elastic Stack + +In order to properly discover and capture container metrics, each component of the Elastic stack should be configured. + +### Elasticsearch + +In the file `/etc/elasticsearch/jvm.options` two values that begin with `-Xm` should be uncommented. These settings instruct the JVM to allocate a specific amount of memory. The recommend value for these settings is 50% of the available system RAM. For example, on a system with 1G of RAM, these settings should be: + +{{< file "/etc/elasticsearch/jvm.options" yml >}} +-Xms512m +-Xmx512m +{{< /file >}} + +1. Before starting Elasticsearch, install some necessary plugins to process geoip and user-agent data. + + sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent + sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip + +1. With these setting in place, start the `elasticsearch` service. + + sudo systemctl start elasticsearch + +1. Wait for a short period of time for Elasticsearch to start, then check that Elasticsearch is responding over the REST API: + + curl http://localhost:9200 + + You should see output similar to the following: + + { + "name" : "iQEk_-M", + "cluster_name" : "elasticsearch", + "cluster_uuid" : "tQeLgbKrTNOp2AoqdmTItw", + "version" : { + "number" : "6.5.4", + "build_flavor" : "default", + "build_type" : "deb", + "build_hash" : "d2ef93d", + "build_date" : "2018-12-17T21:17:40.758843Z", + "build_snapshot" : false, + "lucene_version" : "7.5.0", + "minimum_wire_compatibility_version" : "5.6.0", + "minimum_index_compatibility_version" : "5.0.0" + }, + "tagline" : "You Know, for Search" + } + + Elasticsearch is ready to index documents. + +### Kibana + +Most of Kibana's default settings are suitable for the purposes of this guide. No configuration changes are necessary; start the `kibana` service. + + sudo systemctl start kibana + +### Filebeat + +Use the `docker` input to enable Filebeat to capture started containers dynamically. This alleviates the need to specify Docker log file paths and instead permits Filebeat to discover containers when they start. + +1. Add the following near the top of the Filebeat configuration file to instruct the `filebeat` daemon to capture Docker container logs. These lines should be entered under the configuration key `filebeat.inputs`: + + {{< file "/etc/filebeat/filebeat.yml" yml >}} +filebeat.inputs: +- type: docker + containers.ids: + - '*' + processors: + - add_docker_metadata: ~ +{{< /file >}} + +1. Uncomment the following line and change its value to `true`, which will permit Filebeat to create associated Kibana dashboards for captured container logs: + + {{< file "/etc/filebeat/filebeat.yml" yml >}} +setup.dashboards.enabled: true +{{< /file >}} + +1. Finally, add the following `autodiscover` configuration to the end of the `filebeat.yml` file: + + {{< file "/etc/filebeat/filebeat.yml" yml >}} +filebeat.autodiscover: + providers: + - type: docker + hints.enabled: true +{{< /file >}} + +1. Enable the `nginx` module, which will be used later in this tutorial: + + sudo /usr/bin/filebeat modules enable nginx + +1. The remainder of the configuration file will instruct Filebeat to send logs to the locally-running Elasticsearch instance, which can be left unchanged. Start Filebeat: + + sudo systemctl start filebeat + +### Metricbeat + +Like Filebeat, configure Metricbeat similarly to dynamically discover running containers to monitor. + +1. Metricbeat uses a module to collect container metrics. Issue the following command to enable the `docker` and `nginx` modules: + + sudo /usr/bin/metricbeat modules enable docker + sudo /usr/bin/metricbeat modules enable nginx + +1. Uncomment the following line and change its value to `true`, which will permit Metricbeat to create associated Kibana dashboards for captured container logs: + + {{< file "/etc/metricbeat/metricbeat.yml" yml >}} +setup.dashboards.enabled: true +{{< /file >}} + +1. The remainder of the configuration file will instruct Metricbeat to send logs to the locally-running Elasticsearch instance, which can be left unchanged. Metricbeat can now be started: + + sudo systemctl start metricbeat + +## Visualizing Container Logs and Metrics + +The following example demonstrates how Filebeat and Metricbeat automatically capture container data which can be accessed within Kibana. + +1. To begin, run a simple nginx Docker container on your Linode. + + sudo docker run --name nginx -P -d --label co.elastic.logs/module=nginx nginx + + - This command will run the web server in the background and expose the listening HTTP service under a random port number. + - The `--label` argument is a [hint](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html) to let Filebeat automatically parse the log format of certain container types, which in this case is nginx. + +1. To open a secure connection to Kibana, open an SSH tunnel to port 5601 on your Linode. + + ssh -L 5601:localhost:5601 + + - Replace `` with the username and IP address of your Linode. + - This forwards port 5601 locally to port 5601 on your Linode. + - A comprehensive guide to using SSH tunnels on a variety of platforms is available in our [Create an SSH Tunnel for MySQL guide](/docs/databases/mysql/create-an-ssh-tunnel-for-mysql-remote-access/). + +1. Browse to `http://localhost:5601` in your browser, which will display the following initial landing page for Kibana. + + ![Kibana 6 Landing Page](kibana-landing-page.png "Kibana 6 Landing Page") + +1. Click the **Management** link in the lower left sidebar. The following page will be displayed. Then, click **Index Patterns** to enter the Index Pattern configuration page. + + ![Kibana 6 Management](kibana-management.png "Kibana 6 Management") + +1. Index Patterns dictate how Kibana understands indices that are present in Elasticsearch. In order for some visualizations to display properly, a default index pattern must first be configured. Select **filebeat-\*** on the left side of the page to configure the filebeat-* index pattern. + + ![Kibana 6 Index Patterns](./kibana-filebeat-index.png "Kibana 6 Index Patterns") + +1. Click the **star icon** in the upper right corner of the page to set this index pattern as the default in Kibana. + + ![Kibana 6 Default Index Pattern](./kibana-filebeat-default.png "Kibana 6 Default Index Pattern") + + Kibana is now properly configured with a default index pattern. + +1. Filebeat and Metricbeat are setup to configure Elasticsearch and Kibana automatically, so dashboards and index patterns are loaded and ready to be used. Click on **Dashboard** in the left-hand sidebar, which displays the following page. + + ![Kibana 6 Dashboards](kibana-dashboards.png "Kibana 6 Dashboards") + +1. In the Search bar, type "container" to display pre-populated dashboards for system containers. Click on the **[Metricbeat Docker] Overview** link. + + ![Kibana 6 Container Dashboards](kibana-container-dashboards.png "Kibana 6 Container Dashboards") + +1. The **[Metricbeat Docker] Overview** dashboard will load, which shows several aspects of currently-running container metrics. The dashboard displays a list of running containers, the total number of running, paused, and stopped containers, as well as metrics about container resource consumption. + + ![Kibana 6 Docker Overview](kibana-docker-overview.png "Kibana 6 Docker Overview") + + Scrolling further down, it also shows graphs indicating container resource usage over time, including CPU, memory, and network activity. + + ![Kibana 6 Docker Resources](kibana-docker-resources.png "Kibana 6 Docker Resources") + +1. Before moving on to other Kibana visualizations, generate some log activity from nginx by sending HTTP requests to the listening container. First, find which port the container is listening for requests on using the `docker` command: + + docker ps + + You should see output similar to the following: + + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + 3f0c6d284f1f nginx "nginx -g 'daemon of…" 23 minutes ago Up 23 minutes 0.0.0.0:32769->80/tcp nginx + + From this output, we know that the HTTP server can be reached by issuing requests to port 32769, which is being forwarded to port 80 in the container. The port on your system may be different. + +1. Send several requests to this port using the `curl` command, replacing `` with the number found in the previous step: + + for i in $(seq 1 10) ; do curl localhost: ; done + +1. Now a number of logs are present in Kibana for this container. Click **Discover** in the left-hand sidebar in Kibana. It displays the following screen. + + ![Kibana 6 Discover](kibana-discover.png "Kibana 6 Discover") + + - The histogram near the top of the page indicates the total number of container logs over time. + - The table below the graph contains the contents of individual log contents. + - Clicking on the arrows to the left of each log's timestamp will display the information for each captured log. + +1. Try re-issuing the previous `for ...` command to send another ten `curl` requests to the container and observe how the log histogram changes to reflect the new logs. + +1. Click **Dashboard** in the left-hand sidebar, then click it a second time to enter the dashboard selection screen. Search for "nginx" in the search bar. + + ![Kibana 6 NGINX Dashboards](kibana-nginx-dashboards.png "Kibana 6 NGINX Dashboards") + +1. Click on the **[Filebeat Nginx] Access and error logs** link, which will display a dashboard with a number of visualizations regarding nginx activity. + + ![Kibana 6 NGINX Metricbeat](kibana-nginx-metricbeat.png "Kibana 6 NGINX Metricbeat") + +## Additional Modules + +This tutorial has demonstrated how Filebeat and Metricbeat can automatically capture container metrics and logs without the need to explicitly configure log file paths or configurations. In addition to the nginx examples presented here, the additional links provided below enumerate other modules that can be loaded into Filebeat and Metricbeat for other services. diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-container-dashboards.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-container-dashboards.png new file mode 100644 index 00000000000..3f44c2d1325 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-container-dashboards.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-dashboards.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-dashboards.png new file mode 100644 index 00000000000..03d69da1238 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-dashboards.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-discover.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-discover.png new file mode 100644 index 00000000000..4e02dc7c9dc Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-discover.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-docker-overview.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-docker-overview.png new file mode 100644 index 00000000000..179889d9877 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-docker-overview.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-docker-resources.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-docker-resources.png new file mode 100644 index 00000000000..710ad777636 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-docker-resources.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-filebeat-default.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-filebeat-default.png new file mode 100644 index 00000000000..aefb52b7c0b Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-filebeat-default.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-filebeat-index.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-filebeat-index.png new file mode 100644 index 00000000000..10360c9d3c9 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-filebeat-index.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-initial-page.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-initial-page.png new file mode 100644 index 00000000000..c6689a4fb4e Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-initial-page.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-landing-page.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-landing-page.png new file mode 100644 index 00000000000..81ef368c113 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-landing-page.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-management.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-management.png new file mode 100644 index 00000000000..e990208e8a4 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-management.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-nginx-dashboards.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-nginx-dashboards.png new file mode 100644 index 00000000000..62f9b50b6bd Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-nginx-dashboards.png differ diff --git a/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-nginx-metricbeat.png b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-nginx-metricbeat.png new file mode 100644 index 00000000000..e249d7a49a1 Binary files /dev/null and b/docs/applications/containers/how-to-monitor-containers-with-the-elastic-stack/kibana-nginx-metricbeat.png differ