Observability with Elasticsearch

APM Node.js and APM RUM agents on Docker

Introduction

Elastic Stack (ELK) Docker Composition, with a Node.JS application reporting APM client + server side metrics and traces to APM Server.

Suitable for Demoing and MVPs.

Stack Version: 8.5.3 🎉 - Based on Official Elastic Docker Images

You can change Elastic Stack version by setting ELK_VERSION in .env file and rebuild your images. Any version >= 8.0.0 is compatible with this template.

Based on elastdocker

Main Features 📜

• Node.JS application with Node APM Agent.
• Elasticsearch, configured as a Production Single Node Cluster.
• Security Enabled on Elasticsearch and Kibana.
• APM Server receives Non secure connections.
• Use Docker-Compose and .env to configure your entire stack parameters.
• Persist Elasticsearch's Keystore and SSL Certifications.

---

Requirements

• Docker 20.05 or higher
• Docker-Compose 1.29 or higher
• 4GB RAM (For Windows and MacOS make sure Docker's VM has more than 4GB+ memory.)

Setup

1. Clone the Repository

   git clone https://github.com/lizozom/docker-elastic-observability-with-apm.git

2. Initialize Elasticsearch Keystore and TLS Self-Signed Certificates

   $ make setup

   For Linux's docker hosts only. By default virtual memory is not enough so run the next command as root sysctl -w vm.max_map_count=262144

3. Start Elastic Stack

   $ make elk           <OR>         $ docker-compose up -d		<OR>		$ docker compose up -d

4. Navigate to the web application at http://localhost:3000/ to generate APM traces.

5. Visit Kibana at https://localhost:5601/app/apm to see the APM traces from our webapp

   Default Username: elastic, Password: changeme

   

Configuration

• Some Configuration are parameterized in the .env file.
  • ELASTIC_PASSWORD, user elastic's password (default: changeme pls).
  • ELK_VERSION Elastic Stack Version (default: 8.3.2)
  • ELASTICSEARCH_HEAP, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)
  • Other configurations which their such as cluster name, and node name, etc.
• Elasticsearch Configuration in elasticsearch.yml at ./elasticsearch/config.
• Kibana Configuration in kibana.yml at ./kibana/config.

Setting Up Keystore

You can extend the Keystore generation script by adding keys to ./setup/keystore.sh script. (e.g Add S3 Snapshot Repository Credentials)

To Re-generate Keystore:

make keystore

Notes

• ⚠️ Elasticsearch HTTP layer is using SSL, thus mean you need to configure your elasticsearch clients with the CA in secrets/certs/ca/ca.crt, or configure client to ignore SSL Certificate Verification (e.g --insecure in curl).

• Makefile is a wrapper around Docker-Compose commands, use make help to know every command.

• Elasticsearch will save its data to a volume named elasticsearch-data

• Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the ./secrets directory by the setup command.

• Make sure to run make setup if you changed ELASTIC_PASSWORD and to restart the stack afterwards.

• Important: For Linux Users it's recommended to set the following configuration (run as root)

  sysctl -w vm.max_map_count=262144
  

  By default, Virtual Memory is not enough.

License

MIT License Copyright (c) 2022 Sherif Abdel-Naby

Contribution

PR(s) are Open and Welcomed.