Monitor your Spring Boot application with the Elastic Stack all around
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
img
java
lightsail
local
workshop
.gitignore
LICENSE
readme.md

readme.md

Microservice Monitoring

Monitor logs, metrics, pings, and traces of your distributed (micro-) services. There are also slides walking you through the features of this repository.

Features

  • X-Pack Monitoring: Start the overview page to show the systems we are using for monitoring.
  • Metricbeat System: Show the [Metricbeat System] Overview dashboard in Kibana and then switch to [Metricbeat System] Host overview. If you show all hosts, you will see little spikes approximately every 5 minutes — this is a rogue process we are running with a cron job and we want to find it.
  • Build an overview with the Time Series Visual Builder:
    • A sum over the field system.memory.actual.used.bytes and group by the term beat.name.
    • A sum over the field system.process.memory.rss.bytes and group by the term system.process.name. Optionally move this visualization to the negative axis to make it easier to visualize with a calculation on params.process*-1 (process is your variable name).

  • Packetbeat: Show the [Packetbeat] Overview, [Packetbeat] Flows, [Packetbeat] MySQL performance, and [Packetbeat] HTTP dashboard, let attendees access the various URLs and see the corresponding graphs. In Discover you can point out the proc enrichment for nginx, Java, MySQL, and the APM server. Optionally show the [Packetbeat] TLS Sessions and [Packetbeat] DNS Tunneling dashboards as well.
  • Filebeat modules: Show the [Filebeat Nginx] Access and error logs, [Filebeat MySQL] Overview, [Filebeat System] Syslog dashboard, [Filebeat System] SSH login attempts, and [Osquery Result] Compliance pack dashboards.
  • Filebeat: Let attendees hit /good with a parameter and point out the MDC logging under json.name and the context view for one log message. Let attendees hit /bad and /null to show the stacktrace both in the JSON log file and in Kibana by filtering down on application:java and json.severity: ERROR. Also point out the cloud meta.* and host.* information. And show the json.stack_hash, which you can use for visualizations too.

  • Auditbeat: Show changes to the /opt/ folder with the [Auditbeat File Integrity] Overview dashboard.
  • Heartbeat: Run Heartbeat and show the Heartbeat HTTP monitoring dashboard in Kibana, then stop and start the frontend application with ansible-playbook restart_frontend.yml or do it manually and see the change.
  • Metricbeat: Show the [Metricbeat Nginx] Overview and [Metricbeat MySQL] Overview dashboards.
  • Metricbeat HTTP: Show /health and /metrics with cURL (credentials are admin and secret). Then collect the same information with Metricbeat's HTTP module and show it in Kibana's Discover tab.
  • Metricbeat JMX: Display the same /health and /metrics data and its collection through JMX.
  • Visual Builder: Build a more advanced visualization with the Time Series Visual Builder, for example to show the heap usage in percent by calculating the average of jolokia.metrics.memory.heap_usage.used divided by the max of jolokia.metrics.memory.heap_usage.max.

  • Annotations: Include the deployment events as an annotations.

  • APM: Show the traces so far, point out the MySQL queries (currently on the backend instance only), and where things are slow or throwing errors. If there is not enough activity on the instances, call ./ab.sh on the monitor instance.
  • Kibana Dashboard Mode: Point attendees to the Kibana instance to let them play around on their own.

Setup

If the network connection is decent, show it on Amazon Lightsail. Otherwise fall back to the local setup and have all the dependencies downloaded in advance.

Lightsail

Make sure you have run this before the demo, because some steps take time and require a decent internet connection.

  1. Make sure you have your AWS account set up, access key created, and added as environment variables in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Protip: Use https://github.com/sorah/envchain to keep your environment variables safe.
  2. Create the Elastic Cloud instance with the same version as specified in variables.yml's elastic_version, enable Kibana as well as the GeoIP & user agent plugins, and set the environment variables with the values for ELASTICSEARCH_HOST, ELASTICSEARCH_USER, ELASTICSEARCH_PASSWORD, as well as KIBANA_HOST, KIBANA_ID.
  3. Change into the lightsail/ directory.
  4. Change the settings to a domain you have registered under Route53 in inventory, variables.tf, and variables.yml. Set the Hosted Zone for that domain and export the Zone ID under the environment variable TF_VAR_zone_id. If you haven't created the Hosted Zone yet, you should set it up in the AWS Console first and then set the environment variable.
  5. If you haven't installed the AWS plugin for Terraform, get it with terraform init first. Then create the keypair, DNS settings, and instances with terraform apply.
  6. Open HTTPS on the network configuration on all instances, MySQL on the backend, and TCP 8200 on the monitoring instance (waiting for this Terraform issue).
  7. Apply the base configuration to all instances with ansible-playbook configure_all.yml.
  8. Apply the instance specific configurations with ansible-playbook configure_backend.yml and ansible-playbook configure_monitor.yml.
  9. Deploy the JARs with ansible-playbook deploy_bad.yml, ansible-playbook deploy_backend.yml, and ansible-playbook deploy_frontend.yml (Ansible is also building them).

When you are done, remove the instances, DNS settings, and key with terraform destroy.

Workshop

Very similar to the Lightsail setup above. The main difference is that everything is running on one instance and you need to open the port 5601 for Kibana (Elasticsearch, APM,... are only accessible on localhost) and 88 if you want to include the PHP examples.

Credentials:

  • SSH: ssh elastic-admin@workshop-<number>.xeraa.wtf elastic-admin / secret
  • Elasticsearch: http://localhost:9200 admin / secret
  • Kibana: http://workshop-<number>.xeraa.wtf:5601 admin / secret

Local

Make sure you have run this before the demo, because some steps take time and require a decent internet connection.

  1. Change into the local/ directory.
  2. Run docker-compose up, which will bring up Elasticsearch, Kibana, and all the Beats.
  3. Run the Java applications from their directories with gradle bootRun.

When you are done, stop the Java applications and remove the Docker setup with docker-compose down -v.

Todo