Spring Boot Logging with ELK-B Stack v8.12.0

A comprehensive guide to leverage the power of ELK-B Stack for centralized log management.

This repo contains configuration files for setting up structured logging in Spring Boot applications and learn how to seamlessly ship logs to the ELK-B Stack. Using Docker Compose, streamlined the entire setup process.

Table of Contents:

• Introduction to Centralized Logging Process

• Configuring ELK Stack

• Configuring Filebeat

• Enabling Structured Logging in Spring Boot

• Processing JSON Fields from the logs

• Acknowledgment

Introduction to Centralized Logging Process

Centralized logging involves collecting logs from various sources into a single location for easier management and analysis. This process typically consists of several stages:

1. Log Generation:

Logs are generated by applications, services, and systems as they perform actions and encounter events. These logs contain valuable information about the behavior and health of the system.

2. Log Shipping:

Once generated, logs need to be transported from their source to a centralized location, through various methods such as agents, APIs, or protocols like syslog or HTTP.

3. Log Processing:

Upon arrival at the centralized location, logs are processed to extract relevant information, enrich them with additional context, and filter out noise. This step ensures that only meaningful data is retained for analysis.

4. Log Indexing:

Process of indexing logs in a structured format within a database or search engine, such as Elasticsearch. Indexed logs can be efficiently searched, queried, and analyzed based on different criteria.

5. Query and Visualization:

Indexed logs can be queried and visualized using tools like Kibana. This enables users to search for specific log entries, create dashboards, and generate visualizations to gain insights into system behavior and performance.

6. Monitoring and Analysis:

Real-time monitoring and analysis of logs allow for proactive identification of issues, performance bottlenecks, and security threats. By analyzing log data in real-time, organizations can quickly respond to incidents and optimize system performance.

7. Archiving and Retention:

Archived logs are stored for compliance, auditing, and historical analysis purposes. Proper log retention policies ensure that valuable data is retained for the required duration.

Configuring ELK Stack

I stick with the config of the original author which is:

„Multi-node cluster“ working in a single node mode.

1. Environment Variables Setup:

The .env file contains environment variables used to configure various aspects of the ELK Stack, such as passwords, ports, memory limits, and domain addresses. Variables are referenced in docker-compose-elk.yml and docker-compose-fb.yml files using ${reference} syntax. ELK stack and filebeat may be started on different servers, therefore these two env variables are separated from each other: ELK_DOMAIN, ELK_DOMAIN_FOR_FB; The former is used in ELK stack and the latter is used in filebeat

2. Overview of the docker-compose-elk.yml:

• This Docker Compose file defines a multi-service environment for setting up the ELK (Elasticsearch, Logstash, Kibana) Stack. It consists of several services, including Elasticsearch, Logstash, Redis, and Kibana, each configured to work together to provide centralized log management capabilities.
• The setup service initializes the ELK Stack, creates necessary certificates, and sets up passwords for authentication.
• Services like Elasticsearch (es01), Logstash (logstash-agent and logstash-central), and Kibana (kibana) are configured with specific ports and environment variables for communication and configuration purposes.
• Volumes are defined to persist data for Elasticsearch (esdata01 and kibanadata01), ensuring that data is retained across container restarts or updates.

3. Starting the ELK Stack:

Make sure you have both the docker-compose-elk.yml and the .env files in the same directory on your system. Run the following command to start the ELK Stack:

docker-compose -f "/path/to/docker-compose-elk.yml" up -d

4. Verification steps to ensure that the ELK Stack is up and running correctly:

Open a web browser and navigate to https://localhost:5601. after 1-2 minutes you should see the Kibana login page. Use the credentials configured in the .env file to log in (default username is elastic and password is ELASTIC_PASSWORD).

Configuring Filebeat

1. Starting Filebeat using docker-compose-fb.yml:

This file defines the configuration for running Filebeat as a Docker container. Run filebeat with following command:

docker-compose -f "/path/to/docker-compose-fb.yml" up -d

2. Load Kibana dashboards using Setup Method¹:

In the Filebeat configuration file, the output.elasticsearch must be active when invoking the setup method. furthermore because logstash is enabled in output, it must be deactivated when invoking the setup method. so the final command would be as follows:

docker exec -it filebeat filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.enabled=true -e

# -e is optional and sends output to standard error instead of the configured log output.

3. Verification of Created Files after Running Setup Methods:

During running of the setup phase, the following messages are displayed, which verifies the setup process:

.
.
.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
.
.
.
Loaded dashboards
Loaded Ingest pipelines

after setup process getting finished, following files can be seen after logging into Kibana:

• Ingest Pipelines: filebeat-[version]-[modules]-pipeline @ Stack Management > Ingest Pipelines

• Indices: filebeat-[version]-[Date] @ Stack Management > Index Management > Indices

• Created Data Streams: filebeat-[version] @ Stack Management > Index Management > Data streams

• Index Templates: filebeat-[version] @ Stack Management > Index Management > Index Templates

Enabling Structured Logging in Spring Boot

1. add latest compatible version for logstash-logback-encoder to pom.xml

At the moment of writing this document, I'm using this version of spring boot:

    <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.7.7</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

Referring to dependency versions of above-mentioned spring-boot-starter², logback is version 1.2.*. According to logstash-logback-encoder release page, latest compatible version with version 1.2 is the following one:

    <dependencies>
    ....
    <dependency>
        <groupId>net.logstash.logback</groupId>
        <artifactId>logstash-logback-encoder</artifactId>
        <version>7.3</version>
    </dependency>
    ....
</dependencies>

2. add required resource file

add provided logback-spring.xml file to resources standard maven folder of corresponding project. It's added to a spring boot profile named json. this file is almost like original file with some modifications made³.

3. enable json profile in application.yml file

spring:
  application:
    name: APPLICATION_NAME
  profiles:
    active: OTHER_PROFILES,json

4. verification of created files

doing above-mentioned steps, after running projects, console logs will be same as before. Furthermore, two log file will be generated in configured path for log; One is the original log file (APPLICATION_NAME.log) and the other is APPLICATION_NAME.json.log file which contains JSON-Structured-Logging

Processing JSON Fields from the logs

following processors config section in filebeat.yml file, would cause parsing JSON specified fields from generated file in specified path⁴. List of fields are extracted from original document of logstash-logback-encoder ⁵. fields will be added to created index under target field.

- type: filestream
  id: default-filestream-id
  enabled: true
  paths:
    - /usr/share/filebeat/ingest_data/*.log.json

processors:
- decode_json_fields:
  fields: ["@timestamp", "@version", "message", "logger_name", "thread_name", "level", "level_value", "stack_trace", "tags", "spanId", "traceId"]
  process_array: false
  max_depth: 1
  target: "logFileJson"
  overwrite_keys: false
  add_error_key: true

explanation of parameters

• paths: *.log.json files are gathered from specified path (/usr/share/filebeat/ingest_data/*.log.json)
• fields: specified fields will be extracted from each row in log file.
• max_depth: The maximum parsing depth. A value of 1 will decode the JSON objects in fields indicated in fields, a value of 2 will also decode the objects embedded in the fields of these parsed documents. The default is 1.
• target: all extracted (key,val) from each row will be added under this node, (i.e. "logFileJson"). if "" is set for target value, all fields will be added to root which may cause COLLISION between already available fields and extracted fields
• overwrite_keys: A Boolean value that specifies whether existing keys in the event are overwritten by keys from the decoded JSON object
• add_error_key: If set to true and an error occurs while decoding JSON keys, the error field will become a part of the event with the error message. If set to false, there will not be any error in the event’s field.

Acknowledgment

When I first started learning about setting up centralized logging with the ELK stack, I found it quite challenging, especially when trying to configure a TLS-enabled ELK stack in single-node mode. That's when I came across the original article.

I want to express my gratitude to the original author, paranerd, for the generosity in sharing the knowledge and expertise. The article not only provided me with insights but also saved me considerable time and effort. Thanks a MILLION for sharing knowledge with the community!

While I've summarized key concepts here, added spring-boot configs and extracted environments to separate env files, I highly recommend exploring the original article⁶ for a deeper understanding of the subject.

---

I hope this material has been informative and helpful in your journey towards implementing centralized logging in your spring boot projects.

Thank you for your attention and happy logging!

Footnotes

1. https://www.elastic.co/guide/en/beats/filebeat/8.12/load-kibana-dashboards.html ↩

2. https://docs.spring.io/spring-boot/docs/2.7.7/reference/html/dependency-versions.html#appendix.dependency-versions.properties ↩

3. https://github.com/spring-projects/spring-boot/tree/v2.7.7/spring-boot-project/spring-boot/src/main/resources/org/springframework/boot/logging/logback ↩

4. https://www.elastic.co/guide/en/beats/filebeat/8.12/decode-json-fields.html ↩

5. https://github.com/liangyanfeng/logstash-logback-encoder/blob/master/README.md ↩

6. https://thegermancoder.com/blog/2023/06/02/docker-logging-with-elasticsearch-logstash-and-kibana-8.x/ ↩