Switch branches/tags
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.adoc

README.adoc

JEP-304: Evergreen Client Error Telemetry Logging

Abstract

Sending error logging to the backend is a critical aspect of Evergreen to be able to implement the Automated Health Assessment story [1].

Jenkins Evergreen needs a way to extract the error logging from the Jenkins instance.

The current document describes how we plan to extract those logs so that the evergreen-client can find, process and send them to the backend.

Specification

Jenkins will be configured to send the logs to the evergreen-client. The client being already connected by definition, this will avoid having to maintain two ways to authenticate and access the backend.

Summary

Logs are sent:

  • in JSON format for easier parsing by the client.

    • Each log entry separated by a newline, so that each line can be parsed as a JSON object, without needing a JSON streaming parser.

  • only if the level is greater or equal to WARNING.

  • to the disk, under $JENKINS_VAR/logs/evergreen.log.N rotating files.

    • By rotating, we mean evergreen.log.0 is the one file to "tail" to receive new log entries. It is left to the consumer to decide how to know where to start from in case of a fresh start, or restart. Though it is a subject for another design document, it is suggested to:

      • store the last timestamp that was read ;

      • be able to idempotently send everything to the backend (in case of a fresh startup, or a missing value for the last read and sent timestamp), and that already sent logs are deduplicated or ignored there.

  • in UTF-8 (The JSON specification allows some other encodings. We restrict it to UTF-8 only)

    • 5 files maximum ;

    • Approximately 10 MB maximum per file.

    • This should give largely enough time for the client to parse and send everything before it gets rotated, even more because only logs ≥ WARNING will be sent. A total maximum of 50 MB of disk consumption for all this seems very reasonable with modern disk sizes.

Jenkins Plugin

We will host the code for all this in a new Evergreen Jenkins Plugin that will be required to be installed and active on any Jenkins Evergreen instance.

See Packaging: Jenkins plugin or Jenkins module for why a Jenkins plugin.

Logging format

🔥
The JSON excerpts below are pretty-printed for readability. This will not be the case in real conditions so that one line is one log entry.

The logs is sent in JSON format. A version tag is put with initial value 1. This will make it easier to evolve the format if needed.

The formatting is done by a JSONFormatter extending java.util.logging.Formatter.

Required fields for a single log entry
{
  "version": 1,
  "timestamp": 1522840762769,
  "name": "io.jenkins.plugins.SomeTypicalClass",
  "level": "WARNING",
  "message": "the message\nand another line",
  "exception": {
    "raw": "serialized exception\n many \n many \n lines"
  }
}
version

Initially 1. Will be used in case we need to evolve the format.

timestamp

Timestamp for the log in Epoch format

name

The logger name. It is generally the name of the emitting class, but can be any text or even the empty string.

level

The level of criticality of the log. In practice, should always be SEVERE or WARNING. But it is expected that this could be any value of the java.util.logging.Level class, if this proves necessary in the future.

message

The message for this log entry.

exception

(optional) See below Exceptions handling.

Exceptions handling

Exceptions are serialized in a exception field. This field can be absent.

To make potential future evolution easier, we are making the exception field an object. For now, we only put a single raw field in this object.

Newlines and special characters

As JSON does not allow multiline strings, newlines are encoded using the typical \n.

Destination

The logs are written to disk in rotating log files under $JENKINS_VAR/logs. This uses a custom FileHandler.

ℹ️
This part is really meant to stay an internal implementation detail: the main goal is to have the client able to access and send those logs. So we might in the future change that for instance to send those logs to the client directly using a local socket connection (see Why not send logs to the client using a socket).

Rotation parameters

As explained previously, we plan to rotate on 5 files, of 10 MB each maximum. If contrary to what we think, this proves not enough, or too much, we will be able to easily adjust those settings by delivering a new version of the Evergreen Jenkins Plugin handling this.

Motivation

There is no existing tooling for this.

Reasoning

Why not send logs to the client using a socket

The SocketHandler class provided by the JDK is not usable for any real life usage. There is no reconnection logic at all: this means that any even temporary unavailability of the server socket will break the connection, and no logs will ever be sent again when it becomes available.

So we chose the easiest path to not roll our own SocketHandler at least for now, and use a FileHandler and rotating log files instead for more out-of-the-box robustness (the producer can send its data, and the consumer can read it when ready).

Packaging: Jenkins plugin or Jenkins module

We chose the Jenkins plugin path because this is a simpler path forward. The Jenkins Project has already everything ready to handle the hosting and release process of plugins.

Using a jenkins-module would mean we need to set up a custom Jenkins WAR build for Evergreen. This would also mean we cannot use the quality assurance improvements we have started to put in place in the same way for Evergreen, and for the standard Jenkins core delivery.

Though having a dedicated WAR packaging for Jenkins Evergreen is very likely something we will do in the future, we deemed preferrable to defer it to later, and focus instead on the other yet unresolved design questions of Evergreen.

The exception field in the JSON logs

Following YAGNI, we use a simple exception field where the exception is basically serialized as text. We will likely use hudson.Functions.printThrowable() [2] for the text formatting.

If needed, this might be replaced in the future by a richer structure. In such case we will bump the version field to a higher number.

Backwards Compatibility

JSON format versioning

As explained above, we put a version field in the JSON logs. This will help accomodate a format change if this becomes a necessity.

Logs handling

We might decide at some point to change the way we pass the logs from Jenkins to the Evergreen Client. If this happens, we will probably go through a period of time where we’d log both to the disk, and to the Socket, so that we are not dependent on the version of the client.

Security

There are no security risks related to this proposal.

ℹ️
The sensitive part of this feature lies in sending data outside of the instance. This will be handled in JENKINS-49811.

Infrastructure Requirements

There are no new infrastructure requirements related to this proposal.

The infrastructure side of this feature will be handled in JENKINS-49811.

Testing

JSON Logging as and where expected

The biggest part of this feature should be tested directly in the Evergreen codebase. It is indeed easy to check that the logs are indeed found under the expected location. * See https://github.com/jenkins-infra/evergreen/pull/43 for such example

Test with the plugin

The Evergreen Jenkins Plugin is going to start introducing some core changes to the way Jenkins logs things.

We want to check we do not inadvertently change, or disable totally for instance, the usual logging Jenkins has.

Prototype Implementation

References

❗️

When moving this JEP from a Draft to "Accepted" or "Final" state, include links to the pull requests and mailing list discussions which were involved in the process.


1. a dedicated proposal will be written for this in JENKINS-50294
2. which formats chained exceptions more readably than Exception.printStackTrace().