Skip to content

Commit

Permalink
Renames, cleanup, and more tests for key value pairs
Browse files Browse the repository at this point in the history
  • Loading branch information
philsttr committed Jun 17, 2023
1 parent 9e55a5b commit 07314e5
Show file tree
Hide file tree
Showing 11 changed files with 333 additions and 285 deletions.
65 changes: 34 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ The structure of the output, and the data it contains, is fully configurable.
* [LoggingEvent Fields](#loggingevent-fields)
* [Standard Fields](#standard-fields)
* [MDC fields](#mdc-fields)
* [KeyValuePairs fields](#keyvaluepairs-fields)
* [Key Value Pair fields](#key-value-pair-fields)
* [Context fields](#context-fields)
* [Caller Info Fields](#caller-info-fields)
* [Custom Fields](#custom-fields)
Expand Down Expand Up @@ -991,11 +991,12 @@ The field names can be customized (see [Customizing Standard Field Names](#custo

### MDC fields

By default, each entry in the Mapped Diagnostic Context (MDC) (`org.slf4j.MDC`)
will appear as a field in the LoggingEvent.
By default, `LogstashEncoder`/`LogstashLayout` will write each
[Mapped Diagnostic Context (MDC) (`org.slf4j.MDC`)](https://www.slf4j.org/api/org/slf4j/MDC.html)
entry to the output.

This can be fully disabled by specifying `<includeMdc>false</includeMdc>`,
in the encoder/layout/appender configuration.
To disable writing MDC entries, add `<includeMdc>false</includeMdc>`
to the `LogstashEncoder`/`LogstashLayout` configuration.

You can also configure specific entries in the MDC to be included or excluded as follows:

Expand All @@ -1021,51 +1022,52 @@ It is a configuration error to specify both included and excluded key names.

By default, the MDC key is used as the field name in the output.
To use an alternative field name in the output for an MDC entry,
specify`<mdcKeyFieldName>mdcKeyName=fieldName</mdcKeyFieldName>`:
specify `<mdcKeyFieldName>mdcKeyName=fieldName</mdcKeyFieldName>`:

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<mdcKeyFieldName>key1=alternateFieldNameForKey1</mdcKeyFieldName>
</encoder>
```

### KeyValuePairs fields
### Key Value Pair Fields

By default, each entry in the KeyValuePairs (`org.slf4j.event.KeyValuePair`) via the Fluent Api
will appear as a field in the LoggingEvent.
Slf4j 2's [fluent API](https://www.slf4j.org/manual.html#fluent) supports attaching key value pairs to the log event.

This can be fully disabled by specifying `<includeKeyValuePairs>false</includeKeyValuePairs>`,
in the encoder/layout/appender configuration.
`LogstashEncoder`/`LogstashLayout` will write each key value pair as a field in the output by default.

To disable writing key value pairs, add `<includeKeyValuePairs>false</includeKeyValuePairs>`
to the `LogstashEncoder`/`LogstashLayout` configuration.

You can also configure specific entries in the KeyValuePairs to be included or excluded as follows:
You can also configure specific key value pairs to be included or excluded as follows:

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<includeKeyValuePairsKeyName>key1ToInclude</includeKeyValuePairsKeyName>
<includeKeyValuePairsKeyName>key2ToInclude</includeKeyValuePairsKeyName>
<includeKeyValueKeyName>key1ToInclude</includeKeyValueKeyName>
<includeKeyValueKeyName>key2ToInclude</includeKeyValueKeyName>
</encoder>
```

or

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<excludeKeyValuePairsKeyName>key1ToExclude</excludeKeyValuePairsKeyName>
<excludeKeyValuePairsKeyName>key2ToExclude</excludeKeyValuePairsKeyName>
<excludeKeyValueKeyName>key1ToExclude</excludeKeyValueKeyName>
<excludeKeyValueKeyName>key2ToExclude</excludeKeyValueKeyName>
</encoder>
```

When key names are specified for inclusion, then all other fields will be excluded.
When key names are specified for exclusion, then all other fields will be included.
When key names are specified for inclusion, then all other keys will be excluded.
When key names are specified for exclusion, then all other keys will be included.
It is a configuration error to specify both included and excluded key names.

By default, the KeyValuePairs key is used as the field name in the output.
To use an alternative field name in the output for an KeyValuePairs entry,
specify`<keyValuePairsKeyFieldName>kvpKeyName=fieldName</keyValuePairsKeyFieldName>`:
By default, the key is used as the field name in the output.
To use an alternative field name in the output for an key value pair,
specify`<keyValuePairsKeyFieldName>keyName=fieldName</keyValuePairsKeyFieldName>`:

```xml
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<keyValuePairsKeyFieldName>key1=alternateFieldNameForKey1</keyValuePairsKeyFieldName>
<keyValueKeyFieldName>key1=alternateFieldNameForKey1</keyValueKeyFieldName>
</encoder>
```

Expand Down Expand Up @@ -2633,21 +2635,22 @@ The provider name is the xml element name to use when configuring. Each provider
that specify an alternate field name to output for specific MDC key (none)</li>
</ul>
</td>
</tr> <tr>
</tr>
<tr>
<td valign="top"><tt>keyValuePairs</tt></td>
<td>
<p>Outputs entries from the keyValuePairs.
Will include all entries by default.
When key names are specified for inclusion, then all other fields will be excluded.
When key names are specified for exclusion, then all other fields will be included.
<p>Outputs key value pairs added via slf4j's fluent api.
Will include all key value pairs by default.
When key names are specified for inclusion, then all other keys will be excluded.
When key names are specified for exclusion, then all other keys will be included.
It is a configuration error to specify both included and excluded key names.
</p>
<ul>
<li><tt>fieldName</tt> - Sub-object field name (no sub-object)</li>
<li><tt>includeKeyValuePairsKeyName</tt> - Name of keys to include (all)</li>
<li><tt>excludeKeyValuePairsKeyName</tt> - Name of keys to exclude (none)</li>
<li><tt>keyValuePairsKeyFieldName</tt> - Strings in the form <tt>kvpKeyName=fieldName</tt>
that specify an alternate field name to output for specific KeyValuePairs key (none)</li>
<li><tt>includeKeyName</tt> - Name of keys to include (all)</li>
<li><tt>excludeKeyName</tt> - Name of keys to exclude (none)</li>
<li><tt>keyFieldName</tt> - Strings in the form <tt>keyName=fieldName</tt>
that specify an alternate field name to output for specific key (none)</li>
</ul>
</td>
</tr>
Expand Down
95 changes: 54 additions & 41 deletions src/main/java/net/logstash/logback/LogstashFormatter.java
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,29 @@
import net.logstash.logback.composite.JsonProvider;
import net.logstash.logback.composite.JsonProviders;
import net.logstash.logback.composite.LogstashVersionJsonProvider;
import net.logstash.logback.composite.loggingevent.*;
import net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider;
import net.logstash.logback.composite.loggingevent.CallerDataJsonProvider;
import net.logstash.logback.composite.loggingevent.KeyValuePairsJsonProvider;
import net.logstash.logback.composite.loggingevent.LogLevelJsonProvider;
import net.logstash.logback.composite.loggingevent.LogLevelValueJsonProvider;
import net.logstash.logback.composite.loggingevent.LoggerNameJsonProvider;
import net.logstash.logback.composite.loggingevent.LoggingEventCompositeJsonFormatter;
import net.logstash.logback.composite.loggingevent.LoggingEventFormattedTimestampJsonProvider;
import net.logstash.logback.composite.loggingevent.LoggingEventJsonProviders;
import net.logstash.logback.composite.loggingevent.LoggingEventThreadNameJsonProvider;
import net.logstash.logback.composite.loggingevent.LogstashMarkersJsonProvider;
import net.logstash.logback.composite.loggingevent.MdcJsonProvider;
import net.logstash.logback.composite.loggingevent.MessageJsonProvider;
import net.logstash.logback.composite.loggingevent.StackTraceJsonProvider;
import net.logstash.logback.composite.loggingevent.TagsJsonProvider;
import net.logstash.logback.fieldnames.LogstashFieldNames;

import ch.qos.logback.classic.pattern.ThrowableHandlingConverter;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.spi.ContextAware;
import com.fasterxml.jackson.databind.JsonNode;
import org.slf4j.event.KeyValuePair;
import org.slf4j.MDC;
import org.slf4j.event.KeyValuePair;

/**
* A {@link LoggingEventCompositeJsonFormatter} that contains a common
Expand Down Expand Up @@ -77,9 +91,9 @@ public class LogstashFormatter extends LoggingEventCompositeJsonFormatter {
private MdcJsonProvider mdcProvider = new MdcJsonProvider();
/**
* When not null, {@link KeyValuePair} properties will be included according
* to the logic in {@link KeyValuePairJsonProvider}.
* to the logic in {@link KeyValuePairsJsonProvider}.
*/
private KeyValuePairJsonProvider kvpProvider = new KeyValuePairJsonProvider();
private KeyValuePairsJsonProvider keyValuePairsProvider = new KeyValuePairsJsonProvider();
private GlobalCustomFieldsJsonProvider<ILoggingEvent> globalCustomFieldsProvider;
/**
* When not null, markers will be included according
Expand Down Expand Up @@ -116,7 +130,7 @@ public LogstashFormatter(ContextAware declaredOrigin, boolean includeCallerData,
getProviders().addStackTrace(this.stackTraceProvider);
getProviders().addContext(this.contextProvider);
getProviders().addMdc(this.mdcProvider);
getProviders().addKvp(this.kvpProvider);
getProviders().addKeyValuePairs(this.keyValuePairsProvider);
getProviders().addGlobalCustomFields(this.globalCustomFieldsProvider);
getProviders().addTags(this.tagsProvider);
getProviders().addLogstashMarkers(this.logstashMarkersProvider);
Expand Down Expand Up @@ -216,20 +230,19 @@ public void setIncludeMdc(boolean includeMdc) {
}
}

public boolean isIncludeKvp() {
return this.kvpProvider != null;
public boolean isIncludeKeyValuePairs() {
return this.keyValuePairsProvider != null;
}

public void setIncludeKvp(boolean includeKvp) {
if (isIncludeKvp() == includeKvp)
return;

if (includeKvp) {
kvpProvider = new KeyValuePairJsonProvider();
addProvider(kvpProvider);
} else {
getProviders().removeProvider(kvpProvider);
kvpProvider = null;
public void setIncludeKeyValuePairs(boolean includeKeyValuePairs) {
if (isIncludeKeyValuePairs() != includeKeyValuePairs) {
if (includeKeyValuePairs) {
keyValuePairsProvider = new KeyValuePairsJsonProvider();
addProvider(keyValuePairsProvider);

Check warning on line 241 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L240-L241

Added lines #L240 - L241 were not covered by tests
} else {
getProviders().removeProvider(keyValuePairsProvider);
keyValuePairsProvider = null;
}
}
}

Expand Down Expand Up @@ -310,41 +323,41 @@ public void addMdcKeyFieldName(String mdcKeyFieldName) {
}
}

public List<String> getIncludeKvpKeyNames() {
return isIncludeKvp()
? kvpProvider.getIncludeKvpKeyNames()
public List<String> getIncludeKeyValueKeyNames() {
return isIncludeKeyValuePairs()
? keyValuePairsProvider.getIncludeKeyNames()
: Collections.emptyList();

Check warning on line 329 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L328-L329

Added lines #L328 - L329 were not covered by tests
}

public void addIncludeKvpKeyName(String includedKvpKeyName) {
if (isIncludeKvp()) {
kvpProvider.addIncludeKvpKeyName(includedKvpKeyName);
public void addIncludeKeyValueKeyName(String includedKeyValueKeyName) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.addIncludeKeyName(includedKeyValueKeyName);
}
}
public void setIncludeKvpKeyNames(List<String> includeKvpKeyNames) {
if (isIncludeKvp()) {
kvpProvider.setIncludeKvpKeyNames(includeKvpKeyNames);
public void setIncludeKeyValueKeyNames(List<String> includeKeyValueKeyNames) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.setIncludeKeyNames(includeKeyValueKeyNames);

Check warning on line 339 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L339

Added line #L339 was not covered by tests
}
}

Check warning on line 341 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L341

Added line #L341 was not covered by tests

public List<String> getExcludeKvpKeyNames() {
return isIncludeKvp()
? kvpProvider.getExcludeKvpKeyNames()
public List<String> getExcludeKeyValueKeyNames() {
return isIncludeKeyValuePairs()
? keyValuePairsProvider.getExcludeKeyNames()
: Collections.emptyList();

Check warning on line 346 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L345-L346

Added lines #L345 - L346 were not covered by tests
}
public void addExcludeKvpKeyName(String excludedKvpKeyName) {
if (isIncludeKvp()) {
kvpProvider.addExcludeKvpKeyName(excludedKvpKeyName);
public void addExcludeKeyValueKeyName(String excludedKeyValueKeyName) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.addExcludeKeyName(excludedKeyValueKeyName);
}
}
public void setExcludeKvpKeyNames(List<String> excludeKvpKeyNames) {
if (isIncludeKvp()) {
kvpProvider.setExcludeKvpKeyNames(excludeKvpKeyNames);
public void setExcludeKeyValueKeyNames(List<String> excludeKeyValueKeyNames) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.setExcludeKeyNames(excludeKeyValueKeyNames);

Check warning on line 355 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L355

Added line #L355 was not covered by tests
}
}

Check warning on line 357 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L357

Added line #L357 was not covered by tests
public void addKvpKeyFieldName(String kvpKeyFieldName) {
if (isIncludeKvp()) {
kvpProvider.addKvpKeyFieldName(kvpKeyFieldName);
public void addKeyValueKeyFieldName(String keyValueKeyFieldName) {
if (isIncludeKeyValuePairs()) {
keyValuePairsProvider.addKeyFieldName(keyValueKeyFieldName);
}
}

Expand Down Expand Up @@ -437,9 +450,9 @@ public void addProvider(JsonProvider<ILoggingEvent> provider) {
} else if (provider instanceof MdcJsonProvider) {
getProviders().removeProvider(this.mdcProvider);
this.mdcProvider = (MdcJsonProvider) provider;
} else if (provider instanceof KeyValuePairJsonProvider) {
getProviders().removeProvider(this.kvpProvider);
this.kvpProvider = (KeyValuePairJsonProvider) provider;
} else if (provider instanceof KeyValuePairsJsonProvider) {
getProviders().removeProvider(this.keyValuePairsProvider);
this.keyValuePairsProvider = (KeyValuePairsJsonProvider) provider;

Check warning on line 455 in src/main/java/net/logstash/logback/LogstashFormatter.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/net/logstash/logback/LogstashFormatter.java#L454-L455

Added lines #L454 - L455 were not covered by tests
}
getProviders().addProvider(provider);
}
Expand Down
Loading

0 comments on commit 07314e5

Please sign in to comment.