Skip to content

Commit

Permalink
Virtual Properties added
Browse files Browse the repository at this point in the history
* contextual data serialization
* variable resolution: on startup or on demand
* JacksonJsonLayout is NOT the default anymore due to dependency on Log4j2 Configuration instance
  • Loading branch information
rfoltyns committed Sep 27, 2019
1 parent dd729f7 commit 5369e4c
Show file tree
Hide file tree
Showing 16 changed files with 1,323 additions and 30 deletions.
29 changes: 21 additions & 8 deletions log4j2-elasticsearch-core/README.md
Expand Up @@ -112,28 +112,28 @@ or
</Appenders>
```

NOTE: Be aware that template parsing errors on cluster side DO NOT prevent plugin from loading - error is logged on client side and startup continues.
NOTE: Be aware that template parsing errors on cluster side MAY NOT prevent plugin from loading - error is logged on client side and startup continues.

### Message output

There are numerous ways to generate JSON output:

#### JacksonJsonLayout

(default)

Since 1.3, [org.appenders.log4j2.elasticsearch.JacksonJsonLayout](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/JacksonJsonLayout.java) is the default implemetation of [ItemSourceLayout](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/ItemSourceLayout.java). It will serialize LogEvent using Jackson mapper configured with a set of default and (optional) user-provided mixins (see: [JacksonMixInAnnotations docs](https://github.com/FasterXML/jackson-docs/wiki/JacksonMixInAnnotations)).
Since 1.3, [org.appenders.log4j2.elasticsearch.JacksonJsonLayout](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/JacksonJsonLayout.java) - implemetation of [ItemSourceLayout](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/ItemSourceLayout.java) - can be specified to handle incoming LogEvent(s). It will serialize LogEvent(s) using Jackson mapper configured with a set of default and (optional) user-provided mixins (see: [JacksonMixInAnnotations docs](https://github.com/FasterXML/jackson-docs/wiki/JacksonMixInAnnotations)) and (since 1.4) [Virtual Properties](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/VirtualProperty.java).

Default set of mixins limits LogEvent output by shrinking serialized properties list to a 'reasonable minimum'.
Customization of all aspects of LogEvent and Message output are allowed using `JacksonMixIn` elements (see: [JacksonMixInAnnotations docs](https://github.com/FasterXML/jackson-docs/wiki/JacksonMixInAnnotations)) elements.
Additional properties can be specified with [VirtualProperty](#virtual-properties) elements.
Customizations of all aspects of LogEvent and Message output are allowed using `JacksonMixIn` elements (see: [JacksonMixInAnnotations docs](https://github.com/FasterXML/jackson-docs/wiki/JacksonMixInAnnotations)) elements.

Furthermore, [ItemSource API](#itemsource-api) allows to use pooled [ByteByfItemSource](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/ByteBufItemSource.java) payloads. Pooling is optional.

Config property | Type | Required | Default | Description
------------ | ------------- | ------------- | ------------- | -------------
afterburner | Attribute | no | false | if `true`, `com.fasterxml.jackson.module:jackson-module-afterburner` will be used to optimize (de)serialization. Since this dependency is in `provided` scope by default, it MUST be declared explicitly.
mixins | Element(s) | no | None | Array of `JacksonMixIn` elements. Can be used to override default serialization of LogEvent, Message and related objects
itemSourceFactory | Element | no | `StringItemSourceFactory` | `ItemSourceFactory` used to create wrappers for serialized items. `StringItemSourceFactory` and `PooledItemSourceFactory` are available
virtualProperties (since 1.4) | Element(s) | no | None | Array of `VirtualProperty` elements. Similar to `KeyValuePair`, can be used to define properties resolvable on the fly, not available in LogEvent(s).
itemSourceFactory | Element | yes (since 1.4) | n/a | `ItemSourceFactory` used to create wrappers for serialized items. `StringItemSourceFactory` and `PooledItemSourceFactory` are available

Default output:

Expand All @@ -147,20 +147,33 @@ Example:
<PooledItemSourceFactory itemSizeInBytes="512" initialPoolSize="10000" />
<JacksonMixIn mixInClass="foo.bar.CustomLogEventMixIn"
targetClass="org.apache.logging.log4j.core.LogEvent"/>
<VirtualProperty name="hostname" value="$${env:hostname:-undefined}"
</JacksonJsonLayout>
...
</Elasticsearch>
```

Custom `org.appenders.log4j2.elasticsearch.ItemSourceLayout` can be provided to appender config to use any other serialization mechanism.

##### Virtual Properties

Since 1.4, `VirtualProperty` elements (`KeyValuePair` on steroids) can be appended to serialized objects.

Config property | Type | Required | Default | Description
------------ | ------------- | ------------- | ------------- | -------------
name | Attribute | yes | n/a |
value | Attribute | yes | n/a | Static value or contextual variable resolvable with <a href="https://logging.apache.org/log4j/2.x/manual/lookups.html">Log4j2 Lookups</a>.
dynamic | Attribute | no | false | if `true`, indicates that value may change over time and should be resolved on every serialization (see [Log4jLookup](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/Log4jLookup.java)). Otherwise, will be resolved only on startup.

Custom lookup can implemented with [ValueResolver](https://github.com/rfoltyns/log4j2-elasticsearch/blob/master/log4j2-elasticsearch-core/src/main/java/org/appenders/log4j2/elasticsearch/ValueResolver.java).

#### Log4j2 JsonLayout
JsonLayout will serialize LogEvent using Jackson mapper configured in log4j-core. Custom `org.apache.logging.log4j.core.layout.AbstractLayout` can be provided to appender config to use any other serialization mechanism.
`JsonLayout` will serialize LogEvent using Jackson mapper configured in log4j-core. Custom `org.apache.logging.log4j.core.layout.AbstractLayout` can be provided to appender config to use any other serialization mechanism.

Output may vary across different Log4j2 versions (see: #9)

#### Raw log message
`messageOnly="true"` can be configured for all the layouts mentioned above to make use of user provided (or default) `org.apache.logging.log4j.message.Message.getFormattedMessage()` implementation.
`messageOnly="true"` can be configured for all layouts mentioned above to make use of user provided (or default) `org.apache.logging.log4j.message.Message.getFormattedMessage()` implementation.

Raw log message MUST:
* be logged with Logger that uses `org.apache.logging.log4j.message.MessageFactory` that serializes logged object to a valid JSON output
Expand Down
Expand Up @@ -29,6 +29,7 @@

import java.util.Map;

import com.fasterxml.jackson.databind.annotation.JsonAppend;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.Marker;
import org.apache.logging.log4j.ThreadContext.ContextStack;
Expand All @@ -42,9 +43,18 @@
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonPropertyOrder;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import org.appenders.log4j2.elasticsearch.VirtualPropertiesWriter;
import org.appenders.log4j2.elasticsearch.VirtualProperty;

@JsonPropertyOrder({ "timeMillis", "loggerName", "level", "marker", "message", "thrown", "threadName"})
@JsonSerialize(as = LogEvent.class)
@JsonAppend(props = {
@JsonAppend.Prop(
name = "virtualProperties", // irrelevant at runtime
type = VirtualProperty[].class, // irrelevant at runtime
value = VirtualPropertiesWriter.class
)
})
public abstract class LogEventJacksonJsonMixIn implements LogEvent {

private static final long serialVersionUID = 1L;
Expand Down
Expand Up @@ -35,10 +35,6 @@
import org.apache.logging.log4j.core.layout.AbstractLayout;
import org.apache.logging.log4j.core.layout.AbstractStringLayout;

import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;


Expand Down Expand Up @@ -123,7 +119,7 @@ public ElasticsearchAppender build() {
}

if (layout == null) {
layout = JacksonJsonLayout.newBuilder().build();
throw new ConfigurationException("No layout provided for Elasticsearch appender");
}

return new ElasticsearchAppender(name, filter, layout, ignoreExceptions, batchDelivery, messageOnly, indexNameFormatter);
Expand Down
@@ -0,0 +1,89 @@
package org.appenders.log4j2.elasticsearch;

/*-
* #%L
* log4j2-elasticsearch
* %%
* Copyright (C) 2019 Rafal Foltynski
* %%
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
* #L%
*/

import com.fasterxml.jackson.databind.DeserializationConfig;
import com.fasterxml.jackson.databind.JsonDeserializer;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.KeyDeserializer;
import com.fasterxml.jackson.databind.SerializationConfig;
import com.fasterxml.jackson.databind.cfg.HandlerInstantiator;
import com.fasterxml.jackson.databind.cfg.MapperConfig;
import com.fasterxml.jackson.databind.introspect.Annotated;
import com.fasterxml.jackson.databind.jsontype.TypeIdResolver;
import com.fasterxml.jackson.databind.jsontype.TypeResolverBuilder;

public class JacksonHandlerInstantiator extends HandlerInstantiator {

private final VirtualProperty[] virtualProperties;
private final ValueResolver valueResolver;
private VirtualPropertiesWriter instance;

/**
* @param virtualProperties properties to be appended
* @param valueResolver used to resolve properties if {@link VirtualProperty#isDynamic()} is true
*/
public JacksonHandlerInstantiator(VirtualProperty[] virtualProperties, ValueResolver valueResolver) {
this.virtualProperties = virtualProperties;
this.valueResolver = valueResolver;
}

@Override
public JsonDeserializer<?> deserializerInstance(DeserializationConfig config, Annotated annotated, Class<?> deserClass) {
return null;
}

@Override
public KeyDeserializer keyDeserializerInstance(DeserializationConfig config, Annotated annotated, Class<?> keyDeserClass) {
return null;
}

@Override
public JsonSerializer<?> serializerInstance(SerializationConfig config, Annotated annotated, Class<?> serClass) {
return null;
}

@Override
public TypeResolverBuilder<?> typeResolverBuilderInstance(MapperConfig<?> config, Annotated annotated, Class<?> builderClass) {
return null;
}

@Override
public TypeIdResolver typeIdResolverInstance(MapperConfig<?> config, Annotated annotated, Class<?> resolverClass) {
return null;
}

/**
* Allows to inject {@link VirtualPropertiesWriter} in order to resolve and write {@link VirtualProperty}-ies
*
* @return Shared {@link VirtualPropertiesWriter}
*/
@Override
public VirtualPropertiesWriter virtualPropertyWriterInstance(MapperConfig<?> config, Class<?> implClass) {
if (instance == null) {
instance = new VirtualPropertiesWriter(
virtualProperties,
valueResolver);
}
return instance;
}

}
Expand Up @@ -25,11 +25,13 @@
import com.fasterxml.jackson.core.util.MinimalPrettyPrinter;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectWriter;
import com.fasterxml.jackson.databind.SerializationConfig;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.module.afterburner.AfterburnerModule;
import org.apache.logging.log4j.core.Layout;
import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.Configuration;
import org.apache.logging.log4j.core.config.ConfigurationException;
import org.apache.logging.log4j.core.config.Node;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginBuilderAttribute;
Expand All @@ -43,7 +45,7 @@
import java.util.List;

/**
* Allows to customize serialization of incoming events
* Allows to customize serialization of incoming events. See {@link Builder} API docs for more details
*/
@Plugin(name = JacksonJsonLayout.PLUGIN_NAME, category = Node.CATEGORY, elementType = Layout.ELEMENT_TYPE, printObject = true)
public class JacksonJsonLayout extends AbstractLayout<ItemSource> implements ItemSourceLayout, LifeCycle {
Expand Down Expand Up @@ -104,11 +106,19 @@ public static class Builder extends org.apache.logging.log4j.core.layout.Abstrac
@PluginElement(JacksonMixIn.ELEMENT_TYPE)
private JacksonMixIn[] mixins = new JacksonMixIn[0];

@PluginElement("VirtualProperty")
private VirtualProperty[] virtualProperties = new VirtualProperty[0];

@PluginBuilderAttribute("afterburner")
private boolean useAfterburner;

@Override
public JacksonJsonLayout build() {

if (getConfiguration() == null) {
throw new ConfigurationException("No Configuration instance provided for " + PLUGIN_NAME);
}

return new JacksonJsonLayout(
getConfiguration(),
createConfiguredWriter(Arrays.asList(mixins)),
Expand All @@ -130,7 +140,29 @@ protected ObjectWriter createConfiguredWriter(List<JacksonMixIn> mixins) {
objectMapper.addMixIn(mixin.getTargetClass(), mixin.getMixInClass());
}

for (VirtualProperty property : virtualProperties) {
if (!property.isDynamic()) {
property.setValue(createValueResolver().resolve(property.getValue()));
}
}

SerializationConfig customConfig = objectMapper.getSerializationConfig()
.with(new JacksonHandlerInstantiator(
virtualProperties,
createValueResolver()
));

objectMapper.setConfig(customConfig);

return objectMapper.writer(new MinimalPrettyPrinter());

}

/**
* @return resolver used when {@link VirtualProperty}(-ies) configured
*/
protected ValueResolver createValueResolver() {
return new Log4j2Lookup(getConfiguration().getStrSubstitutor());
}

protected ObjectMapper createDefaultObjectMapper() {
Expand Down Expand Up @@ -160,6 +192,23 @@ public Builder withMixins(JacksonMixIn... mixins) {
return this;
}

/**
* Allows to append properties to serialized {@link LogEvent} and {@link Message}.
*
* Non-dynamic properties ({@code VirtualProperty#dynamic == false}) will be resolved on {@link #build()} call.
*
* Dynamic properties ({@code VirtualProperty#isDynamic == true}) will NOT be resolved on {@link #build()} call and resolution will be deferred to underlying {@link VirtualPropertiesWriter}.
*
* Similar to Log4j2 {@code KeyValuePair}.
*
* @param virtualProperties properties to be appended to JSON output
* @return this
*/
public Builder withVirtualProperties(VirtualProperty... virtualProperties) {
this.virtualProperties = virtualProperties;
return this;
}

/**
* Allows to configure {@link AfterburnerModule} - (de)serialization optimizer
*
Expand Down
@@ -0,0 +1,52 @@
package org.appenders.log4j2.elasticsearch;

/*-
* #%L
* log4j2-elasticsearch
* %%
* Copyright (C) 2019 Rafal Foltynski
* %%
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
* #L%
*/

import org.apache.logging.log4j.core.lookup.StrSubstitutor;

public class Log4j2Lookup implements ValueResolver {

private final StrSubstitutor strSubstitutor;

public Log4j2Lookup(StrSubstitutor strSubstitutor) {
this.strSubstitutor = strSubstitutor;
}

/**
* Resolves given {@link VirtualProperty} if {@link VirtualProperty#isDynamic()} is true
*
* @param property property to resolve
* @return resolved value
*/
@Override
public String resolve(VirtualProperty property) {
if (property.isDynamic()) {
return resolve(property.getValue());
}
return property.getValue();
}

@Override
public String resolve(String unresolved) {
return strSubstitutor.replace(unresolved);
}

}
@@ -0,0 +1,29 @@
package org.appenders.log4j2.elasticsearch;

/*-
* #%L
* log4j2-elasticsearch
* %%
* Copyright (C) 2019 Rafal Foltynski
* %%
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
* #L%
*/

public interface ValueResolver {

String resolve(String unresolved);

String resolve(VirtualProperty property);

}

0 comments on commit 5369e4c

Please sign in to comment.