adding custom mappings

[thaarbach]

@rfoltyns
Hi Rafal
It would be great if you could also specify a path to a custom index template file. E.g. the field timeMillis is a long and it will indexed as long. In this case you can't use this field as a time filter in Kibana. Furthermore, you can not specify any other optimizations, e. g. dynamic templates.
Example index-template.json:

{
	"template": "log4j2*",
	"settings": {
		"index": {
			"refresh_interval": "5s"
		}
	},
	"mappings": {
		"_default_": {
			"dynamic_templates": [
				{
					"strings": {
						"mapping": {
							"type": "keyword"
						},
						"match_mapping_type": "string",
						"match": "*"
					}
				}
			],
			"_all": {
				"enabled": false
			},
			"properties": {
				"loggerName": {
					"type": "text",
					"fields": {
						"keyword": {
							"ignore_above": 256,
							"type": "keyword"
						}
					}
				},
				"message": {
					"type": "text",
					"fields": {
						"keyword": {
							"ignore_above": 256,
							"type": "keyword"
						}
					}
				},
				"timeMillis": {
					"type": "date",
					"format": "epoch_millis"
				}
			}
		}
	}
}

The template should be sent to ES during or after initializing the appender.
Futher informations here Index Templates

[rfoltyns]

Ok, I need to do some experiments here since we have a couple of options:

1. File path - it makes sense to handle absolute paths, relative paths and classpath. I think it should be injectable via env property as well.
2. Interface for user-provided template loader - fully qualified class name of the impl would be required in the config. That'd give maximum flexibility to the user. May be loaded via SPI as well, but that'd make it global for all appenders.

For a start I can implement absolute path and classpath handling just so you can make progress with your work, but do you think it makes sense to handle other cases?

[thaarbach]

Hi Rafal,
IMHO option 1 would be grate and enough. It's only necessary for Elasticsearch and does not make sense to make it globally accessible to other appenders. I don't know, if there is an other use case with could benifit from a template mechanism.
Btw. the index template below is testet on our ES cluster. This could be used as a default index template in the class path of your appenders.
Thx, for your fast anwser

[thaarbach]

Hi Rafal,
have an other Idea. Is it possible to implement sth. like that?

		<Elasticsearch name="elasticsearch">
			<JsonLayout compact="true" properties="true" />
			<AsyncBatchDelivery indexName="log4j2" deliveryInterval="5000" batchSize="500">
				<IndexTemplate>
					{
						"template": "log4j2*",
						"settings": {
						"index": {
							"refresh_interval": "5s"
						}
					},
					"mappings": {
						...
						...
					}
				</IndexTemplate>
				<JestHttp serverUris="http://localhost:9200" />
			</AsyncBatchDelivery>
		</Elasticsearch>

<IndexTemplate> could also offer the options to declare a classpath, filepath etc. E.g.
<IndexTemplate path="classpath:template.json">
What do you think?

[rfoltyns]

I'm not sure whether Log4j config parser would handle the value passed like that, but <IndexTemplate path="classpath:template.json"> is exactly how I planned to add it :)

[thaarbach]

The idea was, to handle it in the same way as Scripts without evaluate it. Maybe it must be surrounded with <![CDATA[]]. In this case the Config Parser should not complain about it and the value is interpreted as a string.

[rfoltyns]

Ok, that might work. But it looks a bit too messy for my taste. File-based approach seems like a way to go here.

[thaarbach]

Yes, it looks really a little bit messy, but you doesn't have spread the configurations arround. Classpath or filepath would be good enought. I just thought about it :)

[rfoltyns]

@thaarbach I've added index template loading via:

1. absolute path and classpath
2. raw value in XML config - if one can cope with it's "noisiness", then why not? ;)
   Thank you for this feature request! :)