Configuration of the fields available in elasticsearch

[abrasat]

Where/how are configured the fields which are created by the serilog elasticsearch sink, when writing logs to elasticsearch? For instance there is not only a "message" field available, but also a "messageTemplate" field? How could the "messageTemplate" field be removed? Or, more generally speaking, how exactly can an application decide which elasticserach fields will be available and which information does contain each of them?

[abrasat]

Can somebody give me some hints where is defined the default elasticsearch index mapping when writing logs to the serilog elasticsearch sink ? How can the elasticsearch index mapping be influenced by configuring the serilog elsticsearch sink, any examples available?

[mivano]

That is what the template is doing. By default it will create a template which has a mapping for the logevents. You can make changes to that mapping by providing your own mapping data. The mapping will be applied to the index when it matches (on name or type).

[abrasat]

I could not find that default template on elasticsearch after writing some logs with serilog... Or maybe I am missing something. How can I check for the template created by the serilog elasticsearch sink and see its content?

[mivano]

See here: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html#getting
This code (https://github.com/serilog/serilog-sinks-elasticsearch/blob/dev/src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchTemplateProvider.cs) will generate the different kinds (based on version) of templates.

[abrasat]

Thanks for info.
Where exactly does the "templateMatchString" parameter come from? Which is the default/actual value for it, can be customized?
Later edit: I found the template name in the serilog-sinks-elasticsearch code : "serilog-events-template". When is this template created and with which content? The template cannot be found on elasticsearch, as the command "GET /_template/serilog-events-template" returns nothing.

[abrasat]

It looks that the template cannot be created/registered (I am using elasticsearch 7.0.1). When executing the Serilog.Sinks.Elasticsearch.Sample, the following exception is thrown:

2019-06-26T08:50:56.5687231Z Failed to create the template. Elasticsearch.Net.ElasticsearchClientException: Request failed to execute
   at Elasticsearch.Net.Transport`1.HandleElasticsearchClientException(RequestData data, Exception clientException, IElasticsearchResponse response) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 196
   at Elasticsearch.Net.Transport`1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List`1 seenExceptions, TResponse response) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 180
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 97
   at Elasticsearch.Net.ElasticLowLevelClient.IndicesPutTemplateForAll[TResponse](String name, PostData body, PutIndexTemplateRequestParameters requestParameters) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\ElasticLowLevelClient.Generated.cs:line 1725
   at Serilog.Sinks.Elasticsearch.ElasticsearchSinkState.RegisterTemplateIfNeeded() in C:\Code\Playground\ElasticSearch\serilog-sinks-elasticsearch-dev\src\Serilog.Sinks.Elasticsearch\Sinks\ElasticSearch\ElasticsearchSinkState.cs:line 152
[10:50:56 INF] Hello, world!
[10:50:56 DBG] Dividing 10 by 0
[10:50:56 ERR] Something went wrong
System.DivideByZeroException: Attempted to divide by zero.
   at Serilog.Sinks.Elasticsearch.Sample.Program.Main(String[] args) in C:\Code\Playground\ElasticSearch\serilog-sinks-elasticsearch-dev\sample\Serilog.Sinks.Elasticsearch.Sample\Program.cs:line 60
[10:50:56 DBG] Reusing string by True
2019-06-26T08:50:57.1385672Z Failed to store event with template 'Reusing {A} by {B}' into Elasticsearch. Elasticsearch reports for index logstash-2019.06.26 the following: {"type":"mapper_parsing_exception","reason":"failed to parse field [fields.A] of type [long] in document with id 'X277kmsBQ9DALJeZFQI_'","caused_by":{"type":"illegal_argument_exception","reason":"For input string: \"string\""}}
Unable to submit event Reusing {A} by {B}
Press any key to continue...

Later edit: the DebugInformation for the Exception:

# FailureReason: BadResponse while attempting PUT on http://elastic:changeme@localhost:9200/_template/serilog-events-template
# Server indicated deprecations:
- 299 Elasticsearch-7.0.1-e4efcb5 "Deprecated field [template] used, replaced by [index_patterns]"
# Audit trail of this API call:
 - [1] BadResponse: Node: http://elastic:changeme@localhost:9200/ Took: 00:00:00.5456422
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute
   at Elasticsearch.Net.Transport`1.HandleElasticsearchClientException(RequestData data, Exception clientException, IElasticsearchResponse response) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 196
   at Elasticsearch.Net.Transport`1.FinalizeResponse[TResponse](RequestData requestData, IRequestPipeline pipeline, List`1 seenExceptions, TResponse response) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 180
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\Transport\Transport.cs:line 97
   at Elasticsearch.Net.ElasticLowLevelClient.IndicesPutTemplateForAll[TResponse](String name, PostData body, PutIndexTemplateRequestParameters requestParameters) in c:\Projects\elastic\net-master\src\Elasticsearch.Net\ElasticLowLevelClient.Generated.cs:line 1725
   at Serilog.Sinks.Elasticsearch.ElasticsearchSinkState.RegisterTemplateIfNeeded() in C:\Code\Playground\ElasticSearch\serilog-sinks-elasticsearch-dev\src\Serilog.Sinks.Elasticsearch\Sinks\ElasticSearch\ElasticsearchSinkState.cs:line 152

Same exception occurs with Elasticsearch 6.2.4.

[mivano]

Old issue, cleaning up. Still facing this issue, feel free to reopen.