Skip to content
Module to Log log4net Messages to ElasticSearch
C# PowerShell Shell
Branch: master
Clone or download
Pull request Compare This branch is 25 commits behind urielha:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
scripts
src
.gitignore
.travis.yml
LICENSE
README.md
log4stash.nuspec
pack-nuget.bat
readme.txt

README.md

log4stash

log4stash is a log4net appender to log messages to the ElasticSearch document database. ElasticSearch offers robust full-text search engine and analyzation so that errors and messages can be indexed quickly and searched easily.

log4stash provide few logging filters similar to the filters on logstash.

The origin of log4stash is @jptoto's log4net.ElasticSearch repository.

Features:

  • Supports .NET 3.5+
  • Easy installation and setup via Nuget
  • Ability to analyze the log event before sending it to elasticsearch using built-in filters and custom filters similar to logstash.

Breaking Changes:

  • The definition of IElasticAppenderFilter has been changed, PrepareEvent has only one parameter and PrepareConfiguration's parameter type has changed to IElasticsearchClient.

Filters:

  • Add - add new key and value to the event.
  • Remove - remove key from the event.
  • Rename - rename key to another name.
  • Kv - analyze value (default is to analyze the 'Message' value) and export key-value pairs using regex (similar to logstash's kv filter).
  • Grok - analyze value (default is 'Message') using custom regex and saved patterns (similar to logstash's grok filter).
  • ConvertToArray - split raw string to an array by given seperators.

Custom filter:

To add your own filters you just need to implement the interface IElasticAppenderFilter on your assembly and configure it on the log4net configuration file.

Issues:

I do my best to reply to issues or questions ASAP. Please use the ISSUES page to submit questions or errors.

Configuration Examples:

Almost all the parameters are optional, to see the default values check the c'tor of the appender and the c'tor of every filter. You can also set any public property in the appender/filter which didn't appear in the example.

Simple configuration:
<appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4stash">
    <Server>localhost</Server>
    <Port>9200</Port>
    <ElasticFilters>
      <!-- example of using filter with default parameters -->
      <kv /> 
    </ElasticFilters>
</appender>
(Almost) Full configuration:
<appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4stash">
    <Server>localhost</Server>
    <Port>9200</Port>
    <IndexName>log_test_%{+yyyy-MM-dd}</IndexName>
    <IndexType>LogEvent</IndexType>
    <Bulksize>2000</Bulksize>
    <BulkIdleTimeout>10000</BulkIdleTimeout>
    <IndexAsync>False</IndexAsync>

    <!-- for more information read about log4net.Core.FixFlags -->
    <FixedFields>Partial</FixedFields>
    
    <Template>
      <Name>templateName</Name>
      <FileName>path2template.json</FileName>
    </Template>

    <!-- all filters goes in ElasticFilters tag -->
    <ElasticFilters>
      <Add>
        <Key>@type</Key>
        <Value>Special</Value>
      </Add>

      <!-- using the @type value from the previous filter -->
      <Add>
        <Key>SmartValue</Key>
        <Value>the type is %{@type}</Value>
      </Add>

      <Remove>
        <Key>@type</Key>
      </Remove>

      <!-- you can load custom filters like I do here -->
      <Filter type="log4net.ElasticSearch.Filters.RenameKeyFilter, log4stash">
        <Key>SmartValue</Key>
        <RenameTo>SmartValue2</RenameTo>
      </Filter>

      <!-- kv and grok filters similar to logstash's filters -->
      <Kv>
      	<SourceKey>Message</SourceKey>
      	<ValueSplit>:=</ValueSplit>
      	<FieldSplit> ,</FieldSplit>
      </kv>

      <Grok>
        <SourceKey>Message</SourceKey>
        <Pattern>the message is %{WORD:Message} and guid %{UUID:the_guid}</Pattern>
        <Overwrite>true</Overwrite>
      </Grok>
    </ElasticFilters>
</appender>

Note that the filters got called by the order they appeared in the config (as shown in the example).

License:

MIT License

Thanks:

Thanks to @jptoto for the idea and the first working ElasticAppender. Many thanks to @mpdreamz and the team for their great work on the NEST library! The inspiration to the filters and style had taken from elasticsearch/logstash project.

Build status:

Status Provider
Build status Mono CI provided by travis-ci
Build Status Windows CI provided by AppVeyor (without tests for now)
You can’t perform that action at this time.