New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rolling appender #9

Closed
mastoj opened this Issue Dec 13, 2013 · 4 comments

Comments

Projects
None yet
2 participants
@mastoj
Contributor

mastoj commented Dec 13, 2013

It would be nice to have an auto cleanup of the logs in elasticsearch, which of course would be optional.

What I am thinking is that you provide a parameter which defines the lenght of a log and how many you should keep. Under the hood an index is created with the defined length and an alias is put on top of those indexes. Example:

Alias: LogEvent
Index1: LogEvent-20131305121212-20131212121212
Index2: LogEvent-20131312121212-20131219121212

And when an event is triggered which should require a new index to be created you create a new index, delete the oldest one and update the alias to include the new one.

If you have a lot of logs this would definitely save space, which might not be a problem :)

@jptoto

This comment has been minimized.

Show comment
Hide comment
@jptoto

jptoto Dec 13, 2013

Owner

Interesting idea! What if we implemented something using ttl instead? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html which would give docs a natural expiration.

Owner

jptoto commented Dec 13, 2013

Interesting idea! What if we implemented something using ttl instead? http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html which would give docs a natural expiration.

@mastoj

This comment has been minimized.

Show comment
Hide comment
@mastoj

mastoj Dec 13, 2013

Contributor

ttl is interesting, but there is other advantages with using the alias -> multiple indexes approach. Using multiple indexes and one alias makes it possible to change the underlying mapping for new indexes compared to the old ones. I'm not sure elasticsearch allow you to change the mapping of an index, which is the drawback if you use just one index and not multiple behind an alias.

Contributor

mastoj commented Dec 13, 2013

ttl is interesting, but there is other advantages with using the alias -> multiple indexes approach. Using multiple indexes and one alias makes it possible to change the underlying mapping for new indexes compared to the old ones. I'm not sure elasticsearch allow you to change the mapping of an index, which is the drawback if you use just one index and not multiple behind an alias.

@jptoto

This comment has been minimized.

Show comment
Hide comment
@jptoto

jptoto Dec 13, 2013

Owner

You can change the mapping of an index but it's a pain in the butt. If there are type conflicts (string to int, for example) ES rejects it.

After thinking about this a bit more, I think we should do both options. I think they're both valuable. We could do an option to alias -> create new indices based on a defined time frame and we can also have a TTL option so that log docs auto expire after X days.

What do you think?

Owner

jptoto commented Dec 13, 2013

You can change the mapping of an index but it's a pain in the butt. If there are type conflicts (string to int, for example) ES rejects it.

After thinking about this a bit more, I think we should do both options. I think they're both valuable. We could do an option to alias -> create new indices based on a defined time frame and we can also have a TTL option so that log docs auto expire after X days.

What do you think?

@jptoto jptoto closed this in eb89319 Jun 21, 2014

@jptoto

This comment has been minimized.

Show comment
Hide comment
@jptoto

jptoto Jun 21, 2014

Owner

Added a rolling appender option in the connection string config. Using 'rolling=true' as a connection string option will cause a new log index to be created daily.

Owner

jptoto commented Jun 21, 2014

Added a rolling appender option in the connection string config. Using 'rolling=true' as a connection string option will cause a new log index to be created daily.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment