Skip to content
This repository has been archived by the owner on Aug 31, 2022. It is now read-only.

Add file rolling function in file sink. #45

Merged

Conversation

youyongsong
Copy link
Contributor

Background

I known File Sink is currently mostly used in local development, but we also tend to use it in our production environment with fluentd, because this combination is very stable and reliable.

Currently File Sink lacks the file rolling function, which is very important for the production environment. So I want to add file rolling function to File Sink through this pr without breaking changes.

Usage

file rolling is disabled by default, and can be enabled by adding the following arguments:

- receivers:
  file:
    path: ./test.log
    maxsize: 20    // if ./test.log is larger than 20MB, it will be renamed to ./test-<datetime>.log as a backup file,
    and create a new file named test.log to receive events. It defaults to 100 megabytes.
    maxage: 20     // if backup files ./test-<datetime>.log are created 20 days ago, they will be deleted. The default
    is not to remove old log files based on age.
    maxbackups: 3  // if backup files are more than 3, older backup files will be removed.

the rolling function is powered by lumberjack. Becasue lumberjack.Logger implmented io.WriteCloser interface, so it is easy to replace file *os.file with writer &lumberjack.Logger.

Test Result

run kubernetes-event-exporter with following config:

logLevel: debug
route:
  match:
    - receiver: "file"
receivers:
  - name: "file"
    file:
      path: "./event.log"
      maxsize: 1
      maxbackups: 4

run following script to generate events and output ./event*.log status:

while true
do
  kubectl run nginx --image nginx || kubectl delete deployment nginx
  ls -lh event*.log
  sleep 2
done

the result is:

deployment.apps "nginx" deleted
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:03 event-2020-03-04T10-03-30.271.log
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:15 event-2020-03-04T10-15-52.496.log
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:28 event-2020-03-04T10-28-27.334.log
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:41 event-2020-03-04T10-41-11.316.log
-rw-r--r-- 1 ysyou staff 1023K Mar  4 18:54 event.log
deployment.apps/nginx created
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:15 event-2020-03-04T10-15-52.496.log
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:28 event-2020-03-04T10-28-27.334.log
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:41 event-2020-03-04T10-41-11.316.log
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:54 event-2020-03-04T10-54-11.451.log
-rw-r--r-- 1 ysyou staff 2.0K Mar  4 18:54 event.log

when event.log reaches 1M, it is renamed to event-2020-03-04T10-54-11.451.log, because backup files are more than 4, the oldest one event-2020-03-04T10-03-30.271.log is deleted.

I known File Sink is currently mostly used in local development, but we also tend to use it in our production environment with fluentd, because this combination is very stable and reliable.

Currently File Sink lacks the file rolling function, which is very important for the production environment. So I want to add file rolling function to File Sink through this pr without breaking changes.

file rolling is disabled by default, and can be enabled by adding the following arguments:

```yaml
- receivers:
  file:
    path: ./test.log
    maxsize: 20    // if ./test.log is larger than 20MB, it will be renamed to ./test-<datetime>.log as a backup file,
    and create a new file named test.log to receive events. It defaults to 100 megabytes.
    maxage: 20     // if backup files ./test-<datetime>.log are created 20 days ago, they will be deleted. The default
    is not to remove old log files based on age.
    maxbackups: 3  // if backup files are more than 3, older backup files will be removed.
```

the rolling function is powered by [lumberjack](https://github.com/natefinch/lumberjack). Becasue `lumberjack.Logger` implmented `io.WriteCloser` interface, so it is easy to replace `file *os.file` with `writer &lumberjack.Logger`.

run kubernetes-event-exporter with following config:
```yaml
logLevel: debug
route:
  match:
    - receiver: "file"
receivers:
  - name: "file"
    file:
      path: "./event.log"
      maxsize: 1
      maxbackups: 4
```
run following script to generate events and output `./event*.log` status:
```shell
while true
do
  kubectl run nginx --image nginx || kubectl delete deployment nginx
  ls -lh event*.log
  sleep 2
done
```
the result is:
```
deployment.apps "nginx" deleted
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:03 event-2020-03-04T10-03-30.271.log
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:15 event-2020-03-04T10-15-52.496.log
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:28 event-2020-03-04T10-28-27.334.log
-rw-r--r-- 1 ysyou staff  1.0M Mar  4 18:41 event-2020-03-04T10-41-11.316.log
-rw-r--r-- 1 ysyou staff 1023K Mar  4 18:54 event.log
deployment.apps/nginx created
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:15 event-2020-03-04T10-15-52.496.log
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:28 event-2020-03-04T10-28-27.334.log
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:41 event-2020-03-04T10-41-11.316.log
-rw-r--r-- 1 ysyou staff 1.0M Mar  4 18:54 event-2020-03-04T10-54-11.451.log
-rw-r--r-- 1 ysyou staff 2.0K Mar  4 18:54 event.log
```
when `event.log` reaches 1M, it is renamed to `event-2020-03-04T10-54-11.451.log`, because backup files are more than 4, the oldest one `event-2020-03-04T10-03-30.271.log` is deleted.
@youyongsong
Copy link
Contributor Author

hi @mustafaakin , please help review this pr~

@mustafaakin mustafaakin merged commit 9d8c746 into opsgenie:master Jun 22, 2020
@youyongsong youyongsong deleted the patch/support-rotate-in-file-sink branch June 22, 2020 13:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants