This is a plugin for Logstash.
This plugin provides infrastructure to automatically consume logs from Aliyun Log Service .
This is an example of logstash config:
input {
logservice{
endpoint => "your project endpoint"
access_id => "your access id"
access_key => "your access key"
project => "your project name"
logstore => "your logstore name"
consumer_group => "consumer group name"
consumer_name => "consumer name"
position => "end"
checkpoint_second => 30
include_meta => true
consumer_name_with_ip => true
}
}
output {
stdout {}
}
This plugin supports the following configuration options
Configuration | Type | Required | Comments |
---|---|---|---|
endpoint | string | true | Your project endpoint |
access_id | string | true | Your access id |
access_key | string | true | Your access key |
project | string | true | Your project name |
logstore | string | true | Your consumer_name name |
consumer_group | string | true | Consumer group name |
consumer_name | string | true | Consumer name,The consumer name in the same consumer group must not be repeated, otherwise undefined behavior will occur. |
position | string | true | Position to consume. Options are begin , end and yyyy-MM-dd HH:mm:ss |
checkpoint_second | number | false | Time to checkpoint,default is 30 |
include_meta | boolean | false | Whether the meta is included,default is true |
consumer_name_with_ip | boolean | false | Whether the consumer name has ip,default is true,Must be set to true under distributed collaborative consumption |
you can build the gem and install it using:
- Install the plugin from the Logstash home
logstash-plugin install logstash-input-sls
- Start Logstash
logstash -f logstash-sample.conf
- cpu : Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz,4 Core
- memory : 8GB
- env : Linux
Data flow(MB/S) | CPU(%) | MEM(GB) |
---|---|---|
32 | 170.3 | 1.3 |
16 | 83.3 | 1.3 |
8 | 41.5 | 1.3 |
4 | 21.0 | 1.3 |
2 | 11.3 | 1.3 |