Filebeat Redis prospector type #4180

ruflin · 2017-05-03T06:18:22Z

This PR adds a new prospector type which reads the slowlog from redis. This slowlog is not in a file but in memory in redis. Because of this filebeat connects to redis and reads out the slowlog. It is important to note that the slow log size is limited in redis that is why after fetching the events from the slowlog filebeat resets the slow log.

Example event looks as following:

{
  "@timestamp": "2017-05-16T06:27:17.000Z",
  "beat": {
    "hostname": "ruflin",
    "name": "ruflin",
    "read_timestamp": "2017-05-16T06:27:19.275Z",
    "version": "6.0.0-alpha2"
  },
  "message": "SET hello world",
  "redis": {
    "slowlog": {
      "args": [
        "world"
      ],
      "cmd": "SET",
      "duration": {
        "us": 11
      },
      "id": 38,
      "key": "hello"
    }
  }
}

All args are combined in the "message" field output for easy retrieval.

elasticmachine · 2017-05-03T06:48:35Z

filebeat/prospector/redis/config.go

+	Password: "",
+}
+
+type Config struct {


[golint] _{reported by reviewdog 🐶}
exported type Config should have comment or be unexported

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/doc.go

@@ -0,0 +1,14 @@
+// Redis package contains prospector and harvester to read the redis slow log


[golint] _{reported by reviewdog 🐶}
package comment should be of the form "Package redis ..."

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/harvester.go

+	rd "github.com/garyburd/redigo/redis"
+)
+
+type Harvester struct {


[golint] _{reported by reviewdog 🐶}
exported type Harvester should have comment or be unexported

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/harvester.go

+	args      []string
+}
+
+func (h *Harvester) Start(conn rd.Conn) error {


[golint] _{reported by reviewdog 🐶}
exported method Harvester.Start should have comment or be unexported

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/prospector.go

+	rd "github.com/garyburd/redigo/redis"
+)
+
+// Stdin is a prospector for stdin


[golint] _{reported by reviewdog 🐶}
comment on exported type Prospector should be of the form "Prospector ..." (with optional leading article)

elasticmachine · 2017-05-03T06:48:36Z

filebeat/prospector/redis/prospector.go

+	config    Config
+}
+
+// NewStdin creates a new stdin prospector


[golint] _{reported by reviewdog 🐶}
comment on exported function NewProspector should be of the form "NewProspector ..."

tsg · 2017-05-03T08:14:58Z

filebeat/_meta/common.full.p2.yml

@@ -215,6 +215,14 @@ filebeat.prospectors:
 # Configuration to use stdin input
 #- input_type: stdin

+#----------------------------- Redis prospector -------------------------------
+#- input_type: redis


I suggest somehow making it more clear that this is strictly about slow logs. People might assume that it's a general Redis input.

That is a tricky one. Thinking more about the prospectors I get the impression we will have prospector with multiple harvester types potentially (similar to what we have in metricbeat). For example in file there could be file.csv or file.xml. What if we not only have slowlog for redis? But TBH I'm not sure yet what we should do here. This could also well be overengineered and we will not need it.

For the moment I think I fix the above with call it Redis slowlog prospector in the docs but still redis as the prospector type.

tsg · 2017-05-03T08:17:13Z

filebeat/prospector/redis/doc.go

+//
+// 	CONFIG SET slowlog-log-slower-than 100
+//
+// This sets the value of the slow log to 100 micro seconds. All queries taking longer will be reported.


I'd use 2s or 5s as an example. 100 micros might affect the Redis performance, so we should be careful.

tsg · 2017-05-03T08:18:32Z

filebeat/prospector/redis/doc.go

+//
+// This sets the value of the slow log to 100 micro seconds. All queries taking longer will be reported.
+//
+// As the slow log is in memory, in can be configured how many items it consists:


small typo: "it can be configured"

tsg · 2017-05-03T08:26:00Z

filebeat/prospector/redis/harvester.go

+
+		data := util.NewData()
+		data.Event = common.MapStr{
+			// TODO: Should we still send the "fetching" timestamp or overwrite it by timestamp?


Hmm, good question. In the end it needs to be the event timestamp, but I see two options:

live it like that here, but replace it in the FB redis module

already set it hear and also set beat.read_timestamp

I decided for option 2 for the moment. The reasons is that I think it most cases that is what people expect in @timestamp here. So also without postprocessing in ingest, the data is already correct.

tsg · 2017-05-03T08:26:26Z

filebeat/prospector/redis/harvester.go

+		if argsLen >= 2 {
+			log.key = args[1]
+		}
+		// TODO: Could args contain confidential data? Should it be optional?


I guess people can use drop_fields if they don't want it.

elasticmachine · 2017-05-15T12:02:00Z

filebeat/harvester/forwarder.go

+	Processors           processors.PluginConfig `config:"processors"`
+}
+
+// updateState updates the prospector state and forwards the event to the spooler


[golint] _{reported by reviewdog 🐶}
comment on exported method Forwarder.Forward should be of the form "Forward ..."

elasticmachine · 2017-05-15T12:02:00Z

filebeat/prospector/redis/harvester.go

+	"github.com/satori/go.uuid"
+)
+
+type Harvester struct {


[golint] _{reported by reviewdog 🐶}
exported type Harvester should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

+	args      []string
+}
+
+func NewHarvester(outlet channel.Outleter, conn rd.Conn) *Harvester {


[golint] _{reported by reviewdog 🐶}
exported function NewHarvester should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

+	}
+}
+
+func (h *Harvester) Start() error {


[golint] _{reported by reviewdog 🐶}
exported method Harvester.Start should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

+	return nil
+}
+
+func (h *Harvester) Stop() {


[golint] _{reported by reviewdog 🐶}
exported method Harvester.Stop should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/harvester.go

+
+}
+
+func (h *Harvester) ID() uuid.UUID {


[golint] _{reported by reviewdog 🐶}
exported method Harvester.ID should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/prospector.go

+	}
+}
+
+func (p *Prospector) Stop() {


[golint] _{reported by reviewdog 🐶}
exported method Prospector.Stop should have comment or be unexported

elasticmachine · 2017-05-15T12:02:01Z

filebeat/prospector/redis/prospector.go

+
+}
+
+func (p *Prospector) Wait() {


[golint] _{reported by reviewdog 🐶}
exported method Prospector.Wait should have comment or be unexported

ruflin · 2017-05-16T06:35:20Z

@tsg Ready for an other look. I think we are at the point now where we have to decide on what to do with it :-)

elasticmachine · 2017-05-18T15:41:27Z

filebeat/harvester/util.go

 )

-// ValidType is a list of all valid input types
+// List of valid input types


[golint] _{reported by reviewdog 🐶}
comment on exported var ValidType should be of the form "ValidType ..."

elasticmachine · 2017-05-18T15:41:27Z

filebeat/prospector/redis/harvester.go

+	}
+}
+
+// Start starts a new redis harvester


[golint] _{reported by reviewdog 🐶}
comment on exported method Harvester.Run should be of the form "Run ..."

ruflin · 2017-05-19T14:29:17Z

@dedemorton So far I only added the minimial options of docs for this = almost nothing. The reason is that this kind of challenges our existing doc structure we have for prospectors. So far the 2 prospectors shared 95% of the config options. With this change, they can be very different. I'm thinking if we should start to organise docs similar to what we do with the modules and have a block for each prospector type. I will need your help here and suggest we will only tackle this after merging the PR.

dedemorton · 2017-05-20T00:51:54Z

@ruflin I remember you mentioned this problem awhile back ago in an email, but I haven't had a chance to follow up with you. What release are we targeting for this PR?

ruflin · 2017-05-22T13:01:09Z

@dedemorton Not sure if I mentioned that before (but could be). It's not urgent. If it happens, somewhere around 6.0

tsg · 2017-05-24T11:05:47Z

filebeat/filebeat.full.yml

@@ -379,6 +379,27 @@ filebeat.prospectors:
 # Configuration to use stdin input
 #- type: stdin

+#------------------------- Redis slowlog prospector ---------------------------
+# Experimental: Config options ofr the redis slow log prospector


small typo on "ofr".

@timestamp

This PR adds a new prospector type which reads the slowlog from redis. This slowlog is not in a file but in memory in redis. Because of this filebeat connects to redis and reads out the slowlog. It is important to note that the slow log size is limited in redis that is why after fetching the events from the slowlog filebeat resets the slow log. Example event looks as following: ``` { "@timestamp": "2017-05-16T06:27:17.000Z", "beat": { "hostname": "ruflin", "name": "ruflin", "read_timestamp": "2017-05-16T06:27:19.275Z", "version": "6.0.0-alpha2" }, "message": "SET hello world", "redis": { "slowlog": { "args": [ "world" ], "cmd": "SET", "duration": { "us": 11 }, "id": 38, "key": "hello" } } } ``` All args are combined in the "message" field output for easy retrieval.

ruflin · 2017-05-26T06:43:39Z

@tsg New version pushed
@dedemorton I created #4398 as a follow up Github issue for the docs.

ruflin added discuss Issue needs further discussion. Filebeat Filebeat in progress Pull request is currently in progress. labels May 3, 2017

elasticmachine reviewed May 3, 2017

View reviewed changes

tsg reviewed May 3, 2017

View reviewed changes

ruflin mentioned this pull request May 10, 2017

Filebeat refactoring / restructuring #4277

Closed

14 tasks

ruflin force-pushed the fb-redis-approach branch 2 times, most recently from 456e2e0 to 9ba6162 Compare May 15, 2017 11:25

elasticmachine reviewed May 15, 2017

View reviewed changes

ruflin force-pushed the fb-redis-approach branch 5 times, most recently from 4a55634 to 36be381 Compare May 16, 2017 06:33

ruflin changed the title ~~[WIP] Filebeat Redis prospector type~~ Filebeat Redis prospector type May 16, 2017

ruflin added review and removed discuss Issue needs further discussion. labels May 16, 2017

ruflin mentioned this pull request May 18, 2017

Including protologbeat in list of community beats #4348

Merged

ruflin force-pushed the fb-redis-approach branch from 36be381 to a9bceb9 Compare May 18, 2017 14:58

elasticmachine reviewed May 18, 2017

View reviewed changes

ruflin mentioned this pull request May 19, 2017

Add varnishlogbeat and varnishstatbeat #4353

Merged

ruflin force-pushed the fb-redis-approach branch 2 times, most recently from e021c74 to 8a8caba Compare May 19, 2017 14:26

ruflin force-pushed the fb-redis-approach branch from 8a8caba to 1a29277 Compare May 22, 2017 13:03

tsg reviewed May 24, 2017

View reviewed changes

ruflin force-pushed the fb-redis-approach branch from 1a29277 to e3e6c2f Compare May 26, 2017 06:41

ruflin mentioned this pull request May 26, 2017

Create Redis Prospector Configuration #4398

Closed

3 tasks

ruflin removed the in progress Pull request is currently in progress. label May 26, 2017

tsg merged commit 4ab7848 into elastic:master May 26, 2017

ruflin deleted the fb-redis-approach branch May 26, 2017 09:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filebeat Redis prospector type #4180

Filebeat Redis prospector type #4180

ruflin commented May 3, 2017 •

edited

Loading

elasticmachine May 3, 2017

elasticmachine May 3, 2017

elasticmachine May 3, 2017

elasticmachine May 3, 2017

elasticmachine May 3, 2017

elasticmachine May 3, 2017

tsg May 3, 2017

ruflin May 16, 2017

tsg May 3, 2017

ruflin May 16, 2017

tsg May 3, 2017

ruflin May 16, 2017

tsg May 3, 2017

ruflin May 16, 2017

tsg May 3, 2017

elasticmachine May 15, 2017

elasticmachine May 15, 2017

elasticmachine May 15, 2017

elasticmachine May 15, 2017

elasticmachine May 15, 2017

elasticmachine May 15, 2017

elasticmachine May 15, 2017

elasticmachine May 15, 2017

ruflin commented May 16, 2017

elasticmachine May 18, 2017

elasticmachine May 18, 2017

ruflin commented May 19, 2017

dedemorton commented May 20, 2017

ruflin commented May 22, 2017

tsg May 24, 2017

ruflin commented May 26, 2017

		@@ -0,0 +1,14 @@
		// Redis package contains prospector and harvester to read the redis slow log

Filebeat Redis prospector type #4180

Filebeat Redis prospector type #4180

Conversation

ruflin commented May 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 19, 2017

dedemorton commented May 20, 2017

ruflin commented May 22, 2017

Choose a reason for hiding this comment

ruflin commented May 26, 2017

ruflin commented May 3, 2017 •

edited

Loading