Syslog input plugin #4181

leodido · 2018-05-22T16:57:44Z

This PR introduces a syslog input service plugin (acting as a syslog receiver) supporting:

TLS (or TCP) as mandated by RFC5425
UDP as mandated by RFC5426

Thanks a lot to @goller for his feedback, help, and support.

Required for all PRs:

Signed CLA.
Associated README.md updated.
Has appropriate unit tests.

Introducing an input service plugin acting like a syslog receiver as specified by RFC5425 (TLS/TCP) or RFC5426 (UDP).

danielnelson · 2018-05-22T18:25:53Z

plugins/inputs/syslog/syslog.go

+}
+
+var sampleConfig = `
+    ## Specify an ip or hostname with port - eg., localhost:6514, 10.0.0.1:6514


Use 2 space indent, comment anything out that has a default and doesn't need to be set (read_timeout)

danielnelson · 2018-05-22T18:28:11Z

plugins/inputs/syslog/syslog.go

+		}
+		s.Closer = l
+		s.tcpListener = l
+		if tlsConfig, _ := s.TLSConfig(); tlsConfig != nil {


If there is an error return it, Telegraf will refuse to start but that is the desired behavior here.

danielnelson · 2018-05-22T18:32:17Z

plugins/inputs/syslog/syslog.go

+    ## Address and port to host the syslog receiver.
+    ## If no server is specified, then localhost is used as the host.
+    ## If no port is specified, 6514 is used (RFC5425#section-4.1).
+	server = ":6514"


Use url style: tcp://:6514 mostly because it matches socket_listener.

danielnelson · 2018-05-22T18:34:06Z

plugins/inputs/syslog/syslog.go

+		n, _, err := s.udpListener.ReadFrom(b)
+		if err != nil {
+			if !strings.HasSuffix(err.Error(), ": use of closed network connection") {
+				log.Println(err)


Don't log since the error is being added to the accumulator, which will also log.

danielnelson · 2018-05-22T18:35:14Z

plugins/inputs/syslog/syslog.go

+		}
+
+		p := rfc5424.NewParser()
+		mex, err := p.Parse(b[:n], &s.BestEffort)


Changed to message.

danielnelson · 2018-05-22T20:30:53Z

plugins/inputs/syslog/README.md

+Other available configurations are:
+
+- `keep_alive_period`, `max_connections` for stream sockets
+- `best_effort` to tell the parser to work until it is able to do and extract partial but valid info


Please add more details on how this works or add link to go-syslog repo.

danielnelson · 2018-05-22T20:36:18Z

plugins/inputs/syslog/README.md

+
+The fields/tags which name is in **bold** will always be present when a valid Syslog message has been received.
+
+### Syslog transport sender


Title this section something like RSYSLOG Integration

danielnelson · 2018-05-22T20:37:23Z

plugins/inputs/syslog/README.md

+
+```toml
+[[inputs.syslog]]
+  address = ":6514"


Just have a single configuration, no need to show it in multiple potential configurations.

Ok, what configuration should we show? The minimal one?

Done showing the config for TLS.

Run telegraf -usage syslog and use the output.

danielnelson · 2018-05-22T20:40:32Z

plugins/inputs/syslog/syslog_test.go

+	require.Error(t, receiver.Start(&testutil.Accumulator{}))
+}
+
+func getRandomString(n int) string {


It would be preferable to have non random strings in unittests, to increase determinism. Programmatic creation is fine of course.

There's a pseudo-random source within this method using now as seed.
If you prefer we can stuck to a preconfigured seed value (so not now) or we can completely remove this and simply generate a string with length 7681.

I would probably just generate a string with the desired length.

danielnelson · 2018-05-22T20:47:42Z

plugins/inputs/syslog/rfc5426_test.go

+	}
+}
+
+func TestUDPInBestEffortMode(t *testing.T) {


The way these tests are structured make it more likely that they will fail due to a previous test case, because you are reusing the unit under test. I would create all tested code new for each test, unless you are specifically testing multiple connections.

Should be better now.

goller · 2018-05-22T21:14:07Z

plugins/inputs/syslog/syslog.go

+	## Otherwise forced to the default.
+	# protocol = "tcp"
+
+    ## TLS Config


This needs to be indented to the same level as the rest of the config

goller · 2018-05-22T21:22:04Z

plugins/inputs/syslog/syslog.go

+		conn, err := s.tcpListener.Accept()
+		if err != nil {
+			if !strings.HasSuffix(err.Error(), ": use of closed network connection") {
+				log.Println(err)


Remove this log statement

…end SDPARAMs

…nput plugin

danielnelson · 2018-05-24T21:52:21Z

plugins/inputs/syslog/syslog.go

+	flds["facility_code"] = int(*msg.Facility())
+
+	if msg.Timestamp() != nil {
+		flds["timestamp"] = *msg.Timestamp()


I think this doesn't work because this is a time.Time and a field has to be one of these types. If this returns nil then the field is silently dropped!

telegraf/metric/metric.go

Lines 251 to 286 in ce3b367

func convertField(v interface{}) interface{} {

switch v := v.(type) {

case float64:

return v

case int64:

return v

case string:

return v

case bool:

return v

case int:

return int64(v)

case uint:

return uint64(v)

case uint64:

return uint64(v)

case []byte:

return string(v)

case int32:

return int64(v)

case int16:

return int64(v)

case int8:

return int64(v)

case uint32:

return uint64(v)

case uint16:

return uint64(v)

case uint8:

return uint64(v)

case float32:

return float64(v)

default:

return nil

}

}

Probably should encode as an integer in nanosecond precision

…recision

leodido added 7 commits May 15, 2018 18:22

New: Syslog input plugin

c1b7de8

Introducing an input service plugin acting like a syslog receiver as specified by RFC5425 (TLS/TCP) or RFC5426 (UDP).

New: Setup test suite utilities for syslog input plugin

c802954

Docs: README of syslog input plugin

1507ea4

Update: Testing syslog input plugin listening on TLS/TCP

f17171e

Update: Testing syslog input plugin listening on UDP

1bb3907

Deps: Syslog library

e4f70f4

Update: List of input plugins

a805e38

leodido added the new plugin label May 22, 2018

leodido self-assigned this May 22, 2018

leodido requested review from goller and danielnelson May 22, 2018 17:00

danielnelson reviewed May 22, 2018

View reviewed changes

goller reviewed May 22, 2018

View reviewed changes

leodido added 6 commits May 23, 2018 15:58

Update: Refinements on the syslog input plugin

b2647ed

Update: Improvements on test suite of syslog input plugin

1c27f07

Update: Syslog input handling of severity and facility values

09f869b

New: Option to let the user choose how the syslog plugin have to prep…

dd6c6c0

…end SDPARAMs

Update: Syslog address option must contain protocol now

6ca6e91

New: Increment reception time when equal to the last one for syslog i…

8da1f80

…nput plugin

leodido force-pushed the feature/syslog-plugin branch from e49fa4f to 8da1f80 Compare May 24, 2018 18:55

danielnelson reviewed May 24, 2018

View reviewed changes

leodido added 3 commits May 25, 2018 12:29

Update: Syslog input plugin stores timestamp as integers in nanosec p…

e0e9d40

…recision

New: Testing sockets for syslog input plugin

716da76

Update: Test avoiding race conditions for RFC5426 (syslog input plugin)

625d18b

leodido force-pushed the feature/syslog-plugin branch from 49a52b5 to 625d18b Compare May 25, 2018 15:24

danielnelson added this to the 1.7.0 milestone May 25, 2018

danielnelson merged commit b789845 into master May 25, 2018

danielnelson deleted the feature/syslog-plugin branch May 25, 2018 18:40

maxunt pushed a commit that referenced this pull request Jun 26, 2018

Add syslog input plugin (#4181)

54c59f8

otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019

Add syslog input plugin (influxdata#4181)

32edb48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Syslog input plugin #4181

Syslog input plugin #4181

leodido commented May 22, 2018

danielnelson May 22, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 24, 2018

danielnelson May 22, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 23, 2018

leodido May 23, 2018

danielnelson May 23, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 23, 2018

danielnelson May 23, 2018

leodido May 23, 2018

danielnelson May 22, 2018

leodido May 23, 2018

goller May 22, 2018

leodido May 23, 2018

goller May 22, 2018

leodido May 23, 2018

danielnelson May 24, 2018

danielnelson May 24, 2018

leodido May 25, 2018


		The fields/tags which name is in bold will always be present when a valid Syslog message has been received.

		### Syslog transport sender

	func convertField(v interface{}) interface{} {
	switch v := v.(type) {
	case float64:
	return v
	case int64:
	return v
	case string:
	return v
	case bool:
	return v
	case int:
	return int64(v)
	case uint:
	return uint64(v)
	case uint64:
	return uint64(v)
	case []byte:
	return string(v)
	case int32:
	return int64(v)
	case int16:
	return int64(v)
	case int8:
	return int64(v)
	case uint32:
	return uint64(v)
	case uint16:
	return uint64(v)
	case uint8:
	return uint64(v)
	case float32:
	return float64(v)
	default:
	return nil
	}
	}

Syslog input plugin #4181

Syslog input plugin #4181

Conversation

leodido commented May 22, 2018

Required for all PRs:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment