[Filebeat] add RFC6587 framing support #23724

leehinman · 2021-01-27T20:52:41Z

What does this PR do?

Adds support for RFC6587 framing.

Adds new config option "framing" for tcp & unix inputs
supported options are "delimiter" & rfc6587
delimiter is current option of newline or custom character(s)
delimiter
rfc6587 adds support for octet counting and non-transparent framing
as described in RFC6587
rfc6587 supports changing of framing on a frame by frame basis
Default is "delimiter"

Why is it important?

Required for Syslog inside TLS
Required for some RFC6587 compliant Syslog clients

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

mage test

Related issues

Closes Adding support for Octet counting in TCP input #23663

andrewkroh

The automatic detection should make this easy to use. I like the approach.

andrewkroh · 2021-01-28T02:41:34Z

filebeat/inputsource/common/streaming/listener.go

-func SplitFunc(lineDelimiter []byte) bufio.SplitFunc {
+// SplitFunc allows to create a `bufio.SplitFunc` based on a framing &
+// delimiter provided.
+func SplitFunc(framing string, lineDelimiter []byte) bufio.SplitFunc {
 	if len(lineDelimiter) == 0 {
 		return nil


Can we make this return an error for this case. Then the message can be constructed in this one place rather than in each input.

andrewkroh · 2021-01-28T02:42:41Z

filebeat/inputsource/common/streaming/listener.go

 	if len(lineDelimiter) == 0 {
 		return nil
 	}
-
 	ld := []byte(lineDelimiter)


Any particular reason not to directly use lineDelimiter?

andrewkroh · 2021-01-28T14:14:40Z

filebeat/input/syslog/config.go

@@ -48,6 +48,7 @@ var defaultConfig = config{
 type syslogTCP struct {
 	tcp.Config    `config:",inline"`
 	LineDelimiter string `config:"line_delimiter" validate:"nonzero"`
+	Framing       string `config:"framing"`


Could this be make into a type FramingType that implements Validate() error such that config unpacking will automatically validate the values.

tried that using type FramingType string and when beats would start I'd get the following panic:

instance/beat.go:167 Failed due to panic. {"panic": "reflect.nameFrom: name too long: ************************************************************************* ....

I think you need to implement https://pkg.go.dev/github.com/elastic/go-ucfg#StringUnpacker. Another option is to make it a type FramingType uint8 and also implement Unpack (that's probably more common in the codebase than having the type be a string).

Thanks for the help. Is there any benefit to doing Validate() if Unpack() does validation?

Nope. If the type itself has Unpack then no need to add another Validate step with similar logic.

- Adds new config option "framing" - supported options are "delimiter" & rfc6587 - delimiter is current option of newline or custom character(s) delimiter - rfc6587 adds support for octet counting and non-transparent framing as described in RFC6587 - rfc6587 supports changing of framing on a frame by frame basis - Default is "delimiter" Closes elastic#23663

elasticmachine · 2021-01-29T19:55:24Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23724 updated
- Start Time: 2021-01-29T19:11:53.743+0000
Duration: 43 min 18 sec
Commit: 1f52de4

Test stats 🧪

Test	Results
Failed	0
Passed	12908
Skipped	2033
Total	14941

Steps errors

Expand to view the steps failures

`x-pack/filebeat-Lint - Install Go/Mage/Python/Docker/Terraform 1.15.7`

Took 0 min 1 sec . View more details on here
Description: .ci/scripts/install-tools.sh

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	12908
Skipped	2033
Total	14941

elasticmachine · 2021-01-29T19:57:08Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh

LGTM

- Adds new config option "framing" - supported options are "delimiter" & rfc6587 - delimiter is current option of newline or custom character(s) delimiter - rfc6587 adds support for octet counting and non-transparent framing as described in RFC6587 - rfc6587 supports changing of framing on a frame by frame basis - Default is "delimiter" Closes elastic#23663 (cherry picked from commit 5cb370e)

- Adds new config option "framing" - supported options are "delimiter" & rfc6587 - delimiter is current option of newline or custom character(s) delimiter - rfc6587 adds support for octet counting and non-transparent framing as described in RFC6587 - rfc6587 supports changing of framing on a frame by frame basis - Default is "delimiter" Closes #23663 (cherry picked from commit 5cb370e)

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 27, 2021

leehinman added enhancement Filebeat Filebeat Team:Security-External Integrations labels Jan 27, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 27, 2021

leehinman force-pushed the 23663_octet_counting branch from 1a789c1 to c623f22 Compare January 28, 2021 04:35

andrewkroh reviewed Jan 28, 2021

View reviewed changes

leehinman force-pushed the 23663_octet_counting branch from 86daa43 to 1f52de4 Compare January 29, 2021 19:11

leehinman marked this pull request as ready for review January 29, 2021 19:57

leehinman added the needs_backport PR is waiting to be backported to other branches. label Jan 29, 2021

andrewkroh approved these changes Feb 1, 2021

View reviewed changes

andrewkroh mentioned this pull request Feb 1, 2021

[Filebeat] Remove TCP input from fortinet fileset #23705

Closed

2 tasks

leehinman merged commit 5cb370e into elastic:master Feb 1, 2021

leehinman deleted the 23663_octet_counting branch February 1, 2021 15:08

leehinman mentioned this pull request Feb 1, 2021

Cherry-pick #23724 to 7.x: [Filebeat] add RFC6587 framing support #23784

Merged

5 tasks

leehinman added v7.12.0 and removed needs_backport PR is waiting to be backported to other branches. labels Feb 1, 2021

mdelapenya mentioned this pull request Feb 3, 2021

[Elastic Agent] Monitoring filebeat and metricbeat not connecting to Agent over GRPC #23833

Closed

jamiehynds mentioned this pull request Apr 23, 2021

[Filebeat][Fortinet Module] Can we add RFC6587 in fortimail fileset when sending the syslog #25253

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] add RFC6587 framing support #23724

[Filebeat] add RFC6587 framing support #23724

leehinman commented Jan 27, 2021 •

edited

andrewkroh left a comment

andrewkroh Jan 28, 2021

andrewkroh Jan 28, 2021

andrewkroh Jan 28, 2021

leehinman Jan 29, 2021

andrewkroh Jan 29, 2021 •

edited

leehinman Jan 29, 2021

andrewkroh Feb 1, 2021

elasticmachine commented Jan 29, 2021

Build stats

Test stats 🧪

`x-pack/filebeat-Lint - Install Go/Mage/Python/Docker/Terraform 1.15.7`

Test stats 🧪

elasticmachine commented Jan 29, 2021

andrewkroh left a comment

[Filebeat] add RFC6587 framing support #23724

[Filebeat] add RFC6587 framing support #23724

Conversation

leehinman commented Jan 27, 2021 • edited

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Jan 28, 2021

Choose a reason for hiding this comment

andrewkroh Jan 28, 2021

Choose a reason for hiding this comment

andrewkroh Jan 28, 2021

Choose a reason for hiding this comment

leehinman Jan 29, 2021

Choose a reason for hiding this comment

andrewkroh Jan 29, 2021 • edited

Choose a reason for hiding this comment

leehinman Jan 29, 2021

Choose a reason for hiding this comment

andrewkroh Feb 1, 2021

Choose a reason for hiding this comment

elasticmachine commented Jan 29, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

x-pack/filebeat-Lint - Install Go/Mage/Python/Docker/Terraform 1.15.7

💚 Flaky test report

Test stats 🧪

elasticmachine commented Jan 29, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

leehinman commented Jan 27, 2021 •

edited

andrewkroh Jan 29, 2021 •

edited

`x-pack/filebeat-Lint - Install Go/Mage/Python/Docker/Terraform 1.15.7`