Add Offset to `libbeat/reader.Message` #39873

AndersonQ · 2024-06-12T12:28:46Z

Proposed commit message

libbeat: add Offset to libbeat/reader.Message

This commit introduces the Offset property to libbeat/reader.Message, which stores the total number of bytes read and discarded before generating the message. The Offset field allows inputs to accurately determine how much data has been read up to the message, calculated as Message.Bytes + Message.Offset.

With this new Offset field, the filestream input correctly updates its state to account for data read but discarded by the include_message parser.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
-~~ [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None

How to test this PR locally

Follow the instructions on #39653

Related issues

Closes Filestream include_message do not correctly track the offset of a file #39653

mergify · 2024-06-12T12:29:29Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @AndersonQ? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

Offset contains the data read before the message itself

…rsonQ/beats into 39653-filestream-include-msg-parser

elasticmachine · 2024-06-17T16:28:35Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

pierrehilbert · 2024-06-18T09:18:18Z

@AndersonQ could you please take care of the linter issues?

cmacknz · 2024-06-18T12:57:11Z

CHANGELOG.next.asciidoc

@@ -94,6 +94,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Fix cache processor expiries infinite growth when large a large TTL is used and recurring keys are cached. {pull}38561[38561]
 - Fix parsing of RFC 3164 process IDs in syslog processor. {issue}38947[38947] {pull}38982[38982]
 - Rename the field "apache2.module.error" to "apache.module.error" in Apache error visualization. {issue}39480[39480] {pull}39481[39481]
+- Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. {pull}39873[39873] {issue}39653[39653]


You don't need this in the user facing changelog, this is more of a developer only detail.

I agree with Craig, the other item you wrote is enough.

Suggested change

- Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. {pull}39873[39873] {issue}39653[39653]

cmacknz · 2024-06-18T12:59:42Z

filebeat/input/filestream/environment_test.go

-		}
-		time.Sleep(10 * time.Millisecond)
-	}
+		fmt.Fprintf(msg, "unexpected number of events; expected: %d, actual: %d",


Suggested change

fmt.Fprintf(msg, "unexpected number of events; expected: %d, actual: %d",

fmt.Fprintf(msg, "unexpected number of events; expected: %d, actual: %d\n",

belimawr · 2024-06-18T14:11:09Z

CHANGELOG.next.asciidoc

@@ -94,6 +94,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
 - Fix cache processor expiries infinite growth when large a large TTL is used and recurring keys are cached. {pull}38561[38561]
 - Fix parsing of RFC 3164 process IDs in syslog processor. {issue}38947[38947] {pull}38982[38982]
 - Rename the field "apache2.module.error" to "apache.module.error" in Apache error visualization. {issue}39480[39480] {pull}39481[39481]
+- Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. {pull}39873[39873] {issue}39653[39653]


I agree with Craig, the other item you wrote is enough.

Suggested change

- Add the Offset property to libbeat/reader.Message to store the total number of bytes read and discarded before generating the message. This enables inputs to accurately determine how much data has been read up to the message, using Message.Bytes + Message.Offset. {pull}39873[39873] {issue}39653[39653]

This commit introduces the Offset property to libbeat/reader.Message, which stores the total number of bytes read and discarded before generating the message. The Offset field allows inputs to accurately determine how much data has been read up to the message, calculated as Message.Bytes + Message.Offset. With this new Offset field, the filestream input correctly updates its state to account for data read but discarded by the include_message parser. (cherry picked from commit 535a174) # Conflicts: # filebeat/input/filestream/environment_test.go

* libbeat: add Offset to libbeat/reader.Message (#39873) This commit introduces the Offset property to libbeat/reader.Message, which stores the total number of bytes read and discarded before generating the message. The Offset field allows inputs to accurately determine how much data has been read up to the message, calculated as Message.Bytes + Message.Offset. With this new Offset field, the filestream input correctly updates its state to account for data read but discarded by the include_message parser. (cherry picked from commit 535a174) # Conflicts: # filebeat/input/filestream/environment_test.go * Fixing conflicts --------- Co-authored-by: Anderson Queiroz <anderson.queiroz@elastic.co> Co-authored-by: Pierre HILBERT <pierre.hilbert@elastic.co>

wip

4d2f713

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 12, 2024

mergify bot assigned AndersonQ Jun 12, 2024

AndersonQ added 4 commits June 12, 2024 16:46

add Offset to message

a5db930

Offset contains the data read before the message itself

account for the offset of discarded messages

4b94cc4

add test for include_message parser

c388c96

avoid tests hanging forever

0725d66

AndersonQ added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team bugfix backport-v8.14.0 Automated backport with mergify labels Jun 13, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 13, 2024

AndersonQ added 2 commits June 17, 2024 18:01

fix double import

eef16ea

Merge branch 'main' into 39653-filestream-include-msg-parser

3b29d3b

AndersonQ changed the title ~~wip~~ Add Offset to libbeat/reader.Message Jun 17, 2024

AndersonQ added 2 commits June 17, 2024 18:28

add changelog

16fa60b

Merge branch '39653-filestream-include-msg-parser' of github.com:Ande…

8789e72

…rsonQ/beats into 39653-filestream-include-msg-parser

AndersonQ marked this pull request as ready for review June 17, 2024 16:28

AndersonQ requested a review from a team as a code owner June 17, 2024 16:28

AndersonQ requested review from belimawr and fearful-symmetry June 17, 2024 16:28

pierrehilbert requested review from rdner and VihasMakwana and removed request for fearful-symmetry June 17, 2024 16:30

VihasMakwana approved these changes Jun 18, 2024

View reviewed changes

cmacknz reviewed Jun 18, 2024

View reviewed changes

belimawr requested changes Jun 18, 2024

View reviewed changes

AndersonQ added 2 commits June 19, 2024 11:01

fix changelog

e8b9edd

fix linter issues

fed6f10

pierrehilbert requested a review from belimawr June 19, 2024 13:12

belimawr approved these changes Jun 20, 2024

View reviewed changes

pierrehilbert and others added 2 commits June 20, 2024 16:35

Merge branch 'main' into 39653-filestream-include-msg-parser

9582100

Merge branch 'main' into 39653-filestream-include-msg-parser

4bcb18a

AndersonQ enabled auto-merge (squash) June 21, 2024 13:28

AndersonQ merged commit 535a174 into elastic:main Jun 21, 2024
121 checks passed

mergify bot mentioned this pull request Jun 21, 2024

[8.14](backport #39873) Add Offset to libbeat/reader.Message #39992

Merged

4 tasks

AndersonQ deleted the 39653-filestream-include-msg-parser branch July 1, 2024 13:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Offset to `libbeat/reader.Message` #39873

Add Offset to `libbeat/reader.Message` #39873

AndersonQ commented Jun 12, 2024 •

edited

Loading

mergify bot commented Jun 12, 2024

elasticmachine commented Jun 17, 2024

pierrehilbert commented Jun 18, 2024

cmacknz Jun 18, 2024

belimawr Jun 18, 2024

cmacknz Jun 18, 2024

belimawr Jun 18, 2024

	fmt.Fprintf(msg, "unexpected number of events; expected: %d, actual: %d",
	fmt.Fprintf(msg, "unexpected number of events; expected: %d, actual: %d\n",

Add Offset to libbeat/reader.Message #39873

Add Offset to libbeat/reader.Message #39873

Conversation

AndersonQ commented Jun 12, 2024 • edited Loading

Proposed commit message

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

mergify bot commented Jun 12, 2024

elasticmachine commented Jun 17, 2024

pierrehilbert commented Jun 18, 2024

cmacknz Jun 18, 2024

Choose a reason for hiding this comment

belimawr Jun 18, 2024

Choose a reason for hiding this comment

cmacknz Jun 18, 2024

Choose a reason for hiding this comment

belimawr Jun 18, 2024

Choose a reason for hiding this comment

Add Offset to `libbeat/reader.Message` #39873

Add Offset to `libbeat/reader.Message` #39873

AndersonQ commented Jun 12, 2024 •

edited

Loading