Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] offset not correctly recorded when max_bytes is exceeded in linereader #28317

Closed
leehinman opened this issue Oct 7, 2021 · 1 comment · Fixed by #28352
Closed

[Filebeat] offset not correctly recorded when max_bytes is exceeded in linereader #28317

leehinman opened this issue Oct 7, 2021 · 1 comment · Fixed by #28352
Assignees
@leehinman
Labels
bug Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label

Comments

@leehinman
Copy link
Contributor

Simple log file:

one
X(repeated 16384 times)
two

log input with max_bytes: 128 results in skipping the X...X line, but the offset for the two message is 4, it should be 16,389. Also in the data/registry/filebeat/log.json file the offset is recorded as 8.

This results in the harvester starting from 8, and resending the two message, but now with offset 8, and then the registry is updated to offset of 12. This results in resending the two message every 5 minutes until the registry catches up to the correct offset.

The LineReader should skip the data when max_bytes is exceeded but it should keep track of the correct offset.
@leehinman leehinman added the bug label Oct 7, 2021
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 7, 2021
@botelastic
Copy link

botelastic bot commented Oct 7, 2021

This issue doesn't have a Team:<team> label.

@leehinman leehinman changed the title offset not correctly recorded when max_bytes is exceeded in linereader [Filebeat] offset not correctly recorded when max_bytes is exceeded in linereader Oct 7, 2021
@leehinman leehinman added the Filebeat Filebeat label Oct 7, 2021
@leehinman leehinman self-assigned this Oct 11, 2021
@leehinman leehinman mentioned this issue Oct 11, 2021
Merged
3 tasks
leehinman added a commit to leehinman/beats that referenced this issue Oct 12, 2021
@leehinman
[Filebeat] keep track of bytes read when max_bytes exceeded
b44be24 
Closes elastic#28317
leehinman added a commit that referenced this issue Oct 13, 2021
@leehinman
[Filebeat] keep track of bytes read when max_bytes exceeded (#28352)
dad462c 
* [Filebeat] keep track of bytes read when max_bytes exceeded in LineReader

Closes #28317
mergify bot pushed a commit that referenced this issue Oct 13, 2021
@leehinman
[Filebeat] keep track of bytes read when max_bytes exceeded (#28352)
3fa79e0 
* [Filebeat] keep track of bytes read when max_bytes exceeded in LineReader

Closes #28317

(cherry picked from commit dad462c)
leehinman added a commit that referenced this issue Oct 13, 2021
@mergify @leehinman
[Filebeat] keep track of bytes read when max_bytes exceeded (#28352) (#…
cef5c6c 
…28379)

* [Filebeat] keep track of bytes read when max_bytes exceeded in LineReader

Closes #28317

(cherry picked from commit dad462c)

Co-authored-by: Lee E Hinman <57081003+leehinman@users.noreply.github.com>
Icedroid pushed a commit to Icedroid/beats that referenced this issue Nov 1, 2021
@leehinman
[Filebeat] keep track of bytes read when max_bytes exceeded (elastic#…
fc05b81 
…28352)

* [Filebeat] keep track of bytes read when max_bytes exceeded in LineReader

Closes elastic#28317
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leehinman leehinman

Labels
bug Filebeat Filebeat needs_team Indicates that the issue/PR needs a Team:* label
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Filebeat] keep track of bytes read when max_bytes exceeded leehinman/beats
1 participant
@leehinman