-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix line skipping issue in receive_lines method #4491
Conversation
@yugeeklab Thanks for this fix! I see the intent of this fix as follows.
Surely, such a fix would allow us to limit memory consumption by the This PR would be effective to some extent, however I believe the problem of memory consumption will remain. Are these my understandings correct? |
Hi, @daipom I've just published an issue #4491 for more information.
When max_line_size isn't set, FIFO's @buffer can grow indefinitely. Or if max_line_size has large value, FIFO's buffer will be limited, but there's still a possibility of fluentd experiencing slowdowns. Summary: as-is: max_line_size helps you avoid buffer overflow configuring via buffer section. If you have any suggestions, such as the fifo_buffer_size parameter or any other ideas, please feel free to discuss them with me. Thank you for your review! |
Thanks so much!
Now I understand!
This fix clears
Thanks! |
Thank you for your comment!! @daipom Please let me know if there's any feedback on my code or idea. I'll review and accept your feedback as soon as possible. Thank you. |
@yugeeklab The CI issue has been resolved. Sorry for the trouble. |
7082a95
to
8463d57
Compare
Hi, @daipom Rebase is done!! Thank you for your review!! |
Sorry for waiting. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yugeeklab Thanks for this fix!
This fix basically looks good to me!
I've commented on some minor details (about the following), please check!
- Keeping the same debug log as before
- Improving codes
- Improving tests
@yugeeklab |
Current CI failures have nothing to do with this PR. |
The CI issue has been resolved. |
b7f5859
to
1c5c571
Compare
0d45c3b
to
bf73efe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix!
The following point is a remaining concern.
I saw bf73efe and realized that there is a problem that needs to be solved about the management of
pos
We should not update pos
like this commit (bf73efe).
We should only update pos
at points where recovery is possible.
This means that we should not update pos
until we can be sure that @lines
has been successfully handled by @receive_lines
.
If updating pos
like this commit, some data may be lost when BufferOverflowError
occurs or when Fluentd is forced to stop.
(see #4491 (comment))
So, we need to consider how to manage pos
correctly for this feature.
It needs to be able to continue processing correctly even if Fluentd is forced to stop.
Repeating process to skip long lines would be acceptable.
Data loss or sending corrupted data would be unacceptable.
For this feature, we need to take care of @was_long_line
in particular.
We need to make sure that the restart of Fluentd does not cause a subsequent incomplete log to be sent.
Oh, sorry, it was wrong. fluentd/lib/fluent/plugin/in_tail.rb Lines 1230 to 1232 in bf73efe
So, we need to consider only the following points.
|
I think we should change fluentd/lib/fluent/plugin/in_tail.rb Lines 1087 to 1089 in bf73efe
It is used in the following fluentd/lib/fluent/plugin/in_tail.rb Lines 1230 to 1241 in bf73efe
fluentd/lib/fluent/plugin/in_tail.rb Lines 1246 to 1248 in bf73efe
The In the following case (
However, that data size should be considered for For this, |
@yugeeklab I have fixed the remaining points and pushed them to my tmp branch (the following 3 commits). The main point is to resolve the issue that is tested on the 'discards a subsequent data in a long line even if restarting occurs between' test in fix to commit the correct pos to continue processing correctly. |
Hi @daipom So, Here is the summary of your code AS-IS: T0-BE: It looks good to me!! Thank you!!! |
@yugeeklab Yes! Thanks for checking it! |
@yugeeklab Sorry, I failed to push. Something wrong happens... |
@yugeeklab Sorry for the trouble. git push -f |
Hi @daipom I recover my origin/master!! Should I also reopen Pull Request? Thank you. |
Of course! Thanks for reopening it! |
Which issue(s) this PR fixes:
Fixes #4494
What this PR does / why we need it:
Before this patch, long lines could cause breakdowns in fluentd, potentially posing a vulnerability. With this patch, max_line_size will be integrated into the FIFO, enabling the system to skip lines exceeding the maximum size before executing receive_lines.
Docs Changes:
Release Note: