Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory usage about LineReader #20497

Closed
wangtao334 opened this issue Aug 7, 2020 · 2 comments
Closed

Memory usage about LineReader #20497

wangtao334 opened this issue Aug 7, 2020 · 2 comments
Labels
enhancement Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@wangtao334
Copy link

wangtao334 commented Aug 7, 2020

Describe the enhancement:
LineReader reads data into a buffer, and then appends the buffer to LineReader's inBuffer, the max memory used by inBuffer depends on the max size of line. but LimitReader will truncate the data at last. So i think the max memory used by inBuffer should be tye max_bytes + harvester_buffer_size and no relation with the max size of line.

Describe a specific use case for the enhancement or feature:
we use filebeat to collect log, but sometimes we got some log files with the max line size over than 200M. sure we try to control memeory usage by setting max_bytes, but we still get OOM. as a tool, we don't want filebeat using a lot of memory.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 7, 2020
@andresrc andresrc added enhancement Team:Services (Deprecated) Label for the former Integrations-Services team labels Aug 9, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 9, 2020
@kvch kvch removed their assignment Oct 7, 2021
@jlind23
Copy link
Collaborator

jlind23 commented Mar 31, 2022

Closing it for now until further activity.

@jlind23 jlind23 closed this as completed Mar 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Development

No branches or pull requests

5 participants