Skip to content

Conversation

@wilhelmagren
Copy link
Contributor

The master collecting all worker buffers is bottleneck. This is because the workers produce buffers faster than the master can write them to disk. When mocking a very large file (>100 million rows) this bottleneck leads to program running out of system memory.

FIX:
add bounded channel, size is kind of arbitrary, and should depend on how much system memory is available.

@codecov-commenter
Copy link

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (7753348) 20.67% compared to head (7aecfef) 20.67%.

Files Patch % Lines
src/mock.rs 0.00% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #27   +/-   ##
=======================================
  Coverage   20.67%   20.67%           
=======================================
  Files           6        6           
  Lines         503      503           
=======================================
  Hits          104      104           
  Misses        399      399           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@wilhelmagren wilhelmagren merged commit 1d743d5 into main Feb 17, 2024
@wilhelmagren wilhelmagren deleted the fix/memory-leak branch February 17, 2024 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants