Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More memory optimizations #248

Merged
merged 5 commits into from Mar 19, 2020
Merged

Conversation

JelleAalbers
Copy link
Member

@JelleAalbers JelleAalbers commented Mar 15, 2020

This:

  • Makes multi-output plugins respect lazy mode (Lazy mailbox #241). Currently divide_outputs doesn't account for lazy mode, so a multi-output plugin could still run away and fetch as much as it likes (potentially blowing up since in lazy mode, the max_messages limits are removed)
  • For plugins with multiple inputs, the input controlling the chunk size ('pacemaker') is chosen to be the input that produces the smallest first chunk. Currently we just take the first dependency listed in depends_on.
  • OverlapWindowPlugin no longer keeps one chunk back, since it is now OK to yield empty chunks.

These together reduce strax's memory usage in single-core mode by about 15-20%, on top of the similar reduction from #241. Lazy mode now uses about ~30% less memory than non-lazy mode.

The only downside of lazy mode, as noted in #241, is that processing slows down significantly (up to 50% or so) when using single-core lazy mode on a multi-core job/machine. The lazy processing topology assumes you cannot do things in parallel, so when you actually can, it is less efficient.

The multicore processing performance and memory use seem unaffected, although the last two changes listed above do affect multicore processing.

@JelleAalbers JelleAalbers merged commit f0dd576 into AxFoundation:master Mar 19, 2020
@JelleAalbers JelleAalbers deleted the more_lazy branch March 19, 2020 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant