Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5.0.0 fix memory leak #115

Merged
merged 16 commits into from
Apr 13, 2021
Merged

5.0.0 fix memory leak #115

merged 16 commits into from
Apr 13, 2021

Conversation

mccraigmccraig
Copy link
Collaborator

yapster APIs and streaming processes were leaking - OOMEing over a couple of days (of low traffic on internal dev instances)

investigation showed lots of our EntityInstance db records referenced from the global downstream graph metadata in manifold.stream.graph/handle->downstreams

this was something of a mysterious issue, in that usually stream-related leaks are because some streams are not being closed, and things stop working because streams are not closed and stream/reduces don't terminate - but in this case streams were apparently being closed and correct operation of the system was observed

two issues were found which are fixed in this PR

  1. qbits.alia.manifold/handle-page-completion-stage recurses to fetch the next page. since manifold often uses the caller thread for callbacks, this could blow the stack, so it has been changed to use deferred/loop
  2. qbits.alia.manifold/execute-stream was using a (stream/transform (mapcat identity) <page-stream>) to concatenate pages from the page-stream on to a record-stream. this appears to be responsible for the leak (replacement of the stream/transform has fixed the leak)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant