Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

src: plug memory leaks #2352

Merged
merged 4 commits into from Aug 13, 2015
Merged

Conversation

bnoordhuis
Copy link
Member

In a few places dynamic memory was passed to the Buffer::New() overload
that makes a copy of the input, not the one that takes ownership.

This commit is a band-aid to fix the memory leaks. Longer term, we
should look into using C++11 move semantics more effectively.

R=@indutny, refs #2308.

I think there is another memory leak in TLSWrap::OnReadSelf() but parallel/test-https-drain starts failing erratically when I fix it... le sigh.

CI: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/72/

@mscdex mscdex added c++ Issues and PRs that require attention from people who are familiar with C++. buffer Issues and PRs related to the buffer subsystem. labels Aug 11, 2015
@indutny
Copy link
Member

indutny commented Aug 11, 2015

Argh, @trevnorris !

@@ -223,7 +223,8 @@ void StreamWrap::OnReadImpl(ssize_t nread,
CHECK_EQ(pending, UV_UNKNOWN_HANDLE);
}

Local<Object> obj = Buffer::New(env, base, nread).ToLocalChecked();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, this one takes the ownership of base. You should not really fix it.

@indutny
Copy link
Member

indutny commented Aug 11, 2015

@bnoordhuis are you sure that they are leaking? I thought ArrayBuffer just takes ownership of them now?

@bnoordhuis
Copy link
Member Author

Buffer::New() makes a copy of the data, see here.

@indutny
Copy link
Member

indutny commented Aug 11, 2015

Oh gosh, I was looking at it yesterday too. Looks like I was looking at wrong overloaded function then. Shouldn't it be Buffer::Use now, then?

@bnoordhuis
Copy link
Member Author

I thought Buffer::Use() had been phased out because nothing in src/ uses it anymore. @trevnorris?

@kzc
Copy link

kzc commented Aug 12, 2015

Buffer::Use() has much better memory utilization running @silverwind's leak detect script than this branch using Buffer::New(env, base, nread, FreePointer)

Buffer::Use() after 2 minutes - doesn't exceeds 600 MB:

303 MB
304 MB
368 MB
468 MB
304 MB
304 MB
390 MB
491 MB
313 MB
313 MB
405 MB
499 MB
302 MB
310 MB
406 MB
507 MB
291 MB

Buffer::New(env, base, nread, FreePointer) after 2 minutes:

670 MB
770 MB
864 MB
965 MB
1066 MB
1167 MB
1268 MB
1369 MB
1470 MB
462 MB
464 MB
483 MB
579 MB
677 MB
777 MB
878 MB
979 MB
1080 MB
1181 MB
1282 MB
1383 MB
1484 MB
462 MB

There's no leak with Buffer::New(env, base, nread, FreePointer) but it has a much higher in-use memory profile. Buffer::Use() appears to be the better fix.

@bnoordhuis
Copy link
Member Author

Switched to Buffer::Use(), PTAL.

@silverwind
Copy link
Contributor

Gave it a test run. It's an improvement, but still not anywhere near 2.5.0's almost constant usage in my FormData testcase. Will test further.

@bnoordhuis
Copy link
Member Author

@silverwind
Copy link
Contributor

Fixed CI: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/76/

(The default parameters were failing because they were still pointing to the io.js repo)

@trevnorris
Copy link
Contributor

The implementation of Buffer::New() is wrong (my bad). It should always take control of memory. Hence why there's now also Buffer::Copy().

It was an oversight on my part to leave the internal implementation of Buffer::Use(). Basically, the fix should be to make Buffer::New() take control of memory, i.e. use Buffer::Use().

@bnoordhuis
Copy link
Member Author

PTAL, I added Buffer::Copy() and renamed Buffer::Use() to Buffer::New().

CI: https://jenkins-iojs.nodesource.com/job/node-test-pull-request/83/

@silverwind
Copy link
Contributor

This looks to fix the leak in my simple test (#2308 (comment)) completely, good job.

I still see leakage in the FormData test (https://gist.github.com/silverwind/54b3829142f93d1127bc), but I'd say it's a important first step to land this now.

@trevnorris
Copy link
Contributor

Nice job. LGTM.

@indutny
Copy link
Member

indutny commented Aug 13, 2015

LGTM!

In a few places dynamic memory was passed to the Buffer::New() overload
that makes a copy of the input, not the one that takes ownership.

This commit is a band-aid to fix the memory leaks.  Longer term, we
should look into using C++11 move semantics more effectively.

Fixes: nodejs#2308
PR-URL: nodejs#2352
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
The circular dependency problem that put them there in the first place
is no longer an issue.  Move them out of the public node_buffer.h header
and into the private node_internals.h header.

Fixes: nodejs#2308
PR-URL: nodejs#2352
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Rename the three argument overload of Buffer::New() to Buffer::Copy()
and update the code base accordingly.  The reason for renaming is to
make it impossible to miss a call site.

This coincidentally plugs a small memory leak in crypto.getAuthTag().

Fixes: nodejs#2308
PR-URL: nodejs#2352
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Fixes: nodejs#2308
PR-URL: nodejs#2352
Reviewed-By: Fedor Indutny <fedor@indutny.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
@bnoordhuis bnoordhuis closed this Aug 13, 2015
@bnoordhuis bnoordhuis deleted the fix-memory-leaks branch August 13, 2015 20:00
@bnoordhuis bnoordhuis merged commit 8841947 into nodejs:master Aug 13, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. memory Issues and PRs related to the memory management or memory footprint.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants