Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wasi: use memory-tracking allocator #30745

Closed
wants to merge 6 commits into from
Closed

Conversation

@addaleax
Copy link
Member

addaleax commented Dec 1, 2019

This addresses a TODO comment.


src: port memory-tracking allocator from QUIC repo

This implements a memory-tracking allocator that can be used to
provide memory allocation facilities to several thread-safe C
libraries, including nghttp2, nghttp3, ngtcp3 and uvwasi.

Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Co-authored-by: @jasnell

http2: use shared memory tracking implementation

The extensive testing done on http2 makes it easier to make sure
the implementation is correct (and doesn’t diverge unnecessarily).

Refs: nodejs/quic#126
Reviewed-By: @danbev
Reviewed-By: @jasnell

http2: track nghttp2-allocated memory in heap snapshot

WIP [upstream PR is open]

See cjihrig/uvwasi#60

deps: update uvwasi

wasi: use memory-tracking allocator

This:

  • Protects against memory leaks in uvwasi.
  • Allows tracking the allocated memory in heap dumps.
Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines
@addaleax addaleax requested a review from cjihrig Dec 1, 2019
@cjihrig
cjihrig approved these changes Dec 1, 2019
addaleax and others added 5 commits Dec 1, 2019
This implements a memory-tracking allocator that can be used to
provide memory allocation facilities to several thread-safe C
libraries, including nghttp2, nghttp3, ngtcp3 and uvwasi.

Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Co-authored-by: James M Snell <jasnell@gmail.com>
The extensive testing done on http2 makes it easier to make sure
the implementation is correct (and doesn’t diverge unnecessarily).

Refs: nodejs/quic#126
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
This:

- Protects against memory leaks in uvwasi.
- Allows tracking the allocated memory in heap dumps.
@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Dec 1, 2019

Rebased to include a full update of uvwasi to its current master branch, rather than cherry-pick only my upstream PR.

@addaleax addaleax force-pushed the addaleax:wasi-changes branch from d244d92 to 05d809e Dec 1, 2019
@nodejs-github-bot

This comment has been minimized.

src/node_mem.h Show resolved Hide resolved
@jasnell
jasnell approved these changes Dec 1, 2019
src/node_mem.h Outdated Show resolved Hide resolved
Co-Authored-By: Jiawen Geng <technicalcute@gmail.com>
@nodejs-github-bot

This comment has been minimized.

@nodejs-github-bot

This comment has been minimized.

@Trott
Trott approved these changes Dec 5, 2019
@joaocgreis

This comment has been minimized.

Copy link
Member

joaocgreis commented Dec 5, 2019

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Dec 7, 2019

Landed in 4e11920...a9abc17

addaleax added a commit that referenced this pull request Dec 7, 2019
This implements a memory-tracking allocator that can be used to
provide memory allocation facilities to several thread-safe C
libraries, including nghttp2, nghttp3, ngtcp3 and uvwasi.

Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Co-authored-by: James M Snell <jasnell@gmail.com>

PR-URL: #30745
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
addaleax added a commit that referenced this pull request Dec 7, 2019
The extensive testing done on http2 makes it easier to make sure
the implementation is correct (and doesn’t diverge unnecessarily).

Refs: nodejs/quic#126
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
addaleax added a commit that referenced this pull request Dec 7, 2019
PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
addaleax added a commit that referenced this pull request Dec 7, 2019
PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
addaleax added a commit that referenced this pull request Dec 7, 2019
This:

- Protects against memory leaks in uvwasi.
- Allows tracking the allocated memory in heap dumps.

PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
@addaleax addaleax closed this Dec 7, 2019
@addaleax addaleax deleted the addaleax:wasi-changes branch Dec 7, 2019
targos added a commit that referenced this pull request Dec 9, 2019
This implements a memory-tracking allocator that can be used to
provide memory allocation facilities to several thread-safe C
libraries, including nghttp2, nghttp3, ngtcp3 and uvwasi.

Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Co-authored-by: James M Snell <jasnell@gmail.com>

PR-URL: #30745
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Dec 9, 2019
The extensive testing done on http2 makes it easier to make sure
the implementation is correct (and doesn’t diverge unnecessarily).

Refs: nodejs/quic#126
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Dec 9, 2019
PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Dec 9, 2019
PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Dec 9, 2019
This:

- Protects against memory leaks in uvwasi.
- Allows tracking the allocated memory in heap dumps.

PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sebastien-Ahkrin added a commit to Sebastien-Ahkrin/node that referenced this pull request Dec 11, 2019
This implements a memory-tracking allocator that can be used to
provide memory allocation facilities to several thread-safe C
libraries, including nghttp2, nghttp3, ngtcp3 and uvwasi.

Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Co-authored-by: James M Snell <jasnell@gmail.com>

PR-URL: nodejs#30745
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sebastien-Ahkrin added a commit to Sebastien-Ahkrin/node that referenced this pull request Dec 11, 2019
The extensive testing done on http2 makes it easier to make sure
the implementation is correct (and doesn’t diverge unnecessarily).

Refs: nodejs/quic#126
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: nodejs#30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sebastien-Ahkrin added a commit to Sebastien-Ahkrin/node that referenced this pull request Dec 11, 2019
PR-URL: nodejs#30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sebastien-Ahkrin added a commit to Sebastien-Ahkrin/node that referenced this pull request Dec 11, 2019
PR-URL: nodejs#30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sebastien-Ahkrin added a commit to Sebastien-Ahkrin/node that referenced this pull request Dec 11, 2019
This:

- Protects against memory leaks in uvwasi.
- Allows tracking the allocated memory in heap dumps.

PR-URL: nodejs#30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
@MylesBorins MylesBorins mentioned this pull request Dec 13, 2019
MylesBorins added a commit that referenced this pull request Dec 16, 2019
This is a security release.

This release includes a single commit, an update to npm to 6.13.4.

For more details about the vulnerability please consult the npm blog:

https://blog.npmjs.org/post/189618601100/binary-planting-with-the-npm-cli

Notable Changes:
* deps:
  - update npm to 6.13.4
    #30904
  - update uvwasi (Anna Henningsen)
    #30745
  - upgrade to libuv 1.34.0 (Colin Ihrig)
    #30783
* doc:
  - docs deprecate http finished (Robert Nagy)
    #28679
* events:
  - add captureRejection option (Matteo Collina)
    #27867
* http:
  - add captureRejection support (Matteo Collina)
    #27867
  - llhttp opt-in insecure HTTP header parsing (Sam Roberts)
    #30567
* http2:
  - implement capture rection for 'request' and 'stream' events (Matteo Collina)
    #27867
* net:
  - implement capture rejections for 'connection' event (Matteo Collina)
    #27867
* repl:
  - support previews by eager evaluating input (Ruben Bridgewater)
    #30811
* stream:
  - add support for captureRejection option (Matteo Collina)
    #27867
* tls:
  - implement capture rejections for 'secureConnection' event (Matteo Collina)
    #27867
  - expose IETF name for current cipher suite (Sam Roberts)
    #30637
* worker:
  - add argv constructor option (legendecas)
    #30559

PR-URL: #30937
MylesBorins added a commit that referenced this pull request Dec 16, 2019
This is a security release.

This release includes a single commit, an update to npm to 6.13.4.

For more details about the vulnerability please consult the npm blog:

https://blog.npmjs.org/post/189618601100/binary-planting-with-the-npm-cli

Notable Changes:
* deps:
  * update npm to 6.13.4
    #30904
  * update uvwasi (Anna Henningsen)
    #30745
  * upgrade to libuv 1.34.0 (Colin Ihrig)
    #30783
* doc:
  * docs deprecate http finished (Robert Nagy)
    #28679
* events:
  * add captureRejection option (Matteo Collina)
    #27867
* http:
  * add captureRejection support (Matteo Collina)
    #27867
  * llhttp opt-in insecure HTTP header parsing (Sam Roberts)
    #30567
* http2:
  * implement capture rection for 'request' and 'stream' events (Matteo Collina)
    #27867
* net:
  * implement capture rejections for 'connection' event (Matteo Collina)
    #27867
* repl:
  * support previews by eager evaluating input (Ruben Bridgewater)
    #30811
* stream:
  * add support for captureRejection option (Matteo Collina)
    #27867
* tls:
  * implement capture rejections for 'secureConnection' event (Matteo Collina)
    #27867
  * expose IETF name for current cipher suite (Sam Roberts)
    #30637
* worker:
  * add argv constructor option (legendecas)
    #30559

PR-URL: #30937
MylesBorins added a commit that referenced this pull request Dec 16, 2019
This is a security release.

For more details about the vulnerability please consult the npm blog:

https://blog.npmjs.org/post/189618601100/binary-planting-with-the-npm-cli

Notable Changes:
* deps:
  * update npm to 6.13.4
    #30904
  * update uvwasi (Anna Henningsen)
    #30745
  * upgrade to libuv 1.34.0 (Colin Ihrig)
    #30783
* doc:
  * docs deprecate http finished (Robert Nagy)
    #28679
* events:
  * add captureRejection option (Matteo Collina)
    #27867
* http:
  * add captureRejection support (Matteo Collina)
    #27867
  * llhttp opt-in insecure HTTP header parsing (Sam Roberts)
    #30567
* http2:
  * implement capture rection for 'request' and 'stream' events (Matteo Collina)
    #27867
* net:
  * implement capture rejections for 'connection' event (Matteo Collina)
    #27867
* repl:
  * support previews by eager evaluating input (Ruben Bridgewater)
    #30811
* stream:
  * add support for captureRejection option (Matteo Collina)
    #27867
* tls:
  * implement capture rejections for 'secureConnection' event (Matteo Collina)
    #27867
  * expose IETF name for current cipher suite (Sam Roberts)
    #30637
* worker:
  * add argv constructor option (legendecas)
    #30559

PR-URL: #30937
MylesBorins added a commit that referenced this pull request Dec 17, 2019
This is a security release.

For more details about the vulnerability please consult the npm blog:

https://blog.npmjs.org/post/189618601100/binary-planting-with-the-npm-cli

Notable Changes:
* deps:
  * update npm to 6.13.4
    #30904
  * update uvwasi (Anna Henningsen)
    #30745
  * upgrade to libuv 1.34.0 (Colin Ihrig)
    #30783
* doc:
  * docs deprecate http finished (Robert Nagy)
    #28679
* events:
  * add captureRejection option (Matteo Collina)
    #27867
* http:
  * add captureRejection support (Matteo Collina)
    #27867
  * llhttp opt-in insecure HTTP header parsing (Sam Roberts)
    #30567
* http2:
  * implement capture rection for 'request' and 'stream' events (Matteo Collina)
    #27867
* net:
  * implement capture rejections for 'connection' event (Matteo Collina)
    #27867
* repl:
  * support previews by eager evaluating input (Ruben Bridgewater)
    #30811
* stream:
  * add support for captureRejection option (Matteo Collina)
    #27867
* tls:
  * implement capture rejections for 'secureConnection' event (Matteo Collina)
    #27867
  * expose IETF name for current cipher suite (Sam Roberts)
    #30637
* worker:
  * add argv constructor option (legendecas)
    #30559

PR-URL: #30937
targos added a commit that referenced this pull request Jan 14, 2020
This implements a memory-tracking allocator that can be used to
provide memory allocation facilities to several thread-safe C
libraries, including nghttp2, nghttp3, ngtcp3 and uvwasi.

Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Co-authored-by: James M Snell <jasnell@gmail.com>

PR-URL: #30745
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Jan 14, 2020
The extensive testing done on http2 makes it easier to make sure
the implementation is correct (and doesn’t diverge unnecessarily).

Refs: nodejs/quic#126
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Jan 14, 2020
PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Jan 14, 2020
PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
targos added a commit that referenced this pull request Jan 14, 2020
This:

- Protects against memory leaks in uvwasi.
- Allows tracking the allocated memory in heap dumps.

PR-URL: #30745
Refs: https://github.com/nodejs/quic/blob/34ee0bc96f804c73cb22b2945a1a78f780938492/src/node_mem.h
Refs: nodejs/quic#126
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.