Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buffer: replace SlowBuffer with Buffer.allocUnsafeSlow(size) #5833

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@jasnell
Copy link
Member

jasnell commented Mar 21, 2016

Pull Request check-list

  • Does make -j8 test (UNIX) or vcbuild test nosign (Windows) pass with
    this change (including linting)?
  • Is the commit message formatted according to [CONTRIBUTING.md][0]?
  • If this change fixes a bug (or a performance problem), is a regression
    test (or a benchmark) included?
  • Is a documentation update included (if this change modifies
    existing APIs, or introduces new ones)?

Affected core subsystem(s)

buffer

Description of change

  • Adds Buffer.allocUnsafeSlow() as a replacement for SlowBuffer
  • Soft-deprecate SlowBuffer

Essentially, this aligns the functionality of SlowBuffer with the new constructor API

Refs: #5799

/cc @ChALkeR @trevnorris

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Mar 21, 2016

semver-major because of the docs-only deprecation.

@ChALkeR

View changes

doc/api/buffer.markdown Outdated

When calling `Buffer.allocUnsafe()`, the segment of allocated memory is
*uninitialized* (it is not zeroed-out). While this design makes the allocation
of memory quite fast, the allocated segment of memory might contain old data
that is potentially sensitive. Using a `Buffer` created by
`Buffer.allocUnsafe(size)` without *completely* overwriting the memory can
allow this old data to be leaked when the `Buffer` memory is read.
`Buffer.allocUnsafe()` without *completely* overwriting the memory can allow

This comment has been minimized.

@ChALkeR

ChALkeR Mar 21, 2016

Member

There are two spaces in «the memory» for some reason.

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Mar 21, 2016

Hm. Overall looks good, but I'm not sure on the exact syntax.

It seems to me that Buffer.allocUnsafe(10, true) is not very clear to someone who does not use (or see) it often. Pehaps adding an options argument would be better? E.g. Buffer.allocUnsafe(10, {pooled: false})?

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Mar 21, 2016

The object would make sense if we'd ever expect there to be additional possible options. I don't think that's the case. Also, part of the point for adding it as a second parameter was to avoid making things unnecessarily complex for this case (as opposed to adding a new function, etc). I know it's less obvious what is happening, but I think it's appropriate here. But that's just me. What do you think @trevnorris ?

@jasnell jasnell modified the milestone: 6.0.0 Mar 22, 2016

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Mar 22, 2016

@trevnorris

This comment has been minimized.

Copy link
Contributor

trevnorris commented Mar 22, 2016

Could also have Buffer.allocUnsafe(n, kUnpooled) as a bitwise flag. :P

Without writing any benchmarks, I think passing in the extra object isn't going to add much if any noticeable overhead. Though after watching the Buffer API evolve over the last several years I can't foresee needing to pass any other options. Possibly use the boolean, but also strict check that the argument is a boolean instead of coercing it.

@trevnorris

View changes

lib/buffer.js Outdated
@@ -118,12 +118,12 @@ Buffer.alloc = function(size, fill, encoding) {
* Equivalent to Buffer(num), by default creates a non-zero-filled Buffer
* instance. If `--zero-fill-buffers` is set, will zero-fill the buffer.
**/
Buffer.allocUnsafe = function(size) {
Buffer.allocUnsafe = function(size, unpooled) {

This comment has been minimized.

@trevnorris

trevnorris Mar 22, 2016

Contributor

something like:

if (unpooled !== undefined && typeof unpooled !== 'boolean')
  throw new TypeError('unpooled must be a boolean');

@jasnell jasnell force-pushed the jasnell:dep-slow-buffer branch Mar 22, 2016

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Mar 22, 2016

@trevnorris ... added the additional type check.

@trevnorris

This comment has been minimized.

Copy link
Contributor

trevnorris commented Mar 23, 2016

Thanks. I'm cool with the new API, but would also like other's opinions.

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Mar 23, 2016

@nodejs/ctc ... please weigh in :-)

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Mar 24, 2016

@Fishrock123

This comment has been minimized.

Copy link
Member

Fishrock123 commented Mar 29, 2016

I think I'd prefer the bitwise flag idea.

Pros:

  • more explicit
  • extensible if we need other booleans in the future

Cons:

  • bitwise flag

I'm not so for adding [, true] to the buffer apis, it's not very easy to tell what that is. Also, I can't imagine many people are actually using SlowBuffer.

@ChALkeR could you test npm to try to see who might be using this?

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Mar 30, 2016

A bitwise flag sounds like a good compromise on this. I can't imagine that we'd actually add any more flags to this but you never know.

@jasnell jasnell force-pushed the jasnell:dep-slow-buffer branch 2 times, most recently Apr 1, 2016

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Apr 1, 2016

@Fishrock123 @trevnorris ... switched to using a Bitwise field. PTAL

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Apr 1, 2016

This approach looks better to me =).

@Fishrock123

@ChALkeR could you test npm to try to see who might be using this?

Using SlowBuffer or Buffer.allocUnsafe? The former won't be affected by this, and the latter isn't used yet because it just recently landed to master.

SlowBuffer( usage: https://gist.github.com/ChALkeR/43f3376f46b3d71e1e62ef37f643fb06

Note that this doesn't include on purpose pure instanceof SlowBuffer (which is broken btw) and SlowBuffer method redifinitions (there were some, copied in various modules, they mostly add some methods from Buffer to SlowBuffer for older Node.js versions).

Note: mocha has its own SlowBuffer that is not related to Node.js SlowBuffer, ignore that match. It affects many lines in the grep result.

@jasnell jasnell force-pushed the jasnell:dep-slow-buffer branch 2 times, most recently Apr 2, 2016

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Apr 2, 2016

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Apr 3, 2016

@jasnell jasnell changed the title buffer: replace SlowBuffer with Buffer.allocUnsafe(size, true) buffer: replace SlowBuffer with Buffer.allocUnsafe(size, Buffer.F_UNPOOLED) Apr 3, 2016

@ChALkeR

View changes

lib/buffer.js Outdated
if (typeof size !== 'number')
throw new TypeError('"size" argument must be a number');
return allocate(size);
return ((allocflags & Buffer.F_UNPOOLED) === Buffer.F_UNPOOLED) ?
createBuffer(size) : allocate(size);

This comment has been minimized.

@ChALkeR

ChALkeR Apr 3, 2016

Member

It looks like this is missing the

    if (size > 0)
      flags[kNoZeroFill] = 1;

in the createBuffer case.

This comment has been minimized.

@jasnell

jasnell Apr 3, 2016

Author Member

Good catch.

@jasnell jasnell force-pushed the jasnell:dep-slow-buffer branch Apr 3, 2016

@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Apr 3, 2016

@ChALkeR ... updated! PTAL

jasnell added a commit that referenced this pull request Apr 15, 2016

doc: minor copy improvement in buffer.markdown
PR-URL: #5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
@jasnell

This comment has been minimized.

Copy link
Member Author

jasnell commented Apr 15, 2016

Landed in 6275249, 3fe204c, and a0579c0

@jasnell jasnell closed this Apr 15, 2016

@ChALkeR ChALkeR referenced this pull request Apr 19, 2016

Closed

What about SlowBuffer? #5799

joelostrowski added a commit to joelostrowski/node that referenced this pull request Apr 25, 2016

buffer: add Buffer.allocUnsafeSlow(size)
Aligns the functionality of SlowBuffer with the new Buffer
constructor API. Next step is to docs-only deprecate
SlowBuffer.

Replace the internal uses of SlowBuffer with
`Buffer.allocUnsafeSlow(size)`

PR-URL: nodejs#5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

joelostrowski added a commit to joelostrowski/node that referenced this pull request Apr 25, 2016

buffer: docs-only deprecate SlowBuffer
With the addition of `Buffer.allocUnsafeSlow(size)`
`SlowBuffer` can be deprecated... but docs-only for now.

PR-URL: nodejs#5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

joelostrowski added a commit to joelostrowski/node that referenced this pull request Apr 25, 2016

doc: minor copy improvement in buffer.markdown
PR-URL: nodejs#5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

jasnell added a commit that referenced this pull request Apr 26, 2016

buffer: add Buffer.allocUnsafeSlow(size)
Aligns the functionality of SlowBuffer with the new Buffer
constructor API. Next step is to docs-only deprecate
SlowBuffer.

Replace the internal uses of SlowBuffer with
`Buffer.allocUnsafeSlow(size)`

PR-URL: #5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

jasnell added a commit that referenced this pull request Apr 26, 2016

buffer: docs-only deprecate SlowBuffer
With the addition of `Buffer.allocUnsafeSlow(size)`
`SlowBuffer` can be deprecated... but docs-only for now.

PR-URL: #5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

jasnell added a commit that referenced this pull request Apr 26, 2016

doc: minor copy improvement in buffer.markdown
PR-URL: #5833
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>

@ChALkeR ChALkeR referenced this pull request May 5, 2016

Closed

buffer: backport new buffer constructor APIs #5763

0 of 4 tasks complete
@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented May 5, 2016

Note that this change wasn't backported to 5.x, unlike all the other new Buffer API methods, which were backported in #5763.

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Jun 5, 2016

@jasnell Backport to v5.x in #7169.

ChALkeR added a commit to ChALkeR/io.js that referenced this pull request Jul 8, 2016

buffer: backport new buffer constructor APIs to v4.x
This backports the new `Buffer.alloc()`, `Buffer.allocUnsafe()`,
`Buffer.from()`, and `Buffer.allocUnsafeSlow()` APIs for v4.

Some backported tests are disabled, but those are not related to the
new API.

Note that `Buffer.from(arrayBuffer[, byteOffset [, length]])` is not
supported in v4.x, only `Buffer.from(arrayBuffer)` is.

Refs: nodejs#4682
Refs: nodejs#5833
Refs: nodejs#7475
PR-URL: nodejs#7562
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Nikolai Vavilov <vvnicholas@gmail.com>

MylesBorins added a commit that referenced this pull request Jul 8, 2016

buffer: backport new buffer constructor APIs to v4.x
This backports the new `Buffer.alloc()`, `Buffer.allocUnsafe()`,
`Buffer.from()`, and `Buffer.allocUnsafeSlow()` APIs for v4.

Some backported tests are disabled, but those are not related to the
new API.

Note that `Buffer.from(arrayBuffer[, byteOffset [, length]])` is not
supported in v4.x, only `Buffer.from(arrayBuffer)` is.

Refs: #4682
Refs: #5833
Refs: #7475
PR-URL: #7562
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Nikolai Vavilov <vvnicholas@gmail.com>

MylesBorins added a commit that referenced this pull request Jul 11, 2016

buffer: backport new buffer constructor APIs to v4.x
This backports the new `Buffer.alloc()`, `Buffer.allocUnsafe()`,
`Buffer.from()`, and `Buffer.allocUnsafeSlow()` APIs for v4.

Some backported tests are disabled, but those are not related to the
new API.

Note that `Buffer.from(arrayBuffer[, byteOffset [, length]])` is not
supported in v4.x, only `Buffer.from(arrayBuffer)` is.

Refs: #4682
Refs: #5833
Refs: #7475
PR-URL: #7562
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Nikolai Vavilov <vvnicholas@gmail.com>

MylesBorins added a commit that referenced this pull request Jul 12, 2016

buffer: backport new buffer constructor APIs to v4.x
This backports the new `Buffer.alloc()`, `Buffer.allocUnsafe()`,
`Buffer.from()`, and `Buffer.allocUnsafeSlow()` APIs for v4.

Some backported tests are disabled, but those are not related to the
new API.

Note that `Buffer.from(arrayBuffer[, byteOffset [, length]])` is not
supported in v4.x, only `Buffer.from(arrayBuffer)` is.

Refs: #4682
Refs: #5833
Refs: #7475
PR-URL: #7562
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Nikolai Vavilov <vvnicholas@gmail.com>

MylesBorins added a commit that referenced this pull request Jul 14, 2016

buffer: backport new buffer constructor APIs to v4.x
This backports the new `Buffer.alloc()`, `Buffer.allocUnsafe()`,
`Buffer.from()`, and `Buffer.allocUnsafeSlow()` APIs for v4.

Some backported tests are disabled, but those are not related to the
new API.

Note that `Buffer.from(arrayBuffer[, byteOffset [, length]])` is not
supported in v4.x, only `Buffer.from(arrayBuffer)` is.

Refs: #4682
Refs: #5833
Refs: #7475
PR-URL: #7562
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Nikolai Vavilov <vvnicholas@gmail.com>

MylesBorins added a commit that referenced this pull request Jul 14, 2016

buffer: backport new buffer constructor APIs to v4.x
This backports the new `Buffer.alloc()`, `Buffer.allocUnsafe()`,
`Buffer.from()`, and `Buffer.allocUnsafeSlow()` APIs for v4.

Some backported tests are disabled, but those are not related to the
new API.

Note that `Buffer.from(arrayBuffer[, byteOffset [, length]])` is not
supported in v4.x, only `Buffer.from(arrayBuffer)` is.

Refs: #4682
Refs: #5833
Refs: #7475
PR-URL: #7562
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Nikolai Vavilov <vvnicholas@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.