New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buffer: runtime-deprecate Buffer constructor #7152

Closed
wants to merge 4 commits into
base: master
from

Conversation

Projects
None yet
@seishun
Member

seishun commented Jun 4, 2016

Checklist
  • tests and code linting passes
  • the commit message follows commit guidelines
Affected core subsystem(s)

buffer

Description of change
  1. New Buffer API was introduced to prevent accidental data leakage and/or DoS (see #4660). But there are older modules that haven't been updated and might still be vulnerable. Runtime deprecation encourages module authors to update to the new API, and the users to update their dependencies.
  2. It is desirable that Buffer be a proper ES6 class that can be extended. However, parts of the deprecated Buffer API (such as creating a Buffer instance without new) make it impossible, while other parts (such as construction from a string) make it more complex. It will make things much simpler if the problematic deprecated parts are removed. But they need to be runtime-deprecated first.
@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 4, 2016

Member

@seishun I don't think this is viable at the moment. New API hasn't been backported even to 4.x afaik.
They don't port not because the old API isn't hard depracated, but because the new API isn't supported everywhere yet.

There should be a single way to allocate Buffers on all supported node versions, so I hope that:

  1. The new API will get backported to 4.x soon.
  2. By the end of the year, when 0.10 and 0.12 get phased out, all supported Node.js versions would have the new Buffer API. Then, library writers can port to the new API without requiring third-party compatibility modules (or with them, if they care about 0.12 for some reason).
  3. When a significant part of libraries will migrate (this would mean that the ecosystem is mostly fine with dropping old versions) — hard-deprecation will be a good solution to give the remaining users an indication that they should also port.

Perhaps target for 8.0 and re-evaluate later?

Member

ChALkeR commented Jun 4, 2016

@seishun I don't think this is viable at the moment. New API hasn't been backported even to 4.x afaik.
They don't port not because the old API isn't hard depracated, but because the new API isn't supported everywhere yet.

There should be a single way to allocate Buffers on all supported node versions, so I hope that:

  1. The new API will get backported to 4.x soon.
  2. By the end of the year, when 0.10 and 0.12 get phased out, all supported Node.js versions would have the new Buffer API. Then, library writers can port to the new API without requiring third-party compatibility modules (or with them, if they care about 0.12 for some reason).
  3. When a significant part of libraries will migrate (this would mean that the ecosystem is mostly fine with dropping old versions) — hard-deprecation will be a good solution to give the remaining users an indication that they should also port.

Perhaps target for 8.0 and re-evaluate later?

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 5, 2016

Member

Btw, this should also probably hard-deprecate SlowBuffer at the same time — it was replaced by Buffer.allocUnsafeSlow.

Member

ChALkeR commented Jun 5, 2016

Btw, this should also probably hard-deprecate SlowBuffer at the same time — it was replaced by Buffer.allocUnsafeSlow.

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Jun 6, 2016

Member

Yeah, this is something that's going to need to wait for a while... I'm thinking at least v8 but likely longer.

Member

jasnell commented Jun 6, 2016

Yeah, this is something that's going to need to wait for a while... I'm thinking at least v8 but likely longer.

@Fishrock123

This comment has been minimized.

Show comment
Hide comment
@Fishrock123

Fishrock123 Jun 6, 2016

Member

I'm thinking V10+ at minimum. Usage of this is very widespread. We should evaluate as time goes on.

Member

Fishrock123 commented Jun 6, 2016

I'm thinking V10+ at minimum. Usage of this is very widespread. We should evaluate as time goes on.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jun 6, 2016

Member

Usage of this is very widespread.

It's unlikely to decrease substantially without hard-deprecation. The whole point of hard-deprecation is to decrease the existing usage.

I think there might be a misunderstanding here. I would like to remind that hard-deprecation means a one-time console notification that can be disabled.

Member

seishun commented Jun 6, 2016

Usage of this is very widespread.

It's unlikely to decrease substantially without hard-deprecation. The whole point of hard-deprecation is to decrease the existing usage.

I think there might be a misunderstanding here. I would like to remind that hard-deprecation means a one-time console notification that can be disabled.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 6, 2016

Member

@seishun At this point, users are not migrating to the new Buffer API not because they didn't get a deprecation message in every app, but because the new API isn't supported by our current LTS version (and the Maintenance versions).

Member

ChALkeR commented Jun 6, 2016

@seishun At this point, users are not migrating to the new Buffer API not because they didn't get a deprecation message in every app, but because the new API isn't supported by our current LTS version (and the Maintenance versions).

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jun 6, 2016

Member

@ChALkeR It's both. I'm not arguing against the lack-of-support-in-LTS factor.

Member

seishun commented Jun 6, 2016

@ChALkeR It's both. I'm not arguing against the lack-of-support-in-LTS factor.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 6, 2016

Member

@Fishrock123 I still hope to see users migrating before v8.0 release. If the new API will be backported to v4, then I plan to file issues to most used repos starting from 2017-01, when 0.12 will become not supported anymore.

So hopefully by April 2017 the usage of old API will decrease, and we should at least re-evaluate the deprecation before v8.0 release.

Member

ChALkeR commented Jun 6, 2016

@Fishrock123 I still hope to see users migrating before v8.0 release. If the new API will be backported to v4, then I plan to file issues to most used repos starting from 2017-01, when 0.12 will become not supported anymore.

So hopefully by April 2017 the usage of old API will decrease, and we should at least re-evaluate the deprecation before v8.0 release.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 6, 2016

Member

To be clear: I am not saying that we should definitely force this in v8.0, but we should postpone this at least until v8.0 (unless the new API will get backported to 0.10/0.12, which I find unlikely).

So, perhaps set a milestone to v8.0 to not forget to reevaluate by then?

Member

ChALkeR commented Jun 6, 2016

To be clear: I am not saying that we should definitely force this in v8.0, but we should postpone this at least until v8.0 (unless the new API will get backported to 0.10/0.12, which I find unlikely).

So, perhaps set a milestone to v8.0 to not forget to reevaluate by then?

@trevnorris

This comment has been minimized.

Show comment
Hide comment
@trevnorris

trevnorris Jun 7, 2016

Contributor

This can never happen. Uint8Array methods will rely on the Buffer constructor (hence why the Buffer constructor has been changed to essentially become compatible with Uint8Array). Going forward the change that will need to be made is that Buffer will automatically zero-fill memory (again to match Uint8Array). We can even start removing functionality that isn't supported by Uint8Array, but the constructor itself will live on forever.

Contributor

trevnorris commented Jun 7, 2016

This can never happen. Uint8Array methods will rely on the Buffer constructor (hence why the Buffer constructor has been changed to essentially become compatible with Uint8Array). Going forward the change that will need to be made is that Buffer will automatically zero-fill memory (again to match Uint8Array). We can even start removing functionality that isn't supported by Uint8Array, but the constructor itself will live on forever.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jun 8, 2016

Member

@trevnorris I don't follow. Uint8Array is a native JS feature, how can it rely on Buffer? Or do you mean methods added to it from Buffer.prototype? That doesn't rely on the constructor either.

Either way, I don't see how hard-deprecating the constructor is incompatible with the constructor living on forever.

Member

seishun commented Jun 8, 2016

@trevnorris I don't follow. Uint8Array is a native JS feature, how can it rely on Buffer? Or do you mean methods added to it from Buffer.prototype? That doesn't rely on the constructor either.

Either way, I don't see how hard-deprecating the constructor is incompatible with the constructor living on forever.

@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Jun 8, 2016

Member

@seishun For some of the methods of Uint8Array that return typed arrays again, the .constructor property is inspected and used for creating of the return value:

> a = Buffer.alloc(4)
<Buffer 00 00 00 00>
> a.map === Uint8Array.prototype.map
true
> a.map(x => x+1)
<Buffer 01 01 01 01> // Not a Uint8Array! .map() called a.constructor, which is Buffer.

This is probably fixable by setting Buffer[Symbol.species] to something that does basically what the current Buffer constructor does, but that’s still behind the --harmony_species flag. (edit: “fixable” in the sense that one could still deprecate the Buffer constructor if that is desired).

Member

addaleax commented Jun 8, 2016

@seishun For some of the methods of Uint8Array that return typed arrays again, the .constructor property is inspected and used for creating of the return value:

> a = Buffer.alloc(4)
<Buffer 00 00 00 00>
> a.map === Uint8Array.prototype.map
true
> a.map(x => x+1)
<Buffer 01 01 01 01> // Not a Uint8Array! .map() called a.constructor, which is Buffer.

This is probably fixable by setting Buffer[Symbol.species] to something that does basically what the current Buffer constructor does, but that’s still behind the --harmony_species flag. (edit: “fixable” in the sense that one could still deprecate the Buffer constructor if that is desired).

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 8, 2016

Member

@addaleax @trevnorris
Is that fixable by checking the argument type of the Buffer() and printing a hard-deprecation message only if argument is not a Buffer or a typed array?

Member

ChALkeR commented Jun 8, 2016

@addaleax @trevnorris
Is that fixable by checking the argument type of the Buffer() and printing a hard-deprecation message only if argument is not a Buffer or a typed array?

@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Jun 8, 2016

Member

@ChALkeR The .map() example will call new Buffer(n) via this line, so I don’t think that will work out.

Member

addaleax commented Jun 8, 2016

@ChALkeR The .map() example will call new Buffer(n) via this line, so I don’t think that will work out.

@trevnorris

This comment has been minimized.

Show comment
Hide comment
@trevnorris

trevnorris Jun 8, 2016

Contributor

@addaleax

This is probably fixable by setting Buffer[Symbol.species] to something that does basically what the current Buffer constructor does

I'm hoping to avoid that all together by making the Buffer constructor work correctly with class inheritance.

EDIT: so that basically any inherited methods will return a Buffer instance. e.g. buffer.subarray() now returns a Uint8Array instead of a Buffer. It would simply things if we could change Buffer to require using new. Then it can properly use class inheritance, and the whole thing would be solved.

Contributor

trevnorris commented Jun 8, 2016

@addaleax

This is probably fixable by setting Buffer[Symbol.species] to something that does basically what the current Buffer constructor does

I'm hoping to avoid that all together by making the Buffer constructor work correctly with class inheritance.

EDIT: so that basically any inherited methods will return a Buffer instance. e.g. buffer.subarray() now returns a Uint8Array instead of a Buffer. It would simply things if we could change Buffer to require using new. Then it can properly use class inheritance, and the whole thing would be solved.

@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Jun 8, 2016

Member

@trevnorris A couple of us chatted in IRC for a bit, and one of the problems would be that Buffer(n) without new wouldn’t work… but apparently setting Symbol.species is not even necessary:

$ node --harmony-species -p 'Buffer[Symbol.species] === Buffer'
true

So my comment above was not actually very meaningful.

Member

addaleax commented Jun 8, 2016

@trevnorris A couple of us chatted in IRC for a bit, and one of the problems would be that Buffer(n) without new wouldn’t work… but apparently setting Symbol.species is not even necessary:

$ node --harmony-species -p 'Buffer[Symbol.species] === Buffer'
true

So my comment above was not actually very meaningful.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jun 16, 2016

Member

@ChALkeR

To be clear: I am not saying that we should definitely force this in v8.0, but we should postpone this at least until v8.0 (unless the new API will get backported to 0.10/0.12, which I find unlikely).

Just 0.12, isn't it? Since support for 0.10 ends in October.

Member

seishun commented Jun 16, 2016

@ChALkeR

To be clear: I am not saying that we should definitely force this in v8.0, but we should postpone this at least until v8.0 (unless the new API will get backported to 0.10/0.12, which I find unlikely).

Just 0.12, isn't it? Since support for 0.10 ends in October.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 17, 2016

Member

@seishun

Just 0.12, isn't it? Since support for 0.10 ends in October.

Ah, yes. 0.10 EOL is at the same time when 7.0 is going to be released. But I really doubt that it would be feasible to backport to 0.12 which ends in December.

Member

ChALkeR commented Jun 17, 2016

@seishun

Just 0.12, isn't it? Since support for 0.10 ends in October.

Ah, yes. 0.10 EOL is at the same time when 7.0 is going to be released. But I really doubt that it would be feasible to backport to 0.12 which ends in December.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jun 18, 2016

Member

After some IRC discussion, it seems there is some agreement that the buffer shim would be sufficient for module authors who want to support old Node.js versions. So perhaps there might be no need to delay this by another 6 months.

Also, I've found a way to hard-deprecate the Buffer constructor without affecting Uint8Array methods that doesn't depend on @@species (see last commit). Its side effect is that Buffer == buf.constructor no longer holds, but it doesn't break any tests and it seems unlikely that a lot of people rely on it.

I've also updated the OP with another reason for hard-deprecation - namely, that it's a necessary step before Buffer can become a proper class that users can extend.

Member

seishun commented Jun 18, 2016

After some IRC discussion, it seems there is some agreement that the buffer shim would be sufficient for module authors who want to support old Node.js versions. So perhaps there might be no need to delay this by another 6 months.

Also, I've found a way to hard-deprecate the Buffer constructor without affecting Uint8Array methods that doesn't depend on @@species (see last commit). Its side effect is that Buffer == buf.constructor no longer holds, but it doesn't break any tests and it seems unlikely that a lot of people rely on it.

I've also updated the OP with another reason for hard-deprecation - namely, that it's a necessary step before Buffer can become a proper class that users can extend.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 19, 2016

Member

@seishun

After some IRC discussion, it seems there is some agreement that the buffer shim would be sufficient for module authors who want to support old Node.js versions.

Actually, on a second thought I'm fine with that, but I still want this backported to v4. I will make a PR for that to start the discussion.

Also, I've found a way to hard-deprecate the Buffer constructor without affecting Uint8Array methods that doesn't depend on @@species (see last commit). Its side effect is that Buffer == buf.constructor no longer holds, but it doesn't break any tests and it seems unlikely that a lot of people rely on it.

See https://gist.github.com/ChALkeR/41cd82a8f3b8822860c77116e880d3c1. bson module does that in one line, could that change break things for someone who uses the bson module?

Member

ChALkeR commented Jun 19, 2016

@seishun

After some IRC discussion, it seems there is some agreement that the buffer shim would be sufficient for module authors who want to support old Node.js versions.

Actually, on a second thought I'm fine with that, but I still want this backported to v4. I will make a PR for that to start the discussion.

Also, I've found a way to hard-deprecate the Buffer constructor without affecting Uint8Array methods that doesn't depend on @@species (see last commit). Its side effect is that Buffer == buf.constructor no longer holds, but it doesn't break any tests and it seems unlikely that a lot of people rely on it.

See https://gist.github.com/ChALkeR/41cd82a8f3b8822860c77116e880d3c1. bson module does that in one line, could that change break things for someone who uses the bson module?

@@ -94,17 +99,19 @@ function alignPool() {
}
}
const bufferDeprecationWarning =
deprecate(() => {}, 'Buffer constructor is deprecated. ' +
'Use Buffer.from, Buffer.allocUnsafe or Buffer.alloc instead.');

This comment has been minimized.

@ChALkeR

ChALkeR Jun 19, 2016

Member

I would prefer the order here to be Buffer.from, Buffer.alloc or Buffer.allocUnsafe instead, because the latter should be used only when people are better aware of what they are doing, i.e. are sure that they don't leak the resulted Buffer in an (even partially) unitialized state in any code branch.

@ChALkeR

ChALkeR Jun 19, 2016

Member

I would prefer the order here to be Buffer.from, Buffer.alloc or Buffer.allocUnsafe instead, because the latter should be used only when people are better aware of what they are doing, i.e. are sure that they don't leak the resulted Buffer in an (even partially) unitialized state in any code branch.

This comment has been minimized.

@seishun

seishun Jun 19, 2016

Member

I just copied from the comment below, but I don't mind changing the order.

@seishun

seishun Jun 19, 2016

Member

I just copied from the comment below, but I don't mind changing the order.

* would ever actually be removed.
* The Buffer() construtor is deprecated ... that is, it should not be used
* moving forward. Rather, developers should use one of the three new factory
* APIs: Buffer.from(), Buffer.allocUnsafe() or Buffer.alloc() based on their

This comment has been minimized.

@ChALkeR

ChALkeR Jun 19, 2016

Member

Ditto.

@ChALkeR

ChALkeR Jun 19, 2016

Member

Ditto.

This comment has been minimized.

@seishun

seishun Jun 19, 2016

Member

I was actually thinking that this comment doesn't really serve any useful purpose anymore and can be removed.

@seishun

seishun Jun 19, 2016

Member

I was actually thinking that this comment doesn't really serve any useful purpose anymore and can be removed.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jun 19, 2016

Member

See https://gist.github.com/ChALkeR/41cd82a8f3b8822860c77116e880d3c1. bson module does that in one line, could that change break things for someone who uses the bson module?

Well, I just ran bson's tests with this change and got 1 unrelated failure ("global var leak detected"), same as on current node. It did throw a bunch of Buffer deprecation warnings though.

Member

seishun commented Jun 19, 2016

See https://gist.github.com/ChALkeR/41cd82a8f3b8822860c77116e880d3c1. bson module does that in one line, could that change break things for someone who uses the bson module?

Well, I just ran bson's tests with this change and got 1 unrelated failure ("global var leak detected"), same as on current node. It did throw a bunch of Buffer deprecation warnings though.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 19, 2016

Member

@seishun I'm not sure if relying on the tests is sufficient here, it looks like they don't calculate the coverage. Not sure if that code is actually used somewhere, though.

The file in question is https://github.com/mongodb/js-bson/blob/master/alternate_parsers/faster_bson.js#L369, and it looks like it's only used by some benchmark in that repo. I couldn't find any mentions of faster_bson in any other packages, btw.

Member

ChALkeR commented Jun 19, 2016

@seishun I'm not sure if relying on the tests is sufficient here, it looks like they don't calculate the coverage. Not sure if that code is actually used somewhere, though.

The file in question is https://github.com/mongodb/js-bson/blob/master/alternate_parsers/faster_bson.js#L369, and it looks like it's only used by some benchmark in that repo. I couldn't find any mentions of faster_bson in any other packages, btw.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Jun 19, 2016

Member

@seishun nodejs/node#7169 landed with allocUnsafeSlow for v5.x, preparing a non-intrusive PR for v4.x.

Member

ChALkeR commented Jun 19, 2016

@seishun nodejs/node#7169 landed with allocUnsafeSlow for v5.x, preparing a non-intrusive PR for v4.x.

@ChALkeR ChALkeR referenced this pull request Jun 29, 2016

Closed

New Buffer API backport to v4.x #7475

4 of 4 tasks complete
@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jul 8, 2016

Member

Now that #7562 has landed, what's the current status of this?

Member

seishun commented Jul 8, 2016

Now that #7562 has landed, what's the current status of this?

@trevnorris

This comment has been minimized.

Show comment
Hide comment
@trevnorris

trevnorris Jul 11, 2016

Contributor

For note, I'm currently working with others on #4701 to allow proper extends support to Buffer. Because there is a non-zero chance that devs want to use class with Buffer means we shouldn't be blindly printing a message when it's not called as a non-class constructor. Been looking for a way around this and think we may be able to by checking
new.target !== undefined && new.target.name !== 'Buffer' before printing the warning.

This implementation is also leaky. Various things to note:

  • Buffer.name !== 'Buffer'
  • Can get the new Buffer (not DummyBuffer) by doing Buffer.prototype.constructor
  • From the retrieved Buffer class, instance methods are available but static methods are not (e.g. isEncoding())
Contributor

trevnorris commented Jul 11, 2016

For note, I'm currently working with others on #4701 to allow proper extends support to Buffer. Because there is a non-zero chance that devs want to use class with Buffer means we shouldn't be blindly printing a message when it's not called as a non-class constructor. Been looking for a way around this and think we may be able to by checking
new.target !== undefined && new.target.name !== 'Buffer' before printing the warning.

This implementation is also leaky. Various things to note:

  • Buffer.name !== 'Buffer'
  • Can get the new Buffer (not DummyBuffer) by doing Buffer.prototype.constructor
  • From the retrieved Buffer class, instance methods are available but static methods are not (e.g. isEncoding())
@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Jul 12, 2016

Member

Fair enough. Does it break any existing code though?

Perhaps you happen to know when V8 is planning to move @@species from behind a flag?

Member

seishun commented Jul 12, 2016

Fair enough. Does it break any existing code though?

Perhaps you happen to know when V8 is planning to move @@species from behind a flag?

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Aug 6, 2016

Member

@nodejs/collaborators can we add a 7.0.0 milestone here?

Member

seishun commented Aug 6, 2016

@nodejs/collaborators can we add a 7.0.0 milestone here?

@addaleax

View changes

Show outdated Hide outdated lib/buffer.js
// to avoid code churn...
const FastBuffer = Buffer;
Buffer = DummyBuffer;

This comment has been minimized.

@addaleax

addaleax Aug 7, 2016

Member

I think I’d be okay with the code churn created by resolving these if it helps clarify what each name here refers to?

@addaleax

addaleax Aug 7, 2016

Member

I think I’d be okay with the code churn created by resolving these if it helps clarify what each name here refers to?

This comment has been minimized.

@seishun

seishun Aug 7, 2016

Member

Well, that would mean changing all the "static" function declarations, e.g. Buffer.alloc to DummyBuffer.alloc, only to change them back later when Buffer becomes a proper class.

@seishun

seishun Aug 7, 2016

Member

Well, that would mean changing all the "static" function declarations, e.g. Buffer.alloc to DummyBuffer.alloc, only to change them back later when Buffer becomes a proper class.

This comment has been minimized.

@addaleax

addaleax Aug 7, 2016

Member

Hm. If the name DummyBuffer is just chosen to avoid clashing with the already-existing Buffer identifier, maybe one could resolve that by turning the above block into

const FastBuffer = (() => class Buffer extends Uint8Array {})();

… not sure if that’s more readable or not.

@addaleax

addaleax Aug 7, 2016

Member

Hm. If the name DummyBuffer is just chosen to avoid clashing with the already-existing Buffer identifier, maybe one could resolve that by turning the above block into

const FastBuffer = (() => class Buffer extends Uint8Array {})();

… not sure if that’s more readable or not.

@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Aug 7, 2016

Member

btw, I think @RReverser had something going with making Buffer subclassable without deprecating the buffer constructor… I’m not 100 % sure how this intersects with this change, but I’d definitely like to hear how that went and/or an opinion on this.

Member

addaleax commented Aug 7, 2016

btw, I think @RReverser had something going with making Buffer subclassable without deprecating the buffer constructor… I’m not 100 % sure how this intersects with this change, but I’d definitely like to hear how that went and/or an opinion on this.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Aug 7, 2016

Member

btw, I think @RReverser had something going with making Buffer subclassable without deprecating the buffer constructor…

I think there's still value in making Buffer a class. There is likely a performance difference, for instance.

Member

seishun commented Aug 7, 2016

btw, I think @RReverser had something going with making Buffer subclassable without deprecating the buffer constructor…

I think there's still value in making Buffer a class. There is likely a performance difference, for instance.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Aug 7, 2016

Member

Simplified things a bit per @addaleax's suggestion.

Member

seishun commented Aug 7, 2016

Simplified things a bit per @addaleax's suggestion.

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Nov 5, 2016

Member

To add a bit more clarity to the discussion for those not in the ctc: the decided intent was to land the Buffer-without-new deprecation as a semver-major in v7 and to deprecate at some as yet undecided time in the future all Buffer() constructor combinations other than that required for Buffer to properly subclass Uint8Array. As I have pointed out in several other threads on this, there are two primary reasons for this approach: 1) we want to more firmly nudge all users towards using the new constructor methods which have been back ported to v4 due to the usability and security concerns around the old constructor, 2) there are a number of significant internal code improvements that can be made by making Buffer an ES6 style class and a proper subclass of Uint8Array. These changes are not just change for the sake of change, they're changes that would make a significant feature of Node.js faster, more reliable, and easier to maintain. That said, the ctc fully realizes the disruption to the ecosystem and we take it quite seriously. That is why there has not yet been any determination as to when Buffer-without-new would actually stop working or when most of the Buffer() constructor patterns would be deprecated. This is also why @ChALkeR has been preemptively opening PRs to help module developers transition well in advance of such changes. It's also why the changes other than the Buffer-without-new deprecation likely wouldn't go into an LTS release until at least Node.js v10 (I highly doubt there's any way we would land that significant of a change in the time remaining until we cut v8.0).

Anyway, that's the background. As is always the case, there's a conversation to be had about whether that direction is the right one. I'm not arguing one way or the other but I am -1 to this PR landing any time soon and I would be ok with temporarily reverting the Buffer-without-new deprecation with a plan towards revisiting it later.

Member

jasnell commented Nov 5, 2016

To add a bit more clarity to the discussion for those not in the ctc: the decided intent was to land the Buffer-without-new deprecation as a semver-major in v7 and to deprecate at some as yet undecided time in the future all Buffer() constructor combinations other than that required for Buffer to properly subclass Uint8Array. As I have pointed out in several other threads on this, there are two primary reasons for this approach: 1) we want to more firmly nudge all users towards using the new constructor methods which have been back ported to v4 due to the usability and security concerns around the old constructor, 2) there are a number of significant internal code improvements that can be made by making Buffer an ES6 style class and a proper subclass of Uint8Array. These changes are not just change for the sake of change, they're changes that would make a significant feature of Node.js faster, more reliable, and easier to maintain. That said, the ctc fully realizes the disruption to the ecosystem and we take it quite seriously. That is why there has not yet been any determination as to when Buffer-without-new would actually stop working or when most of the Buffer() constructor patterns would be deprecated. This is also why @ChALkeR has been preemptively opening PRs to help module developers transition well in advance of such changes. It's also why the changes other than the Buffer-without-new deprecation likely wouldn't go into an LTS release until at least Node.js v10 (I highly doubt there's any way we would land that significant of a change in the time remaining until we cut v8.0).

Anyway, that's the background. As is always the case, there's a conversation to be had about whether that direction is the right one. I'm not arguing one way or the other but I am -1 to this PR landing any time soon and I would be ok with temporarily reverting the Buffer-without-new deprecation with a plan towards revisiting it later.

@substack

This comment has been minimized.

Show comment
Hide comment
@substack

substack Nov 5, 2016

Contributor

I was not complaining about a pull request. I was complaining that I've already heard from users seeing a deprecation warning running browserify which is buried somewhere extremely deep in the dependency graph. It might not even be a module I wrote, I have no idea. Don't do this.

Contributor

substack commented Nov 5, 2016

I was not complaining about a pull request. I was complaining that I've already heard from users seeing a deprecation warning running browserify which is buried somewhere extremely deep in the dependency graph. It might not even be a module I wrote, I have no idea. Don't do this.

@mafintosh

This comment has been minimized.

Show comment
Hide comment
@mafintosh

mafintosh Nov 5, 2016

Member

@ChALkeR I have filed and fixed plenty of security fixes myself, including the one that led to this original discussion about the Buffer constructor. I appreciate getting security patches. But that doesn't mean that it is feasible to deprecate a big part of npm (and yes I think printing a permanent deprecation message is just as disruptive). By far the vast majority of uses of the Buffer constructor are safe, and yes for newer things I'm fine encouraging a new safer API, and removing the old one from docs, but that doesn't mean we should deprecate tens of 1000s of modules that work perfectly fine!

Does this mean we shouldn't fix security issues? Of course not! If you see a buffer being vulnerable, send a PR and it'll get merged. I've found and reported plenty of REDOS attacks in modules in the past. Does that mean we should start printing deprecation messages when regexes are used as well?

Backwards compatible is the one feature I want from Node. It's the only feature that makes it feasible for module authors like myself to easily maintain 100s of modules.

Member

mafintosh commented Nov 5, 2016

@ChALkeR I have filed and fixed plenty of security fixes myself, including the one that led to this original discussion about the Buffer constructor. I appreciate getting security patches. But that doesn't mean that it is feasible to deprecate a big part of npm (and yes I think printing a permanent deprecation message is just as disruptive). By far the vast majority of uses of the Buffer constructor are safe, and yes for newer things I'm fine encouraging a new safer API, and removing the old one from docs, but that doesn't mean we should deprecate tens of 1000s of modules that work perfectly fine!

Does this mean we shouldn't fix security issues? Of course not! If you see a buffer being vulnerable, send a PR and it'll get merged. I've found and reported plenty of REDOS attacks in modules in the past. Does that mean we should start printing deprecation messages when regexes are used as well?

Backwards compatible is the one feature I want from Node. It's the only feature that makes it feasible for module authors like myself to easily maintain 100s of modules.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Nov 5, 2016

Member

I was complaining that I've already heard from users seeing a deprecation warning running browserify which is buried somewhere extremely deep in the dependency graph. It might not even be a module I wrote, I have no idea.

It was actually "buried" in browser-pack, and it's already fixed in browserify/browser-pack@c91b816. All calls of Buffer without new in node-browserify itself were in tests, and I've submitted browserify/browserify#1647 for that.

Using --trace-deprecation switch is handy to figure out where the deprecation is coming from.

I'm fine encouraging a new safer API, and removing the old one from docs

Docs-only deprecation doesn't really work since people will copy-paste code from old stack overflow answers etc.

It's the only feature that makes it feasible for module authors like myself to easily maintain 100s of modules.

If by maintaining you mean actually reading and fixing issues and not just sitting on them, then fixing the deprecation warning is a drop in the bucket, since it only takes a couple of minutes to run tests with --trace-deprecation and make some trivial replacements. And if the maintainer doesn't have time for it, there is a high likelihood that some user will do it and submit a PR, since it doesn't normally require any knowledge about the module internals.

Member

seishun commented Nov 5, 2016

I was complaining that I've already heard from users seeing a deprecation warning running browserify which is buried somewhere extremely deep in the dependency graph. It might not even be a module I wrote, I have no idea.

It was actually "buried" in browser-pack, and it's already fixed in browserify/browser-pack@c91b816. All calls of Buffer without new in node-browserify itself were in tests, and I've submitted browserify/browserify#1647 for that.

Using --trace-deprecation switch is handy to figure out where the deprecation is coming from.

I'm fine encouraging a new safer API, and removing the old one from docs

Docs-only deprecation doesn't really work since people will copy-paste code from old stack overflow answers etc.

It's the only feature that makes it feasible for module authors like myself to easily maintain 100s of modules.

If by maintaining you mean actually reading and fixing issues and not just sitting on them, then fixing the deprecation warning is a drop in the bucket, since it only takes a couple of minutes to run tests with --trace-deprecation and make some trivial replacements. And if the maintainer doesn't have time for it, there is a high likelihood that some user will do it and submit a PR, since it doesn't normally require any knowledge about the module internals.

@yoshuawuyts

This comment has been minimized.

Show comment
Hide comment
@yoshuawuyts

yoshuawuyts Nov 5, 2016

Member

If by maintaining you mean actually reading and fixing issues and not just sitting on them, then fixing the deprecation warning is a drop in the bucket, since it only takes a couple of minutes to run tests with --trace-deprecation and make some trivial replacements

@seishun your consistent downplay of maintainer's concerns regarding the amount of work this creates could be considered insensitive, borderline rude. Trust maintainers to be knowledgeable about how much overhead this creates for them. Dismissing them is NOT a stance that inspires faith in you and in turn Node core which you represent.

If we are to arrive at something resembling a consensus, we must take our time to understand where people with different opinions are coming from - failure to take people seriously with opinions different than your own is what makes situations escalate, and I'd like to think nobody in this thread is keen for that to happen.

edit: did the math. The maintainers I'm referring to in this thread have a shared weight of over 1000.000.000 downloads / month (not accounting for overlapping deps) - it'd be foolish to dismiss them

Member

yoshuawuyts commented Nov 5, 2016

If by maintaining you mean actually reading and fixing issues and not just sitting on them, then fixing the deprecation warning is a drop in the bucket, since it only takes a couple of minutes to run tests with --trace-deprecation and make some trivial replacements

@seishun your consistent downplay of maintainer's concerns regarding the amount of work this creates could be considered insensitive, borderline rude. Trust maintainers to be knowledgeable about how much overhead this creates for them. Dismissing them is NOT a stance that inspires faith in you and in turn Node core which you represent.

If we are to arrive at something resembling a consensus, we must take our time to understand where people with different opinions are coming from - failure to take people seriously with opinions different than your own is what makes situations escalate, and I'd like to think nobody in this thread is keen for that to happen.

edit: did the math. The maintainers I'm referring to in this thread have a shared weight of over 1000.000.000 downloads / month (not accounting for overlapping deps) - it'd be foolish to dismiss them

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Nov 6, 2016

Member

@yoshuawuyts your concerns are being taken seriously, and although I might not have expressed this in my comments, other collaborators clearly have (for instance, by raising the question about reverting #8169). What my previous comment objects to is @mafintosh's implicit generic statement that introducing deprecation warnings makes maintaining a large number of modules "unfeasible".

Trust maintainers to be knowledgeable about how much overhead this creates for them.

No offense, but it's imprudent to just "trust" the word of a couple maintainers out of thousands without any reasoning or evidence when personal experience implies otherwise. Rhetoric doesn't help either.

Also take into account that maintainers of "100s of modules" are a thin minority. The opinion of the ~90% who maintain <10 modules might vastly differ (I'm guessing "indifferent" as none of them have posted here as far as I know).

Member

seishun commented Nov 6, 2016

@yoshuawuyts your concerns are being taken seriously, and although I might not have expressed this in my comments, other collaborators clearly have (for instance, by raising the question about reverting #8169). What my previous comment objects to is @mafintosh's implicit generic statement that introducing deprecation warnings makes maintaining a large number of modules "unfeasible".

Trust maintainers to be knowledgeable about how much overhead this creates for them.

No offense, but it's imprudent to just "trust" the word of a couple maintainers out of thousands without any reasoning or evidence when personal experience implies otherwise. Rhetoric doesn't help either.

Also take into account that maintainers of "100s of modules" are a thin minority. The opinion of the ~90% who maintain <10 modules might vastly differ (I'm guessing "indifferent" as none of them have posted here as far as I know).

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Nov 6, 2016

Member

No offense, but it's imprudent to just "trust" the word of a couple maintainers out of thousands without any reasoning or evidence when personal experience implies otherwise. Rhetoric doesn't help either.

Also take into account that maintainers of "100s of modules" are a thin minority. The opinion of the ~90% who maintain <10 modules might vastly differ (I'm guessing "indifferent" as none of them have posted here as far as I know).

I receive issues every day that are not related to my own code. I try to respond to all of them, but it is a lot of work. My stuff is downloaded a million times a month, which is irrelevant compared to someone with 100s of millions of downloads a month. I would guess that the top 20% module maintainers are accountable for 80% of npm downloads (just guessing). Disrupting those people means disrupting almost all the community.

I recently received an email from someone asking for help, or he would lose his job. The dilemma is about prioritizing family or that guy. This is the reality that I live in, I have chosen it and I am happy about it.

All the buffer-related deprecation might not cause disruption when 4.5.0 fades out. If I want my module to support from v4 to v8, I must consider these API not present. More or less, this means April 2018. At some point, this disruption should happen, but definitely not in Node v8.
Yes, I can use the shim, that adds another download, when everyone is complaining about download sizes.

I think this issue is wrongly framed from the start:

People who haven't updated their code in 6 months are unlikely to do so ever - unless they are given another impetus.

This is stating let's beat module authors into this new api. It does not work this way, what you will create is more stressful situations, more people being annoyed, and it will result in less community engagement. I opened up nodeconf.eu 2016 with a bold statement: in this community there a no vips, everybody is at the same level and accessible. Creating more work for "busy" authors will make them less accessible to everybody else.

I am personally less willing to answer an issue if I have to write the same response hundreds of time (which I am already doing "thanks for reporting, would you like to send a PR?"). @seishun The term you used is "sitting on issues".

Member

mcollina commented Nov 6, 2016

No offense, but it's imprudent to just "trust" the word of a couple maintainers out of thousands without any reasoning or evidence when personal experience implies otherwise. Rhetoric doesn't help either.

Also take into account that maintainers of "100s of modules" are a thin minority. The opinion of the ~90% who maintain <10 modules might vastly differ (I'm guessing "indifferent" as none of them have posted here as far as I know).

I receive issues every day that are not related to my own code. I try to respond to all of them, but it is a lot of work. My stuff is downloaded a million times a month, which is irrelevant compared to someone with 100s of millions of downloads a month. I would guess that the top 20% module maintainers are accountable for 80% of npm downloads (just guessing). Disrupting those people means disrupting almost all the community.

I recently received an email from someone asking for help, or he would lose his job. The dilemma is about prioritizing family or that guy. This is the reality that I live in, I have chosen it and I am happy about it.

All the buffer-related deprecation might not cause disruption when 4.5.0 fades out. If I want my module to support from v4 to v8, I must consider these API not present. More or less, this means April 2018. At some point, this disruption should happen, but definitely not in Node v8.
Yes, I can use the shim, that adds another download, when everyone is complaining about download sizes.

I think this issue is wrongly framed from the start:

People who haven't updated their code in 6 months are unlikely to do so ever - unless they are given another impetus.

This is stating let's beat module authors into this new api. It does not work this way, what you will create is more stressful situations, more people being annoyed, and it will result in less community engagement. I opened up nodeconf.eu 2016 with a bold statement: in this community there a no vips, everybody is at the same level and accessible. Creating more work for "busy" authors will make them less accessible to everybody else.

I am personally less willing to answer an issue if I have to write the same response hundreds of time (which I am already doing "thanks for reporting, would you like to send a PR?"). @seishun The term you used is "sitting on issues".

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Nov 6, 2016

Member

@mcollina I don't doubt that maintainers of popular packages have a lot of work to do, what isn't clear to me is why some of them think that a deprecation warning would create an insurmountable overhead for them.

I would guess that the top 20% module maintainers are accountable for 80% of npm downloads (just guessing).

I haven't analyzed download stats, but the 80/20 rule does somewhat hold for package counts: 18% of maintainers maintain 64% of packages. However, these top 18% maintainers are everyone who maintains more than 4 packages. Maintainers with over 100 packages, who are purported to be particularly affected by the deprecation, make up 0.4% of all maintainers and their packages 12% of npm. (stats from here).

All the buffer-related deprecation might not cause disruption when 4.5.0 fades out. If I want my module to support from v4 to v8, I must consider these API not present.

Could you clarify why v4.5.0 is significant? It's an obsolete release, and it actually does contain the new Buffer APIs. (link)

The term you used is "sitting on issues".

I meant sitting on modules. A lot of modules either have few users or are simple enough that they don't receive any issues.

Member

seishun commented Nov 6, 2016

@mcollina I don't doubt that maintainers of popular packages have a lot of work to do, what isn't clear to me is why some of them think that a deprecation warning would create an insurmountable overhead for them.

I would guess that the top 20% module maintainers are accountable for 80% of npm downloads (just guessing).

I haven't analyzed download stats, but the 80/20 rule does somewhat hold for package counts: 18% of maintainers maintain 64% of packages. However, these top 18% maintainers are everyone who maintains more than 4 packages. Maintainers with over 100 packages, who are purported to be particularly affected by the deprecation, make up 0.4% of all maintainers and their packages 12% of npm. (stats from here).

All the buffer-related deprecation might not cause disruption when 4.5.0 fades out. If I want my module to support from v4 to v8, I must consider these API not present.

Could you clarify why v4.5.0 is significant? It's an obsolete release, and it actually does contain the new Buffer APIs. (link)

The term you used is "sitting on issues".

I meant sitting on modules. A lot of modules either have few users or are simple enough that they don't receive any issues.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Nov 6, 2016

Member

@mcollina I don't doubt that maintainers of popular packages have a lot of work to do, what isn't clear to me is why some of them think that a deprecation warning would create an insurmountable overhead for them.

Here is how it goes, let's imagine this chain of deps: A -> B -> C. Module C is the one that emits the warning, while module A is the one everybody uses. Module A maintainer will receive a lot of issues because C has not been updated yet. Even after C updates, B needs to update the dependency, which might require some manual work. In the worst case scenario All the authors of A, B, and C needs to update their packages to address a simple warning.
As you said, an hard deprecation will create pressure on the maintainer of A to fix both C and B. All those PRs would need to be reviewed etc. it takes a lot of time.

All the buffer-related deprecation might not cause disruption when 4.5.0 fades out. If I want my module to support from v4 to v8, I must consider these API not present.
Could you clarify why v4.5.0 is significant? It's an obsolete release, and it actually does contain the new Buffer APIs. (link)

Sorry, I meant 4.4.x. I still get bug reports on v0.10.29, because that's what is bundled in debian stable (https://packages.debian.org/search?keywords=nodejs). In the general case, v4 does not have those fixes.

Member

mcollina commented Nov 6, 2016

@mcollina I don't doubt that maintainers of popular packages have a lot of work to do, what isn't clear to me is why some of them think that a deprecation warning would create an insurmountable overhead for them.

Here is how it goes, let's imagine this chain of deps: A -> B -> C. Module C is the one that emits the warning, while module A is the one everybody uses. Module A maintainer will receive a lot of issues because C has not been updated yet. Even after C updates, B needs to update the dependency, which might require some manual work. In the worst case scenario All the authors of A, B, and C needs to update their packages to address a simple warning.
As you said, an hard deprecation will create pressure on the maintainer of A to fix both C and B. All those PRs would need to be reviewed etc. it takes a lot of time.

All the buffer-related deprecation might not cause disruption when 4.5.0 fades out. If I want my module to support from v4 to v8, I must consider these API not present.
Could you clarify why v4.5.0 is significant? It's an obsolete release, and it actually does contain the new Buffer APIs. (link)

Sorry, I meant 4.4.x. I still get bug reports on v0.10.29, because that's what is bundled in debian stable (https://packages.debian.org/search?keywords=nodejs). In the general case, v4 does not have those fixes.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Nov 6, 2016

Member

@mcollina right, this takes a lot of time, and I agree it's a painful point. But in nearly all cases the warning is just noise that can be ignored (either mentally or --no-deprecation), so it's not urgent. The important point is that the maintainer doesn't actually have to do anything during that time other than waiting and maybe closing occasional duplicate issues.

Sorry, I meant 4.4.x. I still get bug reports on v0.10.29, because that's what is bundled in debian stable (https://packages.debian.org/search?keywords=nodejs). In the general case, v4 does not have those fixes.

Neither v0.10.29 nor v4.4.x receive security fixes anymore. I'd argue that helping maintainers support insecure versions of Node.js isn't good for the ecosystem in the long term.

Member

seishun commented Nov 6, 2016

@mcollina right, this takes a lot of time, and I agree it's a painful point. But in nearly all cases the warning is just noise that can be ignored (either mentally or --no-deprecation), so it's not urgent. The important point is that the maintainer doesn't actually have to do anything during that time other than waiting and maybe closing occasional duplicate issues.

Sorry, I meant 4.4.x. I still get bug reports on v0.10.29, because that's what is bundled in debian stable (https://packages.debian.org/search?keywords=nodejs). In the general case, v4 does not have those fixes.

Neither v0.10.29 nor v4.4.x receive security fixes anymore. I'd argue that helping maintainers support insecure versions of Node.js isn't good for the ecosystem in the long term.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Nov 6, 2016

Member

@seishun I'm telling everyone to move. But the path of least resistance for module authors is to continue to support old version of Node for the time being. We try to avoid it because dropping support for old version of Node is definitely semver-major, and it will cause everybody else to do the work.

Member

mcollina commented Nov 6, 2016

@seishun I'm telling everyone to move. But the path of least resistance for module authors is to continue to support old version of Node for the time being. We try to avoid it because dropping support for old version of Node is definitely semver-major, and it will cause everybody else to do the work.

@seishun

This comment has been minimized.

Show comment
Hide comment
@seishun

seishun Nov 6, 2016

Member

We try to avoid it because dropping support for old version of Node is definitely semver-major, and it will cause everybody else to do the work.

If that's the only concern, then I'm not sure how postponing the hard-deprecation till April 2018 makes any difference. They'd still have to bump semver-major and cause everybody else to do the work, just a year later.

P.S. Somewhat unrelated, but I don't personally agree with your statement that dropping support for unsupported versions of Node.js should be semver-major. Making it such would only do a (questionable and short-term) service to users of unsupported versions of Node.js (aka users who absolutely must upgrade ASAP) and cause pain to everyone else, for whom this is effectively semver-patch.

Member

seishun commented Nov 6, 2016

We try to avoid it because dropping support for old version of Node is definitely semver-major, and it will cause everybody else to do the work.

If that's the only concern, then I'm not sure how postponing the hard-deprecation till April 2018 makes any difference. They'd still have to bump semver-major and cause everybody else to do the work, just a year later.

P.S. Somewhat unrelated, but I don't personally agree with your statement that dropping support for unsupported versions of Node.js should be semver-major. Making it such would only do a (questionable and short-term) service to users of unsupported versions of Node.js (aka users who absolutely must upgrade ASAP) and cause pain to everyone else, for whom this is effectively semver-patch.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Nov 7, 2016

Member

because that's what is bundled in debian stable (https://packages.debian.org/search?keywords=nodejs)

Side note: that package is not maintained in Debian and is completely unsupported.
Ref: https://www.debian.org/releases/jessie/amd64/release-notes/ch-information.html#libv8
That package should not be used.

Member

ChALkeR commented Nov 7, 2016

because that's what is bundled in debian stable (https://packages.debian.org/search?keywords=nodejs)

Side note: that package is not maintained in Debian and is completely unsupported.
Ref: https://www.debian.org/releases/jessie/amd64/release-notes/ch-information.html#libv8
That package should not be used.

@mafintosh

This comment has been minimized.

Show comment
Hide comment
@mafintosh

mafintosh Nov 7, 2016

Member

This isn't about supporting 0.10 or older versions of 4 - a choice that should be completely up to the module authors btw. This is about deprecating a substantial amount of modules on npm and basically showing that Node core doesn't take backwards compatibility seriously.

This isn't a change that should be postponed to v10. This is a change that should never happen and core should acknowledge that.

Member

mafintosh commented Nov 7, 2016

This isn't about supporting 0.10 or older versions of 4 - a choice that should be completely up to the module authors btw. This is about deprecating a substantial amount of modules on npm and basically showing that Node core doesn't take backwards compatibility seriously.

This isn't a change that should be postponed to v10. This is a change that should never happen and core should acknowledge that.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Nov 7, 2016

Member

@mafintosh, everyone, I have counterarguments to that, but they are not going to fit inside a comment. I am preparing a longer explanation, and it's going to be ready in a few days. I believe that we could continue this conversation then.

Member

ChALkeR commented Nov 7, 2016

@mafintosh, everyone, I have counterarguments to that, but they are not going to fit inside a comment. I am preparing a longer explanation, and it's going to be ready in a few days. I believe that we could continue this conversation then.

@kgryte

This comment has been minimized.

Show comment
Hide comment
@kgryte

kgryte Nov 8, 2016

This should never land. The primary focus for core should be stability above all else. This was part of the Node ethos which has been lost since the end of the Great Node/io.js Schism.

Because of several changes in Node core, I have been forced to spend many hours needlessly updating packages and code which I should not have to be updating, simply because someone wanted core to use some (and in almost all cases, unnecessary) ES2015+ feature.

The current predominant attitude by many contributing to Node core seems to be that breaking things is fine as long as tagged with a semver major. That is an okay attitude for userland packages, but not in the platform itself. What I, as a package author, want more than anything else is stability. Period.

In a way, semver has become weaponized, a cudgel used to "force users to update". I have seen this happen repeatedly in Node core and as embodied by this PR.

And frankly, because of the focus on supposed "innovation" and updating core with the "latest" and "greatest", I have lost trust in core, as I cannot trust that Node will provide a stable API. As a result, I try to avoid using core functionality and, where possible, would rather just write my own implementation over which I have full control.

And if Node core development keeps going down the "breaking stuff is fine" path, I foresee a future where many people will adopt a similar attitude to myself and avoid core altogether. I cannot see how Node can perceive this as desirable.

In short, if you want the ability to subclass, create a new API. Stability is not something which should be flippantly discarded.

kgryte commented Nov 8, 2016

This should never land. The primary focus for core should be stability above all else. This was part of the Node ethos which has been lost since the end of the Great Node/io.js Schism.

Because of several changes in Node core, I have been forced to spend many hours needlessly updating packages and code which I should not have to be updating, simply because someone wanted core to use some (and in almost all cases, unnecessary) ES2015+ feature.

The current predominant attitude by many contributing to Node core seems to be that breaking things is fine as long as tagged with a semver major. That is an okay attitude for userland packages, but not in the platform itself. What I, as a package author, want more than anything else is stability. Period.

In a way, semver has become weaponized, a cudgel used to "force users to update". I have seen this happen repeatedly in Node core and as embodied by this PR.

And frankly, because of the focus on supposed "innovation" and updating core with the "latest" and "greatest", I have lost trust in core, as I cannot trust that Node will provide a stable API. As a result, I try to avoid using core functionality and, where possible, would rather just write my own implementation over which I have full control.

And if Node core development keeps going down the "breaking stuff is fine" path, I foresee a future where many people will adopt a similar attitude to myself and avoid core altogether. I cannot see how Node can perceive this as desirable.

In short, if you want the ability to subclass, create a new API. Stability is not something which should be flippantly discarded.

@bnoordhuis

This comment has been minimized.

Show comment
Hide comment
@bnoordhuis

bnoordhuis Nov 8, 2016

Member

The primary focus for core should be stability above all else. This was part of the original philosophy of Node

Hah, no. Stability wasn't an overriding concern until node was already several years old. We didn't get serious about that until v0.8/v0.10.

I get that you feel stability should come first but you are committing the cardinal sin of rewriting history to fit your narrative.

Member

bnoordhuis commented Nov 8, 2016

The primary focus for core should be stability above all else. This was part of the original philosophy of Node

Hah, no. Stability wasn't an overriding concern until node was already several years old. We didn't get serious about that until v0.8/v0.10.

I get that you feel stability should come first but you are committing the cardinal sin of rewriting history to fit your narrative.

@jasnell jasnell added the blocked label Nov 8, 2016

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Nov 8, 2016

Member

Let me try again to make the status of this PR clear: There is no consensus to land this PR at the current time. The CTC discussed the matter several months ago now and decided that a runtime deprecation of the Buffer() constructor is not feasible or desirable given the ramifications to the ecosystem. For those of you who have weighed in saying that this should never land, trust that we have heard and noted your input and if and when this PR comes back up for consideration by the CTC again, we will absolutely be taking those considerations into account. It does absolutely no good whatsoever to rehash those comments or to demand that the CTC favor stability of all other considerations. Let me be even clearer: there was absolutely no one on the CTC who voiced the opinion that this PR should land at any point in the foreseeable future.

That said, we are not closing the PR because there is a possibility that we might decide to go down this route. Why would we do so? The only reason why would intentionally break the Buffer() constructor would be if it was absolutely certain that doing so would be in the larger ecosystems best interest -- which would be determined by evaluating the possible security, usability, and performance benefits of such a change weighed against the potential ecosystem impact.

Understand that those of us on the CTC are also Node.js users. These kinds of breaking changes affect us also. We are not keen on breaking our own code, much of which depends directly on modules that would be directly impacted by these changes. We do not make decisions to break APIs lightly.

Member

jasnell commented Nov 8, 2016

Let me try again to make the status of this PR clear: There is no consensus to land this PR at the current time. The CTC discussed the matter several months ago now and decided that a runtime deprecation of the Buffer() constructor is not feasible or desirable given the ramifications to the ecosystem. For those of you who have weighed in saying that this should never land, trust that we have heard and noted your input and if and when this PR comes back up for consideration by the CTC again, we will absolutely be taking those considerations into account. It does absolutely no good whatsoever to rehash those comments or to demand that the CTC favor stability of all other considerations. Let me be even clearer: there was absolutely no one on the CTC who voiced the opinion that this PR should land at any point in the foreseeable future.

That said, we are not closing the PR because there is a possibility that we might decide to go down this route. Why would we do so? The only reason why would intentionally break the Buffer() constructor would be if it was absolutely certain that doing so would be in the larger ecosystems best interest -- which would be determined by evaluating the possible security, usability, and performance benefits of such a change weighed against the potential ecosystem impact.

Understand that those of us on the CTC are also Node.js users. These kinds of breaking changes affect us also. We are not keen on breaking our own code, much of which depends directly on modules that would be directly impacted by these changes. We do not make decisions to break APIs lightly.

@Fishrock123

This comment has been minimized.

Show comment
Hide comment
@Fishrock123

Fishrock123 Nov 8, 2016

Member

fwiw I still think this should be a goal in the long run

Member

Fishrock123 commented Nov 8, 2016

fwiw I still think this should be a goal in the long run

@addaleax

This comment has been minimized.

Show comment
Hide comment
@addaleax

addaleax Nov 9, 2016

Member

I would like to ask everyone who has commented here to move the discussion to #9531, if only so the public discussion can take in a central place and not over multiple GH threads at once.

Member

addaleax commented Nov 9, 2016

I would like to ask everyone who has commented here to move the discussion to #9531, if only so the public discussion can take in a central place and not over multiple GH threads at once.

@seishun seishun changed the title from buffer: hard-deprecate Buffer constructor to buffer: runtime-deprecate Buffer constructor Feb 16, 2017

@jasnell jasnell added the stalled label Mar 1, 2017

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Mar 1, 2017

Member

@nodejs/ctc ... should we keep this open? I'm pretty sure the discussion has run it's course here.

Member

jasnell commented Mar 1, 2017

@nodejs/ctc ... should we keep this open? I'm pretty sure the discussion has run it's course here.

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR Mar 2, 2017

Member

@jasnell, I think that keeping a PR for this open makes sense, at least until we decide against that. The actual discussion is in #9531 now, but the PR is valuable by itself. So, -1 to closing this atm.

Member

ChALkeR commented Mar 2, 2017

@jasnell, I think that keeping a PR for this open makes sense, at least until we decide against that. The actual discussion is in #9531 now, but the PR is valuable by itself. So, -1 to closing this atm.

const bindingObj = {};
const internalUtil = require('internal/util');
class FastBuffer extends Uint8Array {}
const FastBuffer = (() => class Buffer extends Uint8Array {})();

This comment has been minimized.

@ChALkeR

ChALkeR Mar 12, 2017

Member

Could you remind why this change was needed? Which code breaks without it?

#11808 and #11806 are missing this, would everything be fine there?

@ChALkeR

ChALkeR Mar 12, 2017

Member

Could you remind why this change was needed? Which code breaks without it?

#11808 and #11806 are missing this, would everything be fine there?

This comment has been minimized.

@seishun

seishun Mar 12, 2017

Member

@ChALkeR it was necessary because the FastBuffer.prototype.constructor = Buffer; line was removed. It was removed to prevent Uint8Array methods (which call the constructor) from being affected by the deprecation.

@seishun

seishun Mar 12, 2017

Member

@ChALkeR it was necessary because the FastBuffer.prototype.constructor = Buffer; line was removed. It was removed to prevent Uint8Array methods (which call the constructor) from being affected by the deprecation.

This comment has been minimized.

@ChALkeR

ChALkeR Mar 12, 2017

Member

Ok, here is the testcase: Buffer.alloc(10).map(x => x + 1) (just for future reference).

@ChALkeR

ChALkeR Mar 12, 2017

Member

Ok, here is the testcase: Buffer.alloc(10).map(x => x + 1) (just for future reference).

This comment has been minimized.

@ChALkeR
@ChALkeR

ChALkeR Mar 12, 2017

Member

/cc @jasnell

This comment has been minimized.

@jasnell

jasnell Mar 12, 2017

Member

I'd been trying to remember where I had seen this :-)

@jasnell

jasnell Mar 12, 2017

Member

I'd been trying to remember where I had seen this :-)

@ChALkeR

This comment has been minimized.

Show comment
Hide comment
@ChALkeR

ChALkeR May 4, 2017

Member

This was mostly superseded by #11968, and there is no concrete path forward runtime-deprecating it without a flag atm, and probably in another half a year.
Also, the PR would have to look different since #11968 landed, and the most important bits (those that make sure that typed arrays aren't broken) have been incorporated there.

Perhaps it's time to close this and revisit later, probably in another PR. /ping @jasnell — could you confirm?

@seishun thanks on the work here! It was important, and helped to reach the current state of runtime-deprecated-under-a-flag state of things.

Member

ChALkeR commented May 4, 2017

This was mostly superseded by #11968, and there is no concrete path forward runtime-deprecating it without a flag atm, and probably in another half a year.
Also, the PR would have to look different since #11968 landed, and the most important bits (those that make sure that typed arrays aren't broken) have been incorporated there.

Perhaps it's time to close this and revisit later, probably in another PR. /ping @jasnell — could you confirm?

@seishun thanks on the work here! It was important, and helped to reach the current state of runtime-deprecated-under-a-flag state of things.

@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell May 4, 2017

Member

Yeah, I think this should just be closed at this point. We can revisit it later.

Member

jasnell commented May 4, 2017

Yeah, I think this should just be closed at this point. We can revisit it later.

@jasnell jasnell closed this May 4, 2017

@ChALkeR ChALkeR removed the stalled label May 4, 2017

@seishun seishun referenced this pull request Sep 11, 2017

Closed

buffer: runtime-deprecate Buffer ctor by default #15346

3 of 4 tasks complete

@ChALkeR ChALkeR referenced this pull request Jul 11, 2018

Open

Password Hashing API #21766

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment