buffer,string_decoder: consolidate encoding validation logic #7207

jasnell · 2016-06-07T23:07:52Z

Checklist

make -j4 test (UNIX) or vcbuild test nosign (Windows) passes
documentation is changed or added
the commit message follows commit guidelines

Affected core subsystem(s)

buffer, string_decoder

Description of change

Buffer.isEncoding and string_decoder.normalizeEncoding shared
quite a bit of logic. This moves the primary logic into
internal/util. The userland modules that monkey patch Buffer.isEncoding
should still work.

@nodejs/buffer @mscdex

mscdex · 2016-06-07T23:16:28Z

I don't think this will work for StringDecoder since some libraries override Buffer.isEncoding(), so we need to check the result of that function if we don't match a known internal encoding. Doing that as-is with these changes will mean executing the normalization function twice in the case that Buffer.isEncoding() isn't overridden.

jasnell · 2016-06-07T23:20:02Z

Ok, let me take one more stab at it...

trevnorris · 2016-06-07T23:26:22Z

lib/buffer.js

-    }
-  }
+  if (!internalUtil)
+    internalUtil = require('internal/util');


Just include it at the top of the file. This is already being required by other modules prior to this point so there's no savings.

jasnell · 2016-06-07T23:29:39Z

@mscdex ... ok, how does that look now?

mscdex · 2016-06-07T23:37:28Z

lib/internal/util.js

+        return enc;
+      default:
+        if (low)
+          return undefined;


minor nit: this can be simplified to just return;

Yeah I did it this way just to make it more explicit. Either way works for me tho

mscdex · 2016-06-07T23:40:48Z

LGTM except for minor nit. Should probably run this through citgm though just to be extra safe.

jasnell · 2016-06-08T04:46:40Z

CI: https://ci.nodejs.org/job/node-test-pull-request/2952/ ... Green except for unrelated build bot failure
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/300/ ... Green

mscdex · 2016-06-08T06:57:44Z

lib/internal/util.js

@@ -114,8 +114,7 @@ exports.normalizeEncoding = function normalizeEncoding(enc) {
      case 'hex':
        return enc;
      default:
-        if (low)
-          return undefined;
+        if (low) return; // undefined


If we're going to add the comment we may as well just use return undefined; ;-)

trevnorris · 2016-06-08T11:11:02Z

lib/buffer.js

-        loweredCase = true;
-    }
-  }
+Buffer[Symbol.for('isEncoding')] = Buffer.isEncoding = function(encoding) {


I'd say either

Prefix the Symbol to help prevent collisions (e.g. node.isEncoding)

Create and export this symbol from internal/util.js so it's no longer global
I lean towards the later.

jasnell · 2016-06-08T13:52:25Z

@trevnorris @mscdex ... updated and added a couple of tests. It turns out that we were not previously testing or verifying that Buffer.isEncoding() with non-string values would return false.

trevnorris · 2016-06-11T05:08:45Z

LGTM

mscdex · 2016-06-11T05:25:30Z

LGTM if CI is still ok: https://ci.nodejs.org/job/node-test-pull-request/2988/

jasnell · 2016-06-12T00:29:49Z

That CI run had a couple buildbot issues. Running again just to be safe
https://ci.nodejs.org/job/node-test-pull-request/2991/

Buffer.isEncoding and string_decoder.normalizeEncoding shared quite a bit of logic. This moves the primary logic into internal/util. The userland modules that monkey patch Buffer.isEncoding should still work.

jasnell · 2016-06-21T15:52:28Z

New CI before landing... just in case.. https://ci.nodejs.org/job/node-test-pull-request/3038/

Buffer.isEncoding and string_decoder.normalizeEncoding shared quite a bit of logic. This moves the primary logic into internal/util. The userland modules that monkey patch Buffer.isEncoding should still work. PR-URL: #7207 Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

jasnell · 2016-06-21T16:29:59Z

Landed in 6dd093d

thefourtheye · 2016-06-21T16:58:20Z

Returning different types of values from a function could have been avoided :(

mscdex · 2016-06-21T17:00:20Z

@thefourtheye What do you mean?

thefourtheye · 2016-06-21T18:13:19Z

@mscdex normalizeEncoding returns string and undefined based on the inputs, right? I was thinking, this could have been avoided.

mscdex · 2016-06-21T18:15:42Z

@thefourtheye How could it have been avoided?

thefourtheye · 2016-06-21T18:35:55Z

@mscdex maybe we could have returned an empty string instead of undefined?

mscdex · 2016-06-21T18:53:31Z

@thefourtheye Returning undefined or an empty string both have their pros and cons so there's really no difference IMHO, and I believe any performance hit from not consistently returning a string is likely to be negligible in this case.

Fishrock123 · 2016-06-27T14:39:49Z

seems to have conflicts on v6

jasnell · 2016-06-27T15:18:25Z

@Fishrock123 ... this PR builds on @mscdex's work in #6777. This can be backported to v6 if #6777 is. I don't believe a don't-land label is appropriate but we might need a better way of tracking dependencies between PRs.

mscdex · 2016-06-27T16:52:33Z

@jasnell That PR is already in v6, it's actually in v6.2.1.

MylesBorins · 2016-07-11T23:07:08Z

I put the don't land label for v4.x please correct if wrong

steal a line from nodejs#7207 to make things work

steal a line from nodejs#7207 to make things work PR-URL: nodejs#8437 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

Buffer.isEncoding and string_decoder.normalizeEncoding shared quite a bit of logic. This moves the primary logic into internal/util. The userland modules that monkey patch Buffer.isEncoding should still work. PR-URL: nodejs#7207 Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Trevor Norris <trev.norris@gmail.com>

Buffer.isEncoding and string_decoder.normalizeEncoding shared quite a bit of logic. This moves the primary logic into internal/util. The userland modules that monkey patch Buffer.isEncoding should still work. PR-URL: #7207 Reviewed-By: Brian White <mscdex@mscdex.net> Reviewed-By: Trevor Norris <trev.norris@gmail.com> Refs: #8463 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

Due to code consolidation in nodejs#7207 the isEncoding function got less strict. This commit makes sure isEncoding returns false for empty strings as before the consolidation. PR-URL: nodejs#18790 Refs: nodejs#7207 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

jasnell added buffer Issues and PRs related to the buffer subsystem. string_decoder Issues and PRs related to the string_decoder subsystem. labels Jun 7, 2016

nodejs-github-bot added the util Issues and PRs related to the built-in util module. label Jun 7, 2016

trevnorris reviewed Jun 7, 2016
View reviewed changes

mscdex reviewed Jun 7, 2016
View reviewed changes

mscdex reviewed Jun 8, 2016
View reviewed changes

RReverser mentioned this pull request Jun 8, 2016

buffer: speed up Buffer.isEncoding() method #5256

Closed

trevnorris reviewed Jun 8, 2016
View reviewed changes

buffer,string_decoder: consolidate encoding validation logic

7b459a9

Buffer.isEncoding and string_decoder.normalizeEncoding shared quite a bit of logic. This moves the primary logic into internal/util. The userland modules that monkey patch Buffer.isEncoding should still work.

jasnell force-pushed the normalized-encoding branch from 873a028 to 7b459a9 Compare June 21, 2016 15:50

jasnell closed this Jun 21, 2016

Fishrock123 added the dont-land-on-v6.x label Jun 27, 2016

MylesBorins mentioned this pull request Jul 11, 2016

Audit commits not found on v4.x-staging #7661

Closed

MylesBorins added the dont-land-on-v4.x label Jul 11, 2016

addaleax added a commit to addaleax/node that referenced this pull request Sep 7, 2016

[squash] fixup for v6 by adding missing require()

763e8b9

steal a line from nodejs#7207 to make things work

Fishrock123 mentioned this pull request Sep 8, 2016

crypto: fix ucs2/ucs-2/utf16le/utf-16le encoding check #8301

Closed

3 tasks

addaleax mentioned this pull request Sep 9, 2016

v6.x backport: buffer,string_decoder: consolidate encoding validation logic #8463

Closed

BridgeAR mentioned this pull request Feb 15, 2018

buffer: improve fill & normalizeEncoding performance #18790

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buffer,string_decoder: consolidate encoding validation logic #7207

buffer,string_decoder: consolidate encoding validation logic #7207

jasnell commented Jun 7, 2016

mscdex commented Jun 7, 2016

jasnell commented Jun 7, 2016

trevnorris Jun 7, 2016

jasnell commented Jun 7, 2016

mscdex Jun 7, 2016

jasnell Jun 7, 2016

mscdex commented Jun 7, 2016 •

edited

Loading

jasnell commented Jun 8, 2016 •

edited

Loading

mscdex Jun 8, 2016

trevnorris Jun 8, 2016 •

edited

Loading

jasnell commented Jun 8, 2016

trevnorris commented Jun 11, 2016

mscdex commented Jun 11, 2016

jasnell commented Jun 12, 2016 •

edited

Loading

jasnell commented Jun 21, 2016

jasnell commented Jun 21, 2016

thefourtheye commented Jun 21, 2016

mscdex commented Jun 21, 2016

thefourtheye commented Jun 21, 2016

mscdex commented Jun 21, 2016

thefourtheye commented Jun 21, 2016 •

edited

Loading

mscdex commented Jun 21, 2016

Fishrock123 commented Jun 27, 2016

jasnell commented Jun 27, 2016

mscdex commented Jun 27, 2016

MylesBorins commented Jul 11, 2016

buffer,string_decoder: consolidate encoding validation logic #7207

buffer,string_decoder: consolidate encoding validation logic #7207

Conversation

jasnell commented Jun 7, 2016

Checklist

Affected core subsystem(s)

Description of change

mscdex commented Jun 7, 2016

jasnell commented Jun 7, 2016

trevnorris Jun 7, 2016

Choose a reason for hiding this comment

jasnell commented Jun 7, 2016

mscdex Jun 7, 2016

Choose a reason for hiding this comment

jasnell Jun 7, 2016

Choose a reason for hiding this comment

mscdex commented Jun 7, 2016 • edited Loading

jasnell commented Jun 8, 2016 • edited Loading

mscdex Jun 8, 2016

Choose a reason for hiding this comment

trevnorris Jun 8, 2016 • edited Loading

Choose a reason for hiding this comment

jasnell commented Jun 8, 2016

trevnorris commented Jun 11, 2016

mscdex commented Jun 11, 2016

jasnell commented Jun 12, 2016 • edited Loading

jasnell commented Jun 21, 2016

jasnell commented Jun 21, 2016

thefourtheye commented Jun 21, 2016

mscdex commented Jun 21, 2016

thefourtheye commented Jun 21, 2016

mscdex commented Jun 21, 2016

thefourtheye commented Jun 21, 2016 • edited Loading

mscdex commented Jun 21, 2016

Fishrock123 commented Jun 27, 2016

jasnell commented Jun 27, 2016

mscdex commented Jun 27, 2016

MylesBorins commented Jul 11, 2016

mscdex commented Jun 7, 2016 •

edited

Loading

jasnell commented Jun 8, 2016 •

edited

Loading

trevnorris Jun 8, 2016 •

edited

Loading

jasnell commented Jun 12, 2016 •

edited

Loading

thefourtheye commented Jun 21, 2016 •

edited

Loading