Process 100, 102-199 status codes according to specs. #18033

miles-po · 2018-01-08T08:33:39Z

Relevant references:
RFC7231 Section 6.2.1
RFC2518
RFC8297

This affects several modules downstream that use the http
module, e.g., node-fetch, all of whom violate HTTP RFCs
due to this module. As such, this could introduce a
breaking change for downstream if HTTP standards were
ignored in an ad-hoc fashion.

101 Upgrade is excluded due to its non-informational
processing according to RFC7231, Section 6.2.2.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

http, http2, https

miles-po · 2018-01-08T08:36:53Z

I did not include documentation as I believe the RFCs cover this adequately, and the aim was to match the specs where Node was currently non-conforming.

I admit that the tests are a bit opaque to me. I tried to find a good template for 100 Continue behavior, but it was unclear whether server or client was being tested or where the status code was being provided. Any help in this area is appreciated.

jasnell · 2018-01-08T17:40:13Z

Generally +1 on this but will need to give it a more thorough review. I'm wondering if this has to be semver-major tho as I can definitely see the possibility of this breaking at least someone.

Ping @nodejs/http and @nodejs/tsc, please take a look

TimothyGu

+1 on this change, but it needs a test.

I'd say semver-major is appropriate for this, given people may be depending on the existing (broken) behavior.

TimothyGu · 2018-01-08T17:43:44Z

lib/_http_client.js

    socket.removeListener('data', socketOnData);
    socket.removeListener('end', socketOnEnd);
    freeParser(parser, req, socket);
  }
 }

+// client
+function informational(status) {


nit: name this statusIsInformational to be a bit more explicit.

mcollina · 2018-01-08T18:10:11Z

👍 for semver-major.

I'm fine with the change itself, but it needs tests.

miles-po · 2018-01-08T18:14:38Z

@TimothyGu As mentioned earlier, I hit a stumbling block with regard to tests. Node's test setup is not familiar to me and the test code for 100s that I'd use as a template is similarly opaque. I'll help any way I can, but I'm blocked by my lack of familiarity at the moment.

BridgeAR · 2018-02-02T12:03:51Z

@miles-po you can run the tests with make -j4 test (check the Makefile for different possible tasks). It is also possible to run tests individually by e.g. python tools/test.py test/parallel/test-http-YOUR_TEST_NAME.js. If you use Windows, please check the vcbuild.bat.

You can write the test just the same as you would write code anywhere else that you would expect to succeed in a specific way. The only differences is that you have to include require('../common'); as first statement and that you have access to more Node.js internal helper functions (e.g. the ones found in common).

Fishrock123 · 2018-02-02T19:39:52Z

I think it is about time we plan to do this in a near-term major version.

addaleax · 2018-02-02T19:41:52Z

@Fishrock123 What are you saying? That this PR should go into 10.0.0?

bnoordhuis · 2018-02-03T10:39:46Z

lib/_http_client.js

+  // to the content-length of the entity-body had the request
+  // been a GET.
+  var isHeadResponse = req.method === 'HEAD';
+  debug('AGENT isHeadResponse', isHeadResponse);


I removed this a while ago, see #17806 (comment). Can you do the same?

It seems like this got back in / was not removed?

bnoordhuis · 2018-02-03T10:40:03Z

lib/_http_client.js

+  debug('AGENT isHeadResponse', isHeadResponse);
+
+  if (statusIsInformational(res.statusCode)) {
+    // restart the parser, as this is a 1xx informational message.


Nit: please capitalize comments.

miles-po · 2018-02-04T07:40:13Z

Added test for 102 Processing. Allows for multiple 1xx responses. Updated docs.

Added a new method to ServerResponse: writeProcessing. This is similar to writeContinue. Had considered making a more generic writeInformation to handle all cases, but 102 does not have headers (like 100) while the currently experimental 103 Extra Hints is useless without headers.

The continue event is specific to 100 Continue and I didn't think should be reused for other information statuses. I created the information event to be triggered for any 1xx status. The information event passes an object with statusCode to distinguish what triggered it. This could be used for 100 Continue as well. When 103 is eventually handled explicitly, passing header information or other relevant information should be straightforward.

Open questions I had about 103 were whether to allow headers to be passed to it like writeHead and whether headers would be shared implicitly through to the main response. These questions however are out of scope for this particular PR but may affect the API.

As this is semver-major, we're in a good time to allow for discussion on the API without harsh deadlines. Suggestions welcome.

mcollina

The title of this PR is about "102-199" status code, but I see code to handle 102 only.

Regarding 103, I would emit the headers as part of the event. I would not mesh them with the proper response header.

I'm not really fond of the 'information' name, but I don't want to bikeshed it either.

mcollina · 2018-02-05T09:48:50Z

test/parallel/test-http-information-processing.js

+// NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+// DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+// USE OR OTHER DEALINGS IN THE SOFTWARE.


The header is not needed.

mcollina · 2018-02-05T09:51:16Z

lib/_http_server.js

+function _writeInformation(statusCode, statusMessage, cb) {
+  this._writeRaw(`HTTP/1.1 ${statusCode} ${statusMessage}${CRLF}${CRLF}`,
+                 'ascii', cb);
+}


can you make writeInformation as a top level function? There is no need to attach it to the prototype.
Another option is to just call _writeRaw in both cases, I do not see much harm in that duplication.

mcollina · 2018-02-05T09:51:43Z

test/parallel/test-http-information-processing.js

+      assert.strictEqual(body, test_res_body, 'Response body doesn\'t match.');
+      assert.ok('abcd' in res.headers, 'Response headers missing.');
+      server.close();
+      process.exit();


is this process.exit() needed? Can we remove it?

mcollina · 2018-02-05T09:58:55Z

doc/api/http.md

@@ -400,6 +400,14 @@ Emitted when the server sends a '100 Continue' HTTP response, usually because
 the request contained 'Expect: 100-continue'. This is an instruction that
 the client should send the request body.

+### Event: 'information'
+<!-- YAML
+added: v10.0.0


This should be REPLACEME. It will be replaced upon release.

miles-po · 2018-02-05T17:38:56Z

The title of the PR is accurate. While only 102 is supported explicitly in the server, the client now allows for any informational message to be passed without ignoring the final response, which was the original intention of this patch. Unfortunately, there was no way to actually test this without added some basic 102 functionality to the server.

I had considered adding 103 support, but as that RFC is still on the "Experimental" track, I decided against it. I guess that was my original thought behind adding that new method, _writeInformation. That it would serve as a launching point for further work in that area. But as you said, it just adds noise to the prototype, so I'm happy to remove it again.

Since 104-199 are not yet defined at all, I thought it best to have the client treat it like one would choose any informational message (allowing for folks with non-standard usage of these codes), but does not affect the client API at all. On the server side, this was not possible. An event can be passed, but how to actually send the headers is not well defined in the RFC.

Should all headers defined so far be sent, or just a user-defined subset? When the headers for the 103 are sent, should those be passed on to next response by default or should this be explicit? What about second or third 103 responses? Should they resend all defined headers or just the changed ones? Should two or three (or ten) 103 messages be allowed? Should there be a limit?

Rather than make a blind stab on this experimental RFC, I punted for now and just concentrated on the specs I could verify.

Comments/suggestions welcome of course.

mcollina

LGTM

jasnell · 2018-02-05T17:43:26Z

doc/api/http.md

+
+Emitted when the server sends a 1xx response (excluding 101 Upgrade). This
+event is emitted with a object, which can be used to retrieve the status code.
+


This change should also be added to the http2 compat API

Should also explain why 101 Upgrade is not included in this

Yeah, good spot.

Also an example illustrating what "This event is emitted with a object" means would be helpful.

Quick question: Should 101s actually fire the event too?

I excluded 101 Upgrade since this was typically used to make TLS upgrades (rare) or Web Socket connections (typical). I don't work on the Socket.IO project—for example—so I don't know the potential drawbacks. I suppose performance wouldn't suffer measurably just from firing an event with no listeners.

I take that back. 101 Upgrade messages are handled so differently, they can break the request/response model dramatically. Web Sockets being a prime example but TLS as well. Leaving as is.

mcollina · 2018-02-05T17:48:10Z

CI: https://ci.nodejs.org/job/node-test-pull-request/12951/

BridgeAR · 2018-02-06T22:24:40Z

New CI https://ci.nodejs.org/job/node-test-pull-request/12979/

BridgeAR · 2018-02-09T02:59:55Z

@miles-po would you be so kind and rebase this PR and force-push-with-lease afterwards? Otherwise it is not possible to start the CI.

Relevant references: RFC7231 Section 6.2.1 RFC2518 RFC8297 This affects several modules downstream that use the http module, e.g., node-fetch, all of whom violate HTTP RFCs due to this module. As such, this could introduce a breaking change for downstream if HTTP standards were ignored in an ad-hoc fashion. 101 Upgrade is excluded due to its non-informational processing according to RFC7231, Section 6.2.2.

Lint is clean.

Must be distinct from `continue` event due to slightly different behavior/triggering rules from 102/103 status codes. For example, 103 is explicitly made to take headers for pre-processing while 100 status codes never include headers.

miles-po · 2018-02-12T16:24:57Z

Not clear to me where those changes for http2 are to be made. I had assumed (obviously incorrectly) that http2 followed the underscore-prefixed functionality. Could someone narrow down the search please? A line number would suffice.

mcollina · 2018-02-12T16:55:31Z

Most of the http2 code lives in:

https://github.com/nodejs/node/blob/master/lib/internal/http2/core.js
https://github.com/nodejs/node/blob/master/lib/internal/http2/compat.js

I think you should add the functionality to https://github.com/nodejs/node/blob/master/lib/internal/http2/core.js#L1447 (and descending classes, the Server and Client pairs). Also the compat API: https://github.com/nodejs/node/blob/master/lib/internal/http2/compat.js#L212 which will need to surface the information as well.

TimothyGu · 2018-02-12T17:38:56Z

I'm okay with separating the HTTP/2 things into a new PR or commit. This will allow easier backporting to releases that do not have the http2 module.

mcollina · 2018-02-12T17:44:06Z

👍to two PRs. They can also be two separate commits.
If that's the approach, the docs for http2 should not be changed in this PR.

HTTP/2 changes will be done in a separate PR.

BridgeAR · 2018-02-12T19:50:21Z

lib/_http_client.js

    socket.removeListener('data', socketOnData);
    socket.removeListener('end', socketOnEnd);
    freeParser(parser, req, socket);
  }
 }

+// client


I am not sure what this comment stands for.

I do not either. I was following what looked like existing convention in this source file. Removing.

BridgeAR · 2018-02-12T19:51:42Z

lib/_http_client.js

+  // to the content-length of the entity-body had the request
+  // been a GET.
+  var isHeadResponse = req.method === 'HEAD';
+  debug('AGENT isHeadResponse', isHeadResponse);


It seems like this got back in / was not removed?

BridgeAR · 2018-02-12T19:55:32Z

test/parallel/test-http-information-processing.js

+    res.on('end', function() {
+      console.error('Got full response.');
+      assert.strictEqual(body, test_res_body, 'Response body doesn\'t match.');
+      assert.ok('abcd' in res.headers, 'Response headers missing.');


Leaving out the error message is actually better because in case of an error it would say:
The expression assert.ok('abcd' in res.headers) evaluated to a falsy value.

BridgeAR · 2018-02-12T19:56:09Z

test/parallel/test-http-information-processing.js

+    res.on('data', function(chunk) { body += chunk; });
+    res.on('end', function() {
+      console.error('Got full response.');
+      assert.strictEqual(body, test_res_body, 'Response body doesn\'t match.');


Not having any message will probably be even better because it will show what values got in.

BridgeAR · 2018-02-12T20:00:31Z

test/parallel/test-http-information-processing.js

+  });
+
+  req.on('response', function(res) {
+    assert.strictEqual(processing_count, 2,


This could use common/countdown.js. That would be more intuitive.

BridgeAR · 2018-02-12T20:02:47Z

doc/api/http.md

+  path: '/length_request'
+};
+
+// make a request


Non-blocking nit: Using capital letters as first character would be great :-) And maybe also a dot at the end.

* While rebasing, old content resurfaced. Re-removed. * Removed errant 'client' comment * Removed assert descriptions * Used common/countdown instead of simple int * Capitalized initial letter in example comment

miles-po · 2018-02-12T22:59:17Z

Addressed comments.

BridgeAR

LGTM with a nit.

BridgeAR · 2018-02-12T23:11:32Z

test/parallel/test-http-information-processing.js


 const test_res_body = 'other stuff!\n';
-let processing_count = 0;
+const countdown = new Countdown(3, common.mustCall(() => server.close()));


The Countdown is already a mustCall the wrapping is therefore not necessary :-)

BridgeAR · 2018-02-12T23:39:20Z

@mcollina PTAL

mcollina

LGTM

mcollina · 2018-02-14T00:21:18Z

CI: https://ci.nodejs.org/job/node-test-pull-request/13157/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1289/

miles-po · 2018-02-14T01:21:00Z

Failing on the markdown linter for docs (which is not invoked by make -j4 test). Looks like it's only installed after you run make link-md-build. I don't think this is mentioned in the contrib docs. (Or at least I didn't see it.)

Sorry about the hiccup. Markdown warning fixed.

mcollina · 2018-02-14T01:23:26Z

CI: https://ci.nodejs.org/job/node-test-pull-request/13161/

miles-po · 2018-02-14T01:28:05Z

This is in the failing test
java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1604_sharedlibs_openssl102_x64/2233/console

mcollina · 2018-02-15T10:58:30Z

Landing.

mcollina · 2018-02-15T11:20:57Z

Landed as baf8495.

Adding ServerResponse.writeProcessing to send 102 status codes. Added an `'information'` event to ClientRequest to handle 1xx status codes except 101 Upgrade. 101 Upgrade is excluded due to its non-informational processing according to RFC7231, Section 6.2.2. This affects several modules downstream that use the http module, e.g., node-fetch, all of whom violate HTTP RFCs due to this module. As such, this could introduce a breaking change for downstream if HTTP standards were ignored in an ad-hoc fashion. See also RFC2518 RFC8297. PR-URL: #18033 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>

Adding ServerResponse.writeProcessing to send 102 status codes. Added an `'information'` event to ClientRequest to handle 1xx status codes except 101 Upgrade. 101 Upgrade is excluded due to its non-informational processing according to RFC7231, Section 6.2.2. This affects several modules downstream that use the http module, e.g., node-fetch, all of whom violate HTTP RFCs due to this module. As such, this could introduce a breaking change for downstream if HTTP standards were ignored in an ad-hoc fashion. See also RFC2518 RFC8297. PR-URL: nodejs#18033 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>

nodejs-github-bot added the http Issues or PRs related to the http subsystem. label Jan 8, 2018

This was referenced Jan 8, 2018

node does not process HTTP informational responses correctly #9282

Closed

"102 Processing" not following the spec node-fetch/node-fetch#377

Closed

TimothyGu reviewed Jan 8, 2018

View reviewed changes

jasnell added the semver-major PRs that contain breaking changes and should be released in the next major version. label Jan 8, 2018

maclover7 force-pushed the master branch from bb5575a to 993b716 Compare January 26, 2018 22:02

cjihrig force-pushed the master branch from 993b716 to 082f952 Compare January 26, 2018 22:36

addaleax added this to the 10.0.0 milestone Feb 2, 2018

bnoordhuis reviewed Feb 3, 2018

View reviewed changes

mcollina reviewed Feb 5, 2018

View reviewed changes

mcollina approved these changes Feb 5, 2018

View reviewed changes

jasnell reviewed Feb 5, 2018

View reviewed changes

miles-po added 6 commits February 10, 2018 10:30

Cleaning up status code comment to be clearer.

fba79a8

Cleaning up code to reuse informational status check.

58bb0ca

Lint is clean.

Fine tuning comment change.

79301fa

Renamed 'informational' function to be more explicit.

b2c81fc

Firing event for information messages

fd33e98

Must be distinct from `continue` event due to slightly different behavior/triggering rules from 102/103 status codes. For example, 103 is explicitly made to take headers for pre-processing while 100 status codes never include headers.

Removed HTTP/2 doc updates

130e729

HTTP/2 changes will be done in a separate PR.

BridgeAR reviewed Feb 12, 2018

View reviewed changes

Fixed rebase errors and nits

89e4552

* While rebasing, old content resurfaced. Re-removed. * Removed errant 'client' comment * Removed assert descriptions * Used common/countdown instead of simple int * Capitalized initial letter in example comment

BridgeAR approved these changes Feb 12, 2018

View reviewed changes

Unwrapped redundant mustCall(...)

1eee678

BridgeAR requested review from jasnell, addaleax, cjihrig and joyeecheung February 12, 2018 23:39

jasnell approved these changes Feb 13, 2018

View reviewed changes

mcollina approved these changes Feb 14, 2018

View reviewed changes

Fixed markdown typo; removed warning

1c2e86d

TimothyGu approved these changes Feb 14, 2018

View reviewed changes

mcollina closed this Feb 15, 2018

gareth-robinson mentioned this pull request May 14, 2018

test-gzip now failing when using node 10.1 request/request#2940

Closed

flotwig mentioned this pull request Jul 29, 2019

Electron v5.0.10 cypress-io/cypress#4720

Merged


		Emitted when the server sends a 1xx response (excluding 101 Upgrade). This
		event is emitted with a object, which can be used to retrieve the status code.

Process 100, 102-199 status codes according to specs. #18033

Process 100, 102-199 status codes according to specs. #18033

Conversation

miles-po commented Jan 8, 2018 • edited Loading

Checklist

Affected core subsystem(s)

miles-po commented Jan 8, 2018 • edited Loading

jasnell commented Jan 8, 2018

TimothyGu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Jan 8, 2018 • edited Loading

miles-po commented Jan 8, 2018

BridgeAR commented Feb 2, 2018

Fishrock123 commented Feb 2, 2018

addaleax commented Feb 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miles-po commented Feb 4, 2018

mcollina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miles-po commented Feb 5, 2018

mcollina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Feb 5, 2018

BridgeAR commented Feb 6, 2018

BridgeAR commented Feb 9, 2018

miles-po commented Feb 12, 2018

mcollina commented Feb 12, 2018

TimothyGu commented Feb 12, 2018 • edited Loading

mcollina commented Feb 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miles-po commented Feb 12, 2018

BridgeAR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BridgeAR commented Feb 12, 2018

mcollina left a comment

Choose a reason for hiding this comment

mcollina commented Feb 14, 2018 • edited Loading

miles-po commented Feb 14, 2018

mcollina commented Feb 14, 2018

miles-po commented Feb 14, 2018 • edited Loading

mcollina commented Feb 15, 2018

mcollina commented Feb 15, 2018

miles-po commented Jan 8, 2018 •

edited

Loading

miles-po commented Jan 8, 2018 •

edited

Loading

mcollina commented Jan 8, 2018 •

edited

Loading

TimothyGu commented Feb 12, 2018 •

edited

Loading

mcollina commented Feb 14, 2018 •

edited

Loading

miles-po commented Feb 14, 2018 •

edited

Loading