http2: Introducing http/2 #14239

jasnell · 2017-07-14T19:57:36Z

At long last: The initial experimental implementation of HTTP/2.

This is an accumulation of the work that has been done in the nodejs/http2
repository, squashed down to a couple of commits. The original commit
history has been preserved in the nodejs/http2 repository.

Contributors to this work include:

Anna Henningsen / @addaleax
Colin Ihrig / @cjihrig
Daniel Bevenius / @danbev
James M Snell / @jasnell
Jun Mukai / @jmuk
Kelvin Jin / @kjin
Matteo Collina / @mcollina
Robert Kowalski / @robertkowalski
Santiago Gimeno / @santigimeno
Sebastiaan Deckers / @sebdeckers
Yosuke Furukawa / @yosuke-furukawa

This PR introduces the nghttp2 C library as a new dependency. This library
provides the majority of the HTTP/2 protocol implementation, with the rest
of the code here providing the mapping of the library into a usable JS API.

Within src, a handful of new node_http2_.c and node_http2_.h files are
introduced. These provide the internal mechanisms that interface with nghttp
and define the process.binding('http2') interface.

The JS API is defined within internal/http2/*.js.

There are two APIs provided: Core and Compat.

The Core API is HTTP/2 specific and is designed to be as minimal and as
efficient as possible.

The Compat API is intended to be as close to the existing HTTP/1 API as
possible, with some exceptions.

Tests, documentation and initial benchmarks are included.

The http2 module is gated by a new --expose-http2 command line flag.
When used, require('http2') will be exposed to users. Note that there
is an existing http2 module on npm that would be impacted by the introduction
of this module, which is the main reason for gating this behind a flag.

When using require('http2') the first time, a process warning will be
emitted indicating that an experimental feature is being used.

To run the benchmarks, the h2load tool (part of the nghttp project) is
required: ./node benchmarks/http2/simple.js benchmarker=h2load. Only
two benchmarks are currently available.

Additional configuration options to enable verbose debugging are provided:

$ ./configure --debug-http2 --debug-nghttp2
$ NODE_DEBUG=http2 ./node

The --debug-http2 configuration option enables verbose debug statements
from the src/node_http2_* files. The --debug-nghttp2 enables the nghttp
library's own verbose debug output. The NODE_DEBUG=http2 enables JS-level
debug output.

The following illustrates as simple HTTP/2 server and client interaction:

(The HTTP/2 client and server support both plain text and TLS connections)

const req = client.request({ ':path': '/some/path' });
req.on('data', (chunk) => { /* do something with the data */ });
req.on('end', () => {
  client.destroy();
});

// Plain text (non-TLS server)
const server = http2.createServer();
server.on('stream', (stream, requestHeaders) => {
  stream.respond({ ':status': 200 });
  stream.write('hello ');
  stream.end('world');
});
server.listen(80);

const http2 = require('http2');
const client = http2.connect('http://localhost');

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

http2

[Edit:ofrobots: fixed github handle for Kelvin Jin]

Fishrock123 · 2017-07-14T20:04:18Z

Just a thought: separating the new dep checkin to a separate commit might make review easier?

jasnell · 2017-07-14T20:05:14Z

This thing is going to be massive and difficult to review no matter how it is split up, unfortunately.

lucaslago · 2017-07-14T20:15:52Z

deps/nghttp2/lib/nghttp2_pq.h

+  nghttp2_pq_entry **q;
+  /* Memory allocator */
+  nghttp2_mem *mem;
+  /* The number of items sotred */


Nitpick: typo on this comment

That's in the upstream dependency so it would need to be fixed there :-) https://github.com/nghttp2/nghttp2

Fixed upstream nghttp2/nghttp2#958

vsemozhetbyt · 2017-07-14T20:36:36Z

test/parallel/test-tls-disable-renegotiation.js

+
+if (!common.hasCrypto) {
+  common.skip('missing crypto');
+  return;


A nit: redundant after #14021

thinking about it, I think I need to go add the crypto skip in a couple of the other tests.

@lucaslago

Came up in downstream code review by @lucaslago nodejs/node#14239 (comment)

addaleax · 2017-07-15T00:00:41Z

@jasnell We also split out the deps commits for node-inspect and for the inspector iirc; that worked out pretty well and did help with reviewing. :)

addaleax · 2017-07-15T00:04:09Z

Also, for N-API, async_hooks, and some other PRs, we used Author: name <email> metadata (example: 56e881d) for specifying multiple authors. Feel invited to adapt that format. :) (That has the nice side effect of not pinging everybody everytime you rebase/the commit is landed/etc.)

addaleax · 2017-07-15T01:46:39Z

doc/api/http2.md

+
+An object describing the current status of this `Http2Session`.
+
+#### http2session.priority(stream, options)


This does the same thing as stream.priority(options), right? In that case I think I’d prefer not to expose it (same thing for rstStream)

Currently, yes, but eventually this will support the ability to pass in a numeric stream identifier rather than the Http2Stream object. The HTTP/2 spec allows priorities and rst_stream frames to be sent independently, even without a stream having been previously established.

yoshuawuyts · 2017-07-15T12:17:50Z

The colon prefix in :status looks a bit odd to me. I'm sure there have been careful considerations made as to why that makes sense. Would you mind sharing the reasoning behind using colon prefixes? Thanks!

mcollina · 2017-07-15T12:46:59Z

@yoshuawuyts it's part of the HTTP2 spec: https://http2.github.io/http2-spec/#HttpResponse

yoshuawuyts · 2017-07-15T13:04:30Z

@mcollina cool, thanks for answering!

…

On Sat, Jul 15, 2017, 14:47 Matteo Collina ***@***.***> wrote: @yoshuawuyts <https://github.com/yoshuawuyts> it's part of the HTTP2 spec: https://http2.github.io/http2-spec/#HttpResponse — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14239 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACWleuhN8XM3Dk7G-OD3ZsFjZciMOVUsks5sOLThgaJpZM4OYrK1> .

dougwilson · 2017-07-15T17:39:26Z

Awesome work! Above you said the following:

The Compat API is intended to be as close to the existing HTTP/1 API as possible, with some exceptions.

I tried looking in the documentation included in this PR, but couldn't figure out what these exceptions are. Can you ellaborate? Is the compatibility API just the same how the "spdy" module does the HTTP/2 -> 1 compatibility API or different somehow? I'm not really looking for a comparison to the "spdy" module, but thought if you knew that would help, otherwise would love to see the docs around what these differences are.

jasnell · 2017-07-15T17:42:49Z

@dougwilson .... I'm hoping to get that compat api documented this next week... Including those caveats.

vsemozhetbyt · 2017-07-15T17:59:18Z

http2 should also be added to all.md.

vsemozhetbyt · 2017-07-15T18:09:59Z

doc/api/errors.md

+<a id="ERR_HTTP2_STREAM_ERROR"></a>
+### ERR_HTTP2_STREAM_ERROR
+
+Used when a non-zero error code has been specified in an RST_STREAM frame.


`RST_STREAM`?

vsemozhetbyt · 2017-07-15T18:13:15Z

doc/guides/writing-and-running-benchmarks.md

+
+To run the `http2` benchmarks, the `h2load` benchmarker must be used. The
+`h2load` tool is a component of the `nghttp2` project and may be installed
+from [nghttp.org][] or built from source.


nghttp2.org

dougwilson · 2017-07-15T18:13:17Z

Thanks @jasnell ! Please feel free to ping me when you do, I would love to take a look.

vsemozhetbyt · 2017-07-16T00:44:28Z

doc/api/http2.md

+[`tls.TLSSocket`]: tls.html
+[`tls.createServer()`]: tls.html#tls_tls_createserver_options_secureconnectionlistener
+[ClientHttp2Stream]: #http2_class_clienthttp2stream
+[Compatibility API: #http2_compatibility_api


missing ]

vsemozhetbyt · 2017-07-16T00:59:04Z

doc/api/http2.md

+[`net.Socket`]: net.html
+[`tls.TLSSocket`]: tls.html
+[`tls.createServer()`]: tls.html#tls_tls_createserver_options_secureconnectionlistener
+[ClientHttp2Stream]: #http2_class_clienthttp2stream


According to using in text,

[ServerHttp2Stream]: [ClientHttp2Stream]: [Http2Stream]:

should be:

[`ServerHttp2Stream`]: [`ClientHttp2Stream`]: [`Http2Stream`]:

vsemozhetbyt · 2017-07-16T01:00:22Z

doc/api/http2.md

+* `settings` {[Settings Object][]}
+* Returns: {Buffer}
+
+Returns a [Buffer][] instance containing serialized representation of the given


Missing reference for [Buffer][]

vsemozhetbyt · 2017-07-16T01:03:12Z

doc/api/http2.md

+* A new HTTP/2 `HEADERS` frame with a previously unused stream ID is received;
+* The `http2stream.pushStream()` method is called.
+
+On the client side, instances of [`ClientHttp2Stream`[] are created when the


Missing ]

vsemozhetbyt · 2017-07-16T14:02:02Z

Issue with the first examples from the main doc: #14304

sebdeckers · 2017-07-16T15:45:39Z

@dougwilson Some notable differences, off the top of my head:

Most private variables are not supported. Only the documented methods, properties, and events are supported. Code that relies on undocumented behaviour is not supported by the compat api. E.g. no more _header checking; use headersSent instead.
The request and response objects do not inherit from http.IncomingMessage and http.ServerResponse respectively. This poses a challenge for express@4, in my experience.
No status message in sendHead because HTTP/2 dropped the concept of text status messages. Status code is now the :status header. A process warning is emitted if the (optional) status message is passed. The message itself is ignored. Status code and rest of the headers are sent.
The socket property should not be written to. HTTP/2 has a framing layer and does not directly expose the socket. Only the low-level nghttp2 library deals with the socket. However meta properties on the socket can be useful, like detecting http vs https, address/port, TLS info, etc.
Only status codes 1xx-5xx are supported. This follows the HTTP spec. Previously in the HTTP/1 API status codes 1xx-9xx were allowed.
Extensions:
- The [req|res].stream property exposes the HTTP/2 stream, which in turn exposes the HTTP/2 session.
- res.createPushResponse(headers, cb) can use used to send a PUSH_PROMISE frame. The callback receives the corresponding Http2ServerResponse instance via which headers and a body can be sent.

Backport-PR-URL: #14813 Backport-Reviewed-By: Anna Henningsen <anna@addaleax.net> Backport-Reviewed-By: Timothy Gu <timothygu99@gmail.com> PR-URL: #14239 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

As discussed in the review for #14239, these buffers should be per-Environment rather than static. PR-URL: #14744 Reviewed-By: James M Snell <jasnell@gmail.com>

Notable changes * **HTTP2** * Experimental support for the built-in `http2` has been added via the `--expose-http2` flag. [#14239](#14239) * **Inspector** * `require()` is available in the inspector console now. [#8837](#8837) * Multiple contexts, as created by the `vm` module, are supported now. [#14465](#14465) * **N-API** * New APIs for creating number values have been introduced. [#14573](#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](#14558) PR-URL: #14811

vsemozhetbyt · 2017-08-16T10:33:09Z

@jasnell https://nodejs.org/api/http2.html has many not rendered links now (just search ][ in a browser, skipping headings).

refack · 2017-08-16T11:44:09Z

AFAICT the md has has broken links (or rather links with no definition in the footer) in certain areas. Some were fixed in 85d7d97

refack · 2017-08-16T11:54:41Z

Cross ref: #14857

Notable changes * **HTTP2** * Experimental support for the built-in `http2` has been added via the `--expose-http2` flag. [#14239](nodejs/node#14239) * **Inspector** * `require()` is available in the inspector console now. [#8837](nodejs/node#8837) * Multiple contexts, as created by the `vm` module, are supported now. [#14465](nodejs/node#14465) * **N-API** * New APIs for creating number values have been introduced. [#14573](nodejs/node#14573) * **Stream** * For `Duplex` streams, the high water mark option can now be set independently for the readable and the writable side. [#14636](nodejs/node#14636) * **Util** * `util.format` now supports the `%o` and `%O` specifiers for printing objects. [#14558](nodejs/node#14558) PR-URL: nodejs/node#14811

MylesBorins · 2018-01-11T18:18:11Z

Setting don't land on v6.x. Assuming we are not going to backport this

In the initial version of this test there were two zero-length writes to force tls state to cycle. The second is not necessary, at least not now, but the first was. The renegotiate() API should ensure that packet exchange takes place, not its users, so move the zero-length write into tls. See: nodejs#14239 See: nodejs@b1909d3a70f9

In the initial version of this test there were two zero-length writes to force tls state to cycle. The second is not necessary, at least not now, but the first was. The renegotiate() API should ensure that packet exchange takes place, not its users, so move the zero-length write into tls. See: #14239 See: b1909d3a70f9 PR-URL: #25997 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

In the initial version of this test there were two zero-length writes to force tls state to cycle. The second is not necessary, at least not now, but the first was. The renegotiate() API should ensure that packet exchange takes place, not its users, so move the zero-length write into tls. See: nodejs#14239 See: nodejs@b1909d3a70f9 PR-URL: nodejs#25997 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

In the initial version of this test there were two zero-length writes to force tls state to cycle. The second is not necessary, at least not now, but the first was. The renegotiate() API should ensure that packet exchange takes place, not its users, so move the zero-length write into tls. See: #14239 See: b1909d3a70f9 Backport-PR-URL: #27938 PR-URL: #25997 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

jasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Jul 14, 2017

nodejs-github-bot added the build Issues and PRs related to build files or the CI. label Jul 14, 2017

jasnell added experimental Issues and PRs related to experimental features. http2 Issues or PRs related to the http2 subsystem. labels Jul 14, 2017

jasnell changed the title ~~Initial pr~~ http2: Introducing http/2 Jul 14, 2017

lucaslago reviewed Jul 14, 2017

View reviewed changes

vsemozhetbyt reviewed Jul 14, 2017

View reviewed changes

sebdeckers added a commit to sebdeckers/nghttp2 that referenced this pull request Jul 14, 2017

fix: typo

c601e60

Came up in downstream code review by @lucaslago nodejs/node#14239 (comment)

sebdeckers mentioned this pull request Jul 14, 2017

fix: typo nghttp2/nghttp2#958

Merged

addaleax reviewed Jul 15, 2017

View reviewed changes

vsemozhetbyt reviewed Jul 15, 2017

View reviewed changes

vsemozhetbyt reviewed Jul 16, 2017

View reviewed changes

vsemozhetbyt mentioned this pull request Jul 16, 2017

http2: code examples does not work on Windows with port 80 #14304

Closed

addaleax added a commit that referenced this pull request Aug 14, 2017

http2: use per-environment buffers

8a0d101

As discussed in the review for #14239, these buffers should be per-Environment rather than static. PR-URL: #14744 Reviewed-By: James M Snell <jasnell@gmail.com>

addaleax mentioned this pull request Aug 14, 2017

[8.x] Backport multiple http2 PRs #14813

Closed

4 tasks

addaleax mentioned this pull request Aug 14, 2017

v8.4.0 proposal #14811

Merged

mcollina mentioned this pull request Sep 28, 2017

Revisit putting new features behind flags nodejs/TSC#369

Closed

MylesBorins added the dont-land-on-v6.x label Jan 11, 2018

Hackzzila mentioned this pull request May 2, 2018

brotli: add brotli support #20458

Closed

4 tasks

sam-github mentioned this pull request Feb 7, 2019

tls: renegotiate should take care of its own state #25997

Closed

3 tasks

mildsunrise mentioned this pull request Oct 6, 2019

HTTP/2 Host header disallowed #29858

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http2: Introducing http/2 #14239

http2: Introducing http/2 #14239

jasnell commented Jul 14, 2017 •

edited by ofrobots

Loading

Fishrock123 commented Jul 14, 2017

jasnell commented Jul 14, 2017

lucaslago Jul 14, 2017

jasnell Jul 14, 2017

sebdeckers Jul 15, 2017

vsemozhetbyt Jul 14, 2017 •

edited

Loading

jasnell Jul 14, 2017

addaleax commented Jul 15, 2017

addaleax commented Jul 15, 2017

addaleax Jul 15, 2017

jasnell Jul 17, 2017

yoshuawuyts commented Jul 15, 2017

mcollina commented Jul 15, 2017

yoshuawuyts commented Jul 15, 2017 via email

dougwilson commented Jul 15, 2017

jasnell commented Jul 15, 2017

vsemozhetbyt commented Jul 15, 2017 •

edited

Loading

vsemozhetbyt Jul 15, 2017

vsemozhetbyt Jul 15, 2017

dougwilson commented Jul 15, 2017

vsemozhetbyt Jul 16, 2017

vsemozhetbyt Jul 16, 2017

vsemozhetbyt Jul 16, 2017

vsemozhetbyt Jul 16, 2017

vsemozhetbyt commented Jul 16, 2017

sebdeckers commented Jul 16, 2017 •

edited

Loading

vsemozhetbyt commented Aug 16, 2017

refack commented Aug 16, 2017

refack commented Aug 16, 2017

MylesBorins commented Jan 11, 2018


		An object describing the current status of this `Http2Session`.

		#### http2session.priority(stream, options)

http2: Introducing http/2 #14239

http2: Introducing http/2 #14239

Conversation

jasnell commented Jul 14, 2017 • edited by ofrobots Loading

Checklist

Affected core subsystem(s)

Fishrock123 commented Jul 14, 2017

jasnell commented Jul 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vsemozhetbyt Jul 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addaleax commented Jul 15, 2017

addaleax commented Jul 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yoshuawuyts commented Jul 15, 2017

mcollina commented Jul 15, 2017

yoshuawuyts commented Jul 15, 2017 via email

dougwilson commented Jul 15, 2017

jasnell commented Jul 15, 2017

vsemozhetbyt commented Jul 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dougwilson commented Jul 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vsemozhetbyt commented Jul 16, 2017

sebdeckers commented Jul 16, 2017 • edited Loading

vsemozhetbyt commented Aug 16, 2017

refack commented Aug 16, 2017

refack commented Aug 16, 2017

MylesBorins commented Jan 11, 2018

jasnell commented Jul 14, 2017 •

edited by ofrobots

Loading

vsemozhetbyt Jul 14, 2017 •

edited

Loading

vsemozhetbyt commented Jul 15, 2017 •

edited

Loading

sebdeckers commented Jul 16, 2017 •

edited

Loading