New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http2: set decodeStrings to false, test for createWriteReq #15140

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
8 participants
@apapirovski
Member

apapirovski commented Sep 2, 2017

Adds a test for all the encodings in createWriteReq, including the default case. The other option was mocking all the handlers but that didn't seem any better. Also, don't think there's a way to set objectMode without digging into _writableState.

If anyone has better ideas on how to properly test this, I'm happy to adjust. Thanks!

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines
Affected core subsystem(s)

http2, test

@mscdex mscdex added the http2 label Sep 2, 2017

@benjamingr

benjamingr approved these changes Sep 2, 2017 edited

Not sure if this is how I would test it - but overall LGTM. Thank you for making a contribution!

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 2, 2017

Member

@benjamingr If you have suggestions, I would be happy to implement/take into consideration for the future. Thanks!

Member

apapirovski commented Sep 2, 2017

@benjamingr If you have suggestions, I would be happy to implement/take into consideration for the future. Thanks!

@jasnell jasnell requested a review from mcollina Sep 5, 2017

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 5, 2017

Member

I am a bit lost on why the objectMode = true need to be tested. Is it needed to validate the encodings bit? In that case it should be left out from the commit message, the title of the PR and put a comment on top on the test itself.

Member

mcollina commented Sep 5, 2017

I am a bit lost on why the objectMode = true need to be tested. Is it needed to validate the encodings bit? In that case it should be left out from the commit message, the title of the PR and put a comment on top on the test itself.

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 5, 2017

Member

@mcollina it converts to Buffer before it gets to createWriteReq without objectMode = true. It only hits those code paths if it's in that mode so it felt like it was key to mention but I can adjust the commit message.

Member

apapirovski commented Sep 5, 2017

@mcollina it converts to Buffer before it gets to createWriteReq without objectMode = true. It only hits those code paths if it's in that mode so it felt like it was key to mention but I can adjust the commit message.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 6, 2017

Member

Can you point me to this code path?

Member

mcollina commented Sep 6, 2017

Can you point me to this code path?

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 6, 2017

Member

Here's the full chain that leads to this:

Writable.prototype.write = function(chunk, encoding, cb) {
var state = this._writableState;
var ret = false;
var isBuf = !state.objectMode && Stream._isUint8Array(chunk);

function writeOrBuffer(stream, state, isBuf, chunk, encoding, cb) {
if (!isBuf) {
var newChunk = decodeChunk(state, chunk, encoding);
if (chunk !== newChunk) {
isBuf = true;
encoding = 'buffer';
chunk = newChunk;
}
}

stream._write(chunk, encoding, state.onwrite);

function decodeChunk(state, chunk, encoding) {
if (!state.objectMode &&
state.decodeStrings !== false &&
typeof chunk === 'string') {
chunk = Buffer.from(chunk, encoding);
}

I could've also used decodeStrings: false but either way I have to set it through _writableState. Although it probably makes it clearer what's going on.

Member

apapirovski commented Sep 6, 2017

Here's the full chain that leads to this:

Writable.prototype.write = function(chunk, encoding, cb) {
var state = this._writableState;
var ret = false;
var isBuf = !state.objectMode && Stream._isUint8Array(chunk);

function writeOrBuffer(stream, state, isBuf, chunk, encoding, cb) {
if (!isBuf) {
var newChunk = decodeChunk(state, chunk, encoding);
if (chunk !== newChunk) {
isBuf = true;
encoding = 'buffer';
chunk = newChunk;
}
}

stream._write(chunk, encoding, state.onwrite);

function decodeChunk(state, chunk, encoding) {
if (!state.objectMode &&
state.decodeStrings !== false &&
typeof chunk === 'string') {
chunk = Buffer.from(chunk, encoding);
}

I could've also used decodeStrings: false but either way I have to set it through _writableState. Although it probably makes it clearer what's going on.

@apapirovski apapirovski changed the title from test: http2 test for createWriteReq in objectMode to test: http2 test for createWriteReq Sep 6, 2017

@apapirovski apapirovski changed the title from test: http2 test for createWriteReq to test: add http2 test for createWriteReq Sep 6, 2017

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 6, 2017

Member

Updated the commit message and the comment in the test itself.

Member

apapirovski commented Sep 6, 2017

Updated the commit message and the comment in the test itself.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 6, 2017

Member

It seems al of these are releated to streams, is there an http2 part of this? Otherwise, let's just write a test for streams.

Member

mcollina commented Sep 6, 2017

It seems al of these are releated to streams, is there an http2 part of this? Otherwise, let's just write a test for streams.

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 6, 2017

Member

Sorry, should've been clearer. That's just the code leading up to our _write.

const err = createWriteReq(req, handle, data, encoding);

function createWriteReq(req, handle, data, encoding) {
switch (encoding) {
case 'latin1':
case 'binary':
return handle.writeLatin1String(req, data);
case 'buffer':
return handle.writeBuffer(req, data);
case 'utf8':
case 'utf-8':
return handle.writeUtf8String(req, data);
case 'ascii':
return handle.writeAsciiString(req, data);
case 'ucs2':
case 'ucs-2':
case 'utf16le':
case 'utf-16le':
return handle.writeUcs2String(req, data);
default:
return handle.writeBuffer(req, Buffer.from(data, encoding));
}
}

Usually we only hit the 'buffer' case, hence switching to objectMode or turning off decodeStrings.

Member

apapirovski commented Sep 6, 2017

Sorry, should've been clearer. That's just the code leading up to our _write.

const err = createWriteReq(req, handle, data, encoding);

function createWriteReq(req, handle, data, encoding) {
switch (encoding) {
case 'latin1':
case 'binary':
return handle.writeLatin1String(req, data);
case 'buffer':
return handle.writeBuffer(req, data);
case 'utf8':
case 'utf-8':
return handle.writeUtf8String(req, data);
case 'ascii':
return handle.writeAsciiString(req, data);
case 'ucs2':
case 'ucs-2':
case 'utf16le':
case 'utf-16le':
return handle.writeUcs2String(req, data);
default:
return handle.writeBuffer(req, Buffer.from(data, encoding));
}
}

Usually we only hit the 'buffer' case, hence switching to objectMode or turning off decodeStrings.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 6, 2017

Member

So, we should be able to receive strings straight into HTTP2. By moving them to buffers we are losing throughput.
I think we should turn off decodeStrings in HTTP2 core (not for the tests). We have a fast path to writing those straight to the HTTP2Stream, let's use it.

Member

mcollina commented Sep 6, 2017

So, we should be able to receive strings straight into HTTP2. By moving them to buffers we are losing throughput.
I think we should turn off decodeStrings in HTTP2 core (not for the tests). We have a fast path to writing those straight to the HTTP2Stream, let's use it.

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 6, 2017

Member

@mcollina Yep, I already started working on PR after this conversation. Thanks for the review!

Would it make sense to just update this PR for it or would you prefer a new one?

Member

apapirovski commented Sep 6, 2017

@mcollina Yep, I already started working on PR after this conversation. Thanks for the review!

Would it make sense to just update this PR for it or would you prefer a new one?

@apapirovski apapirovski changed the title from test: add http2 test for createWriteReq to http2: set decodeStrings to false, test for createWriteReq Sep 6, 2017

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 6, 2017

Member

Ok, everything has been updated. I added a benchmark too. First benchmark so let me know if I should change anything. It's a substantial speed up btw.

Old:

http2/write.js length=65536 streams=100 requests=100 benchmarker="h2load": 1,040.47
http2/write.js length=131072 streams=100 requests=100 benchmarker="h2load": 0.00
http2/write.js length=262144 streams=100 requests=100 benchmarker="h2load": 342.20
http2/write.js length=1048576 streams=100 requests=100 benchmarker="h2load": 89.34
http2/write.js length=65536 streams=200 requests=100 benchmarker="h2load": 1,117.59
http2/write.js length=131072 streams=200 requests=100 benchmarker="h2load": 623.94
http2/write.js length=262144 streams=200 requests=100 benchmarker="h2load": 331.93
http2/write.js length=1048576 streams=200 requests=100 benchmarker="h2load": 89.68
http2/write.js length=65536 streams=1000 requests=100 benchmarker="h2load": 1,105.97
http2/write.js length=131072 streams=1000 requests=100 benchmarker="h2load": 656.71
http2/write.js length=262144 streams=1000 requests=100 benchmarker="h2load": 351.67
http2/write.js length=1048576 streams=1000 requests=100 benchmarker="h2load": 85.21
http2/write.js length=65536 streams=100 requests=1000 benchmarker="h2load": 1,279.40
http2/write.js length=131072 streams=100 requests=1000 benchmarker="h2load": 80.07
http2/write.js length=262144 streams=100 requests=1000 benchmarker="h2load": 381.78
http2/write.js length=1048576 streams=100 requests=1000 benchmarker="h2load": 46.25
http2/write.js length=65536 streams=200 requests=1000 benchmarker="h2load": 1,302.14
http2/write.js length=131072 streams=200 requests=1000 benchmarker="h2load": 746.88
http2/write.js length=262144 streams=200 requests=1000 benchmarker="h2load": 386.10
http2/write.js length=1048576 streams=200 requests=1000 benchmarker="h2load": 0.00
http2/write.js length=65536 streams=1000 requests=1000 benchmarker="h2load": 1,096.86
http2/write.js length=131072 streams=1000 requests=1000 benchmarker="h2load": 704.04
http2/write.js length=262144 streams=1000 requests=1000 benchmarker="h2load": 368.96
http2/write.js length=1048576 streams=1000 requests=1000 benchmarker="h2load": 80.00
http2/write.js length=65536 streams=100 requests=10000 benchmarker="h2load": 591.57
http2/write.js length=131072 streams=100 requests=10000 benchmarker="h2load": 580.47
http2/write.js length=262144 streams=100 requests=10000 benchmarker="h2load": 351.71
http2/write.js length=1048576 streams=100 requests=10000 benchmarker="h2load": 78.56
http2/write.js length=65536 streams=200 requests=10000 benchmarker="h2load": 1,118.39
http2/write.js length=131072 streams=200 requests=10000 benchmarker="h2load": 693.63
http2/write.js length=262144 streams=200 requests=10000 benchmarker="h2load": 376.24
http2/write.js length=1048576 streams=200 requests=10000 benchmarker="h2load": 70.16
http2/write.js length=65536 streams=1000 requests=10000 benchmarker="h2load": 1,114.98
http2/write.js length=131072 streams=1000 requests=10000 benchmarker="h2load": 612.16
http2/write.js length=262144 streams=1000 requests=10000 benchmarker="h2load": 280.43
http2/write.js length=1048576 streams=1000 requests=10000 benchmarker="h2load": 68.78

New:

http2/write.js length=65536 streams=100 requests=100 benchmarker="h2load": 1,431.64
http2/write.js length=131072 streams=100 requests=100 benchmarker="h2load": 848.23
http2/write.js length=262144 streams=100 requests=100 benchmarker="h2load": 460.35
http2/write.js length=1048576 streams=100 requests=100 benchmarker="h2load": 125.00
http2/write.js length=65536 streams=200 requests=100 benchmarker="h2load": 1,407.76
http2/write.js length=131072 streams=200 requests=100 benchmarker="h2load": 821.02
http2/write.js length=262144 streams=200 requests=100 benchmarker="h2load": 488.43
http2/write.js length=1048576 streams=200 requests=100 benchmarker="h2load": 121.48
http2/write.js length=65536 streams=1000 requests=100 benchmarker="h2load": 1,464.69
http2/write.js length=131072 streams=1000 requests=100 benchmarker="h2load": 848.33
http2/write.js length=262144 streams=1000 requests=100 benchmarker="h2load": 469.26
http2/write.js length=1048576 streams=1000 requests=100 benchmarker="h2load": 124.49
http2/write.js length=65536 streams=100 requests=1000 benchmarker="h2load": 1,692.65
http2/write.js length=131072 streams=100 requests=1000 benchmarker="h2load": 954.82
http2/write.js length=262144 streams=100 requests=1000 benchmarker="h2load": 536.26
http2/write.js length=1048576 streams=100 requests=1000 benchmarker="h2load": 127.43
http2/write.js length=65536 streams=200 requests=1000 benchmarker="h2load": 1,602.40
http2/write.js length=131072 streams=200 requests=1000 benchmarker="h2load": 980.88
http2/write.js length=262144 streams=200 requests=1000 benchmarker="h2load": 583.23
http2/write.js length=1048576 streams=200 requests=1000 benchmarker="h2load": 135.87
http2/write.js length=65536 streams=1000 requests=1000 benchmarker="h2load": 1,503.10
http2/write.js length=131072 streams=1000 requests=1000 benchmarker="h2load": 877.49
http2/write.js length=262144 streams=1000 requests=1000 benchmarker="h2load": 494.46
http2/write.js length=1048576 streams=1000 requests=1000 benchmarker="h2load": 120.87
http2/write.js length=65536 streams=100 requests=10000 benchmarker="h2load": 1,579.05
http2/write.js length=131072 streams=100 requests=10000 benchmarker="h2load": 984.19
http2/write.js length=262144 streams=100 requests=10000 benchmarker="h2load": 496.87
http2/write.js length=1048576 streams=100 requests=10000 benchmarker="h2load": 109.32
http2/write.js length=65536 streams=200 requests=10000 benchmarker="h2load": 1,799.27
http2/write.js length=131072 streams=200 requests=10000 benchmarker="h2load": 990.87
http2/write.js length=262144 streams=200 requests=10000 benchmarker="h2load": 538.14
http2/write.js length=1048576 streams=200 requests=10000 benchmarker="h2load": 134.92
http2/write.js length=65536 streams=1000 requests=10000 benchmarker="h2load": 1,797.03
http2/write.js length=131072 streams=1000 requests=10000 benchmarker="h2load": 965.08
http2/write.js length=262144 streams=1000 requests=10000 benchmarker="h2load": 543.61
http2/write.js length=1048576 streams=1000 requests=10000 benchmarker="h2load": 133.38

The format of these tests doesn't seem compatible with the compare script but it looks like a 35% speed up even if I'm being conservative, maybe higher.

Member

apapirovski commented Sep 6, 2017

Ok, everything has been updated. I added a benchmark too. First benchmark so let me know if I should change anything. It's a substantial speed up btw.

Old:

http2/write.js length=65536 streams=100 requests=100 benchmarker="h2load": 1,040.47
http2/write.js length=131072 streams=100 requests=100 benchmarker="h2load": 0.00
http2/write.js length=262144 streams=100 requests=100 benchmarker="h2load": 342.20
http2/write.js length=1048576 streams=100 requests=100 benchmarker="h2load": 89.34
http2/write.js length=65536 streams=200 requests=100 benchmarker="h2load": 1,117.59
http2/write.js length=131072 streams=200 requests=100 benchmarker="h2load": 623.94
http2/write.js length=262144 streams=200 requests=100 benchmarker="h2load": 331.93
http2/write.js length=1048576 streams=200 requests=100 benchmarker="h2load": 89.68
http2/write.js length=65536 streams=1000 requests=100 benchmarker="h2load": 1,105.97
http2/write.js length=131072 streams=1000 requests=100 benchmarker="h2load": 656.71
http2/write.js length=262144 streams=1000 requests=100 benchmarker="h2load": 351.67
http2/write.js length=1048576 streams=1000 requests=100 benchmarker="h2load": 85.21
http2/write.js length=65536 streams=100 requests=1000 benchmarker="h2load": 1,279.40
http2/write.js length=131072 streams=100 requests=1000 benchmarker="h2load": 80.07
http2/write.js length=262144 streams=100 requests=1000 benchmarker="h2load": 381.78
http2/write.js length=1048576 streams=100 requests=1000 benchmarker="h2load": 46.25
http2/write.js length=65536 streams=200 requests=1000 benchmarker="h2load": 1,302.14
http2/write.js length=131072 streams=200 requests=1000 benchmarker="h2load": 746.88
http2/write.js length=262144 streams=200 requests=1000 benchmarker="h2load": 386.10
http2/write.js length=1048576 streams=200 requests=1000 benchmarker="h2load": 0.00
http2/write.js length=65536 streams=1000 requests=1000 benchmarker="h2load": 1,096.86
http2/write.js length=131072 streams=1000 requests=1000 benchmarker="h2load": 704.04
http2/write.js length=262144 streams=1000 requests=1000 benchmarker="h2load": 368.96
http2/write.js length=1048576 streams=1000 requests=1000 benchmarker="h2load": 80.00
http2/write.js length=65536 streams=100 requests=10000 benchmarker="h2load": 591.57
http2/write.js length=131072 streams=100 requests=10000 benchmarker="h2load": 580.47
http2/write.js length=262144 streams=100 requests=10000 benchmarker="h2load": 351.71
http2/write.js length=1048576 streams=100 requests=10000 benchmarker="h2load": 78.56
http2/write.js length=65536 streams=200 requests=10000 benchmarker="h2load": 1,118.39
http2/write.js length=131072 streams=200 requests=10000 benchmarker="h2load": 693.63
http2/write.js length=262144 streams=200 requests=10000 benchmarker="h2load": 376.24
http2/write.js length=1048576 streams=200 requests=10000 benchmarker="h2load": 70.16
http2/write.js length=65536 streams=1000 requests=10000 benchmarker="h2load": 1,114.98
http2/write.js length=131072 streams=1000 requests=10000 benchmarker="h2load": 612.16
http2/write.js length=262144 streams=1000 requests=10000 benchmarker="h2load": 280.43
http2/write.js length=1048576 streams=1000 requests=10000 benchmarker="h2load": 68.78

New:

http2/write.js length=65536 streams=100 requests=100 benchmarker="h2load": 1,431.64
http2/write.js length=131072 streams=100 requests=100 benchmarker="h2load": 848.23
http2/write.js length=262144 streams=100 requests=100 benchmarker="h2load": 460.35
http2/write.js length=1048576 streams=100 requests=100 benchmarker="h2load": 125.00
http2/write.js length=65536 streams=200 requests=100 benchmarker="h2load": 1,407.76
http2/write.js length=131072 streams=200 requests=100 benchmarker="h2load": 821.02
http2/write.js length=262144 streams=200 requests=100 benchmarker="h2load": 488.43
http2/write.js length=1048576 streams=200 requests=100 benchmarker="h2load": 121.48
http2/write.js length=65536 streams=1000 requests=100 benchmarker="h2load": 1,464.69
http2/write.js length=131072 streams=1000 requests=100 benchmarker="h2load": 848.33
http2/write.js length=262144 streams=1000 requests=100 benchmarker="h2load": 469.26
http2/write.js length=1048576 streams=1000 requests=100 benchmarker="h2load": 124.49
http2/write.js length=65536 streams=100 requests=1000 benchmarker="h2load": 1,692.65
http2/write.js length=131072 streams=100 requests=1000 benchmarker="h2load": 954.82
http2/write.js length=262144 streams=100 requests=1000 benchmarker="h2load": 536.26
http2/write.js length=1048576 streams=100 requests=1000 benchmarker="h2load": 127.43
http2/write.js length=65536 streams=200 requests=1000 benchmarker="h2load": 1,602.40
http2/write.js length=131072 streams=200 requests=1000 benchmarker="h2load": 980.88
http2/write.js length=262144 streams=200 requests=1000 benchmarker="h2load": 583.23
http2/write.js length=1048576 streams=200 requests=1000 benchmarker="h2load": 135.87
http2/write.js length=65536 streams=1000 requests=1000 benchmarker="h2load": 1,503.10
http2/write.js length=131072 streams=1000 requests=1000 benchmarker="h2load": 877.49
http2/write.js length=262144 streams=1000 requests=1000 benchmarker="h2load": 494.46
http2/write.js length=1048576 streams=1000 requests=1000 benchmarker="h2load": 120.87
http2/write.js length=65536 streams=100 requests=10000 benchmarker="h2load": 1,579.05
http2/write.js length=131072 streams=100 requests=10000 benchmarker="h2load": 984.19
http2/write.js length=262144 streams=100 requests=10000 benchmarker="h2load": 496.87
http2/write.js length=1048576 streams=100 requests=10000 benchmarker="h2load": 109.32
http2/write.js length=65536 streams=200 requests=10000 benchmarker="h2load": 1,799.27
http2/write.js length=131072 streams=200 requests=10000 benchmarker="h2load": 990.87
http2/write.js length=262144 streams=200 requests=10000 benchmarker="h2load": 538.14
http2/write.js length=1048576 streams=200 requests=10000 benchmarker="h2load": 134.92
http2/write.js length=65536 streams=1000 requests=10000 benchmarker="h2load": 1,797.03
http2/write.js length=131072 streams=1000 requests=10000 benchmarker="h2load": 965.08
http2/write.js length=262144 streams=1000 requests=10000 benchmarker="h2load": 543.61
http2/write.js length=1048576 streams=1000 requests=10000 benchmarker="h2load": 133.38

The format of these tests doesn't seem compatible with the compare script but it looks like a 35% speed up even if I'm being conservative, maybe higher.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 6, 2017

Member

@benjamingr @claudiorodriguez this would require a new review.

Member

mcollina commented Sep 6, 2017

@benjamingr @claudiorodriguez this would require a new review.

@mcollina

LGTM

@mcollina

This comment has been minimized.

Show comment
Hide comment
Member

mcollina commented Sep 6, 2017

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 6, 2017

Member

@mcollina it doesn't seem to work for benchmarks using h2load for some reason Actually, what I'm having trouble is getting it to load the different benchmarker with compare

Update: Got it to work. Forgot "--set"... *sigh* totally my bad.

Member

apapirovski commented Sep 6, 2017

@mcollina it doesn't seem to work for benchmarks using h2load for some reason Actually, what I'm having trouble is getting it to load the different benchmarker with compare

Update: Got it to work. Forgot "--set"... *sigh* totally my bad.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 6, 2017

Member

which command did you use in the end?

Member

mcollina commented Sep 6, 2017

which command did you use in the end?

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 6, 2017

Member

This is what I used

node benchmark/compare.js --runs 3 --old ./out/Release-old/node --new ./out/Release/node --filter write --set benchmarker=h2load http2 > results.csv

I was just forgetting --set hence my confusion above.

I got about 45% improvement across the board but didn't have time to keep running it for more than 3 runs. Might run it for longer later but I think it's safe to say this is a big improvement.

Member

apapirovski commented Sep 6, 2017

This is what I used

node benchmark/compare.js --runs 3 --old ./out/Release-old/node --new ./out/Release/node --filter write --set benchmarker=h2load http2 > results.csv

I was just forgetting --set hence my confusion above.

I got about 45% improvement across the board but didn't have time to keep running it for more than 3 runs. Might run it for longer later but I think it's safe to say this is a big improvement.

@jasnell

jasnell approved these changes Sep 6, 2017

@mcollina

This comment has been minimized.

Show comment
Hide comment
@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 7, 2017

Member

linux-one failed on the test case which I think might mean there's an issue in the underlying C++ code. @jasnell @mcollina can one of you take a look maybe? This is a bit outside of my area of expertise.

Member

apapirovski commented Sep 7, 2017

linux-one failed on the test case which I think might mean there's an issue in the underlying C++ code. @jasnell @mcollina can one of you take a look maybe? This is a bit outside of my area of expertise.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 7, 2017

Member

I think it's a problem of having a timeout-based test:

not ok 800 parallel/test-http2-createwritereq
  ---
  duration_ms: 0.306
  severity: fail
  stack: |-
    Mismatched <anonymous> function calls. Expected exactly 11, actual 8.
        at Object.exports.mustCall (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/common/index.js:477:10)
        at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-http2-createwritereq.js:29:42)
        at Module._compile (module.js:549:30)
        at Object.Module._extensions..js (module.js:560:10)
        at Module.load (module.js:483:32)
        at tryModuleLoad (module.js:446:12)
        at Function.Module._load (module.js:438:3)
        at Function.Module.runMain (module.js:585:10)
        at startup (bootstrap_node.js:194:16)
    (node:34482) ExperimentalWarning: The http2 module is an experimental API.

Those platforms are slow :(

Member

mcollina commented Sep 7, 2017

I think it's a problem of having a timeout-based test:

not ok 800 parallel/test-http2-createwritereq
  ---
  duration_ms: 0.306
  severity: fail
  stack: |-
    Mismatched <anonymous> function calls. Expected exactly 11, actual 8.
        at Object.exports.mustCall (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/common/index.js:477:10)
        at Object.<anonymous> (/data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/test/parallel/test-http2-createwritereq.js:29:42)
        at Module._compile (module.js:549:30)
        at Object.Module._extensions..js (module.js:560:10)
        at Module.load (module.js:483:32)
        at tryModuleLoad (module.js:446:12)
        at Function.Module._load (module.js:438:3)
        at Function.Module.runMain (module.js:585:10)
        at startup (bootstrap_node.js:194:16)
    (node:34482) ExperimentalWarning: The http2 module is an experimental API.

Those platforms are slow :(

@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 7, 2017

Member

@mcollina this one isn't timeout based, this one just writes strings and expects to get the right thing on the other end. Although I think it does have to do with the platform being slow. I think it closes the server before it received everything. Will adjust the test case to be less flaky. Good lesson for me re: writing tests for slower platforms, have to make less assumptions about what is finished and what isn't.

Member

apapirovski commented Sep 7, 2017

@mcollina this one isn't timeout based, this one just writes strings and expects to get the right thing on the other end. Although I think it does have to do with the platform being slow. I think it closes the server before it received everything. Will adjust the test case to be less flaky. Good lesson for me re: writing tests for slower platforms, have to make less assumptions about what is finished and what isn't.

http2: set decodeStrings to false, test
Set writableStream decodeStrings to false to let the
native layer handle converting strings to buffer.
@apapirovski

This comment has been minimized.

Show comment
Hide comment
@apapirovski

apapirovski Sep 7, 2017

Member

@mcollina I think it should work now. Could we restart the CI? Thanks.

Member

apapirovski commented Sep 7, 2017

@mcollina I think it should work now. Could we restart the CI? Thanks.

@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

This comment has been minimized.

Show comment
Hide comment
@mcollina

mcollina Sep 7, 2017

Member

CI is green, this is ok to go.

Member

mcollina commented Sep 7, 2017

CI is green, this is ok to go.

jasnell added a commit that referenced this pull request Sep 7, 2017

http2: set decodeStrings to false, test
Set writableStream decodeStrings to false to let the
native layer handle converting strings to buffer.

PR-URL: #15140
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Claudio Rodriguez <cjrodr@yahoo.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@jasnell

This comment has been minimized.

Show comment
Hide comment
@jasnell

jasnell Sep 7, 2017

Member

Landed in 2ffc8ac

Member

jasnell commented Sep 7, 2017

Landed in 2ffc8ac

@jasnell jasnell closed this Sep 7, 2017

@apapirovski apapirovski deleted the apapirovski:patch-test-http2-createwritereq branch Sep 9, 2017

MylesBorins added a commit that referenced this pull request Sep 10, 2017

http2: set decodeStrings to false, test
Set writableStream decodeStrings to false to let the
native layer handle converting strings to buffer.

PR-URL: #15140
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Claudio Rodriguez <cjrodr@yahoo.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

@MylesBorins MylesBorins referenced this pull request Sep 10, 2017

Merged

v8.5.0 proposal #15308

MylesBorins added a commit that referenced this pull request Sep 11, 2017

http2: set decodeStrings to false, test
Set writableStream decodeStrings to false to let the
native layer handle converting strings to buffer.

PR-URL: #15140
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Claudio Rodriguez <cjrodr@yahoo.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins added a commit that referenced this pull request Sep 12, 2017

http2: set decodeStrings to false, test
Set writableStream decodeStrings to false to let the
native layer handle converting strings to buffer.

PR-URL: #15140
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Claudio Rodriguez <cjrodr@yahoo.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

addaleax added a commit to addaleax/node that referenced this pull request Sep 13, 2017

http2: set decodeStrings to false, test
Set writableStream decodeStrings to false to let the
native layer handle converting strings to buffer.

PR-URL: nodejs#15140
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Claudio Rodriguez <cjrodr@yahoo.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment