New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http: add strictMode option and checkIsHttpToken check #2526

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
9 participants
@jasnell
Member

jasnell commented Aug 24, 2015

Per: nodejs/node-convergence-archive#13

Adds a check that will cause a TypeError to be thrown if the HTTP
method or header field name does not conform to the Token rule.

This adds a new strictMode flag to OutgoingMessage that, when
enabled, will cause a TypeError to be thrown if the HTTP Method
or header field name does not conform to the Token rule.

The strictMode flag ensures that, in the common case, there
is minimal performance hit and that existing code should continue
to work. The Token check is only performed when strictMode = true.

Doc and test case are included.

On the client-side

var http = require('http');
var url = require('url');
var p = url.parse('http://localhost:8888');
p.headers = {'testing 123': 123};
p.strictMode = true;
http.client(p, function(res) { }); // throws

On the server-side

var http = require('http');
var server = http.createServer(function(req,res) {
  res.strictMode = true;
  res.setHeader('testing 123', 123); // throws
  res.end('...');
});
@brendanashworth

This comment has been minimized.

Member

brendanashworth commented Aug 24, 2015

Would the goal be to eventually have this as a default setting?

@jasnell

This comment has been minimized.

Member

jasnell commented Aug 24, 2015

Perhaps eventually.
On Aug 24, 2015 1:41 PM, "Brendan Ashworth" notifications@github.com
wrote:

Would the goal be to eventually have this as a default setting?


Reply to this email directly or view it on GitHub
#2526 (comment).

@jasnell

This comment has been minimized.

Member

jasnell commented Aug 25, 2015

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Aug 25, 2015

LGTM if CI is happy.

@jasnell

This comment has been minimized.

Member

jasnell commented Aug 26, 2015

Just for reference, I ran the http benchmarks with the strict check gating the token check and without... the numbers follow:

With the token check behind the strictMode switch:

http/chunked.js
http/chunked.js num=1 size=1 c=100: 18309.44000
http/chunked.js num=1 size=64 c=100: 18601.50000
http/chunked.js num=1 size=256 c=100: 17928.67000
http/chunked.js num=4 size=1 c=100: 9724.05000
http/chunked.js num=4 size=64 c=100: 8057.15000
http/chunked.js num=4 size=256 c=100: 8346.31000
http/chunked.js num=8 size=1 c=100: 5359.49000
http/chunked.js num=8 size=64 c=100: 5744.77000
http/chunked.js num=8 size=256 c=100: 5536.85000
http/chunked.js num=16 size=1 c=100: 3222.92000
http/chunked.js num=16 size=64 c=100: 3118.45000
http/chunked.js num=16 size=256 c=100: 3305.70000

http/client-request-body.js
http/client-request-body.js dur=5 type=asc bytes=32 method=write: 7993.87798
http/client-request-body.js dur=5 type=asc bytes=32 method=end: 7851.92373
http/client-request-body.js dur=5 type=asc bytes=256 method=write: 7749.83784
http/client-request-body.js dur=5 type=asc bytes=256 method=end: 7369.24317
http/client-request-body.js dur=5 type=asc bytes=1024 method=write: 7454.41692
http/client-request-body.js dur=5 type=asc bytes=1024 method=end: 7300.01900
http/client-request-body.js dur=5 type=utf bytes=32 method=write: 7389.21361
http/client-request-body.js dur=5 type=utf bytes=32 method=end: 7284.74477
http/client-request-body.js dur=5 type=utf bytes=256 method=write: 7584.45917
http/client-request-body.js dur=5 type=utf bytes=256 method=end: 7744.85882
http/client-request-body.js dur=5 type=utf bytes=1024 method=write: 7314.39565
http/client-request-body.js dur=5 type=utf bytes=1024 method=end: 6971.98767
http/client-request-body.js dur=5 type=buf bytes=32 method=write: 7472.07152
http/client-request-body.js dur=5 type=buf bytes=32 method=end: 7537.53015
http/client-request-body.js dur=5 type=buf bytes=256 method=write: 7071.40617
http/client-request-body.js dur=5 type=buf bytes=256 method=end: 7259.03421
http/client-request-body.js dur=5 type=buf bytes=1024 method=write: 7152.04854
http/client-request-body.js dur=5 type=buf bytes=1024 method=end: 7273.43387

http/cluster.js
http/cluster.js type=bytes length=4 c=50: 35922.87000
http/cluster.js type=bytes length=4 c=500: 32476.47000
http/cluster.js type=bytes length=1024 c=50: 35521.69000
http/cluster.js type=bytes length=1024 c=500: 29737.69000
http/cluster.js type=bytes length=102400 c=50: 970.28000
http/cluster.js type=bytes length=102400 c=500: 952.84000
http/cluster.js type=buffer length=4 c=50: 34290.67000
http/cluster.js type=buffer length=4 c=500: 27015.43000
http/cluster.js type=buffer length=1024 c=50: 33668.67000
http/cluster.js type=buffer length=1024 c=500: 27010.19000
http/cluster.js type=buffer length=102400 c=50: 20441.56000
http/cluster.js type=buffer length=102400 c=500: 17763.59000

http/end-vs-write-end.js
http/end-vs-write-end.js type=asc kb=64 c=100 method=write: 9154.38000
http/end-vs-write-end.js type=asc kb=64 c=100 method=end: 10289.53000
http/end-vs-write-end.js type=asc kb=128 c=100 method=write: 4703.83000
http/end-vs-write-end.js type=asc kb=128 c=100 method=end: 4925.43000
http/end-vs-write-end.js type=asc kb=256 c=100 method=write: 2819.15000
http/end-vs-write-end.js type=asc kb=256 c=100 method=end: 2799.53000
http/end-vs-write-end.js type=asc kb=1024 c=100 method=write: 517.85000
http/end-vs-write-end.js type=asc kb=1024 c=100 method=end: 524.39000
http/end-vs-write-end.js type=utf kb=64 c=100 method=write: 8564.17000
http/end-vs-write-end.js type=utf kb=64 c=100 method=end: 8923.13000
http/end-vs-write-end.js type=utf kb=128 c=100 method=write: 3704.52000
http/end-vs-write-end.js type=utf kb=128 c=100 method=end: 3683.32000
http/end-vs-write-end.js type=utf kb=256 c=100 method=write: 2234.35000
http/end-vs-write-end.js type=utf kb=256 c=100 method=end: 2141.71000
http/end-vs-write-end.js type=utf kb=1024 c=100 method=write: 630.73000
http/end-vs-write-end.js type=utf kb=1024 c=100 method=end: 631.89000
http/end-vs-write-end.js type=buf kb=64 c=100 method=write: 15640.49000
http/end-vs-write-end.js type=buf kb=64 c=100 method=end: 15401.98000
http/end-vs-write-end.js type=buf kb=128 c=100 method=write: 9950.23000
http/end-vs-write-end.js type=buf kb=128 c=100 method=end: 9892.13000
http/end-vs-write-end.js type=buf kb=256 c=100 method=write: 7294.47000
http/end-vs-write-end.js type=buf kb=256 c=100 method=end: 7748.78000
http/end-vs-write-end.js type=buf kb=1024 c=100 method=write: 3381.16000
http/end-vs-write-end.js type=buf kb=1024 c=100 method=end: 3670.61000

http/http_server_for_chunky_client.js
http/_chunky_http_client.js len=1 num=5 type=send: 240.67929
http/_chunky_http_client.js len=1 num=50 type=send: 443.25122
http/_chunky_http_client.js len=1 num=500 type=send: 498.04445
http/_chunky_http_client.js len=1 num=2000 type=send: 589.10199
http/_chunky_http_client.js len=4 num=5 type=send: 280.37217
http/_chunky_http_client.js len=4 num=50 type=send: 479.71579
http/_chunky_http_client.js len=4 num=500 type=send: 545.75310
http/_chunky_http_client.js len=4 num=2000 type=send: 550.50077
http/_chunky_http_client.js len=8 num=5 type=send: 351.46966
http/_chunky_http_client.js len=8 num=50 type=send: 409.37434
http/_chunky_http_client.js len=8 num=500 type=send: 514.70477
http/_chunky_http_client.js len=8 num=2000 type=send: 587.12519
http/_chunky_http_client.js len=16 num=5 type=send: 347.20481
http/_chunky_http_client.js len=16 num=50 type=send: 429.39704
http/_chunky_http_client.js len=16 num=500 type=send: 514.95940
http/_chunky_http_client.js len=16 num=2000 type=send: 567.01406
http/_chunky_http_client.js len=32 num=5 type=send: 318.15355
http/_chunky_http_client.js len=32 num=50 type=send: 443.94173
http/_chunky_http_client.js len=32 num=500 type=send: 548.03194
http/_chunky_http_client.js len=32 num=2000 type=send: 585.29080
http/_chunky_http_client.js len=64 num=5 type=send: 331.60598
http/_chunky_http_client.js len=64 num=50 type=send: 525.38081
http/_chunky_http_client.js len=64 num=500 type=send: 508.09694
http/_chunky_http_client.js len=64 num=2000 type=send: 562.81997
http/_chunky_http_client.js len=128 num=5 type=send: 271.15379
http/_chunky_http_client.js len=128 num=50 type=send: 441.15488
http/_chunky_http_client.js len=128 num=500 type=send: 554.85011
http/_chunky_http_client.js len=128 num=2000 type=send: 548.49425

http/simple.js
http/simple.js type=bytes length=4 chunks=0 c=50: 24577.06000
http/simple.js type=bytes length=4 chunks=0 c=500: 3.88000
http/simple.js type=bytes length=4 chunks=1 c=50: 15095.45000
http/simple.js type=bytes length=4 chunks=1 c=500: 13825.68000
http/simple.js type=bytes length=4 chunks=4 c=50: 13924.70000
http/simple.js type=bytes length=4 chunks=4 c=500: 7826.80000
http/simple.js type=bytes length=1024 chunks=0 c=50: 20230.83000
http/simple.js type=bytes length=1024 chunks=0 c=500: 16739.87000
http/simple.js type=bytes length=1024 chunks=1 c=50: 14211.76000
http/simple.js type=bytes length=1024 chunks=1 c=500: 11736.93000
http/simple.js type=bytes length=1024 chunks=4 c=50: 12830.61000
http/simple.js type=bytes length=1024 chunks=4 c=500: 8884.46000
http/simple.js type=bytes length=102400 chunks=0 c=50: 515.46000
http/simple.js type=bytes length=102400 chunks=0 c=500: 500.52000
http/simple.js type=bytes length=102400 chunks=1 c=50: 297.43000
http/simple.js type=bytes length=102400 chunks=1 c=500: 285.68000
http/simple.js type=bytes length=102400 chunks=4 c=50: 4510.47000
http/simple.js type=bytes length=102400 chunks=4 c=500: 3329.14000
http/simple.js type=buffer length=4 chunks=0 c=50: 21342.39000
http/simple.js type=buffer length=4 chunks=0 c=500: 16453.30000
http/simple.js type=buffer length=4 chunks=1 c=50: 20254.93000
http/simple.js type=buffer length=4 chunks=1 c=500: 15086.55000
http/simple.js type=buffer length=4 chunks=4 c=50: 17383.41000
http/simple.js type=buffer length=4 chunks=4 c=500: 13195.10000
http/simple.js type=buffer length=1024 chunks=0 c=50: 21583.29000
http/simple.js type=buffer length=1024 chunks=0 c=500: 15310.24000
http/simple.js type=buffer length=1024 chunks=1 c=50: 20588.59000
http/simple.js type=buffer length=1024 chunks=1 c=500: 15330.04000
http/simple.js type=buffer length=1024 chunks=4 c=50: 17098.69000
http/simple.js type=buffer length=1024 chunks=4 c=500: 13270.80000
http/simple.js type=buffer length=102400 chunks=0 c=50: 16089.51000
http/simple.js type=buffer length=102400 chunks=0 c=500: 12665.69000
http/simple.js type=buffer length=102400 chunks=1 c=50: 15290.38000
http/simple.js type=buffer length=102400 chunks=1 c=500: 11838.35000
http/simple.js type=buffer length=102400 chunks=4 c=50: 13788.54000
http/simple.js type=buffer length=102400 chunks=4 c=500: 10679.99000

Always using the token check (no strictMode switch):

http/chunked.js
http/chunked.js num=1 size=1 c=100: 18663.11000
http/chunked.js num=1 size=64 c=100: 18565.59000
http/chunked.js num=1 size=256 c=100: 19239.92000
http/chunked.js num=4 size=1 c=100: 10191.70000
http/chunked.js num=4 size=64 c=100: 9880.02000
http/chunked.js num=4 size=256 c=100: 10182.05000
http/chunked.js num=8 size=1 c=100: 6340.44000
http/chunked.js num=8 size=64 c=100: 5886.72000
http/chunked.js num=8 size=256 c=100: 5778.35000
http/chunked.js num=16 size=1 c=100: 3475.42000
http/chunked.js num=16 size=64 c=100: 3256.98000
http/chunked.js num=16 size=256 c=100: 3308.70000

http/client-request-body.js
http/client-request-body.js dur=5 type=asc bytes=32 method=write: 7741.45646
http/client-request-body.js dur=5 type=asc bytes=32 method=end: 7610.65535
http/client-request-body.js dur=5 type=asc bytes=256 method=write: 7756.04929
http/client-request-body.js dur=5 type=asc bytes=256 method=end: 6971.66502
http/client-request-body.js dur=5 type=asc bytes=1024 method=write: 7339.83640
http/client-request-body.js dur=5 type=asc bytes=1024 method=end: 7147.62425
http/client-request-body.js dur=5 type=utf bytes=32 method=write: 7426.14685
http/client-request-body.js dur=5 type=utf bytes=32 method=end: 7412.24635
http/client-request-body.js dur=5 type=utf bytes=256 method=write: 7672.52594
http/client-request-body.js dur=5 type=utf bytes=256 method=end: 7459.27922
http/client-request-body.js dur=5 type=utf bytes=1024 method=write: 7460.66451
http/client-request-body.js dur=5 type=utf bytes=1024 method=end: 7597.84984
http/client-request-body.js dur=5 type=buf bytes=32 method=write: 7609.09872
http/client-request-body.js dur=5 type=buf bytes=32 method=end: 7335.76260
http/client-request-body.js dur=5 type=buf bytes=256 method=write: 7555.51166
http/client-request-body.js dur=5 type=buf bytes=256 method=end: 7606.14852
http/client-request-body.js dur=5 type=buf bytes=1024 method=write: 7400.66296
http/client-request-body.js dur=5 type=buf bytes=1024 method=end: 7209.35418

http/cluster.js
http/cluster.js type=bytes length=4 c=50: 42871.50000
http/cluster.js type=bytes length=4 c=500: 32957.56000
http/cluster.js type=bytes length=1024 c=50: 34720.35000
http/cluster.js type=bytes length=1024 c=500: 29112.61000
http/cluster.js type=bytes length=102400 c=50: 968.38000
http/cluster.js type=bytes length=102400 c=500: 972.06000
http/cluster.js type=buffer length=4 c=50: 34677.93000
http/cluster.js type=buffer length=4 c=500: 28911.57000
http/cluster.js type=buffer length=1024 c=50: 35889.32000
http/cluster.js type=buffer length=1024 c=500: 28730.63000
http/cluster.js type=buffer length=102400 c=50: 22793.73000
http/cluster.js type=buffer length=102400 c=500: 20172.45000

http/end-vs-write-end.js
http/end-vs-write-end.js type=asc kb=64 c=100 method=write: 9785.71000
http/end-vs-write-end.js type=asc kb=64 c=100 method=end: 10834.95000
http/end-vs-write-end.js type=asc kb=128 c=100 method=write: 4690.76000
http/end-vs-write-end.js type=asc kb=128 c=100 method=end: 4877.38000
http/end-vs-write-end.js type=asc kb=256 c=100 method=write: 2689.61000
http/end-vs-write-end.js type=asc kb=256 c=100 method=end: 2784.90000
http/end-vs-write-end.js type=asc kb=1024 c=100 method=write: 527.32000
http/end-vs-write-end.js type=asc kb=1024 c=100 method=end: 530.89000
http/end-vs-write-end.js type=utf kb=64 c=100 method=write: 8824.99000
http/end-vs-write-end.js type=utf kb=64 c=100 method=end: 9110.39000
http/end-vs-write-end.js type=utf kb=128 c=100 method=write: 4094.09000
http/end-vs-write-end.js type=utf kb=128 c=100 method=end: 4248.21000
http/end-vs-write-end.js type=utf kb=256 c=100 method=write: 2266.85000
http/end-vs-write-end.js type=utf kb=256 c=100 method=end: 2307.44000
http/end-vs-write-end.js type=utf kb=1024 c=100 method=write: 642.20000
http/end-vs-write-end.js type=utf kb=1024 c=100 method=end: 651.53000
http/end-vs-write-end.js type=buf kb=64 c=100 method=write: 17079.15000
http/end-vs-write-end.js type=buf kb=64 c=100 method=end: 17469.43000
http/end-vs-write-end.js type=buf kb=128 c=100 method=write: 13556.84000
http/end-vs-write-end.js type=buf kb=128 c=100 method=end: 13849.33000
http/end-vs-write-end.js type=buf kb=256 c=100 method=write: 10149.57000
http/end-vs-write-end.js type=buf kb=256 c=100 method=end: 10378.89000
http/end-vs-write-end.js type=buf kb=1024 c=100 method=write: 3926.52000
http/end-vs-write-end.js type=buf kb=1024 c=100 method=end: 3956.10000

http/http_server_for_chunky_client.js
http/_chunky_http_client.js len=1 num=5 type=send: 249.10514
http/_chunky_http_client.js len=1 num=50 type=send: 464.66644
http/_chunky_http_client.js len=1 num=500 type=send: 542.55153
http/_chunky_http_client.js len=1 num=2000 type=send: 565.85424
http/_chunky_http_client.js len=4 num=5 type=send: 364.90072
http/_chunky_http_client.js len=4 num=50 type=send: 431.12156
http/_chunky_http_client.js len=4 num=500 type=send: 530.86822
http/_chunky_http_client.js len=4 num=2000 type=send: 582.79023
http/_chunky_http_client.js len=8 num=5 type=send: 367.59354
http/_chunky_http_client.js len=8 num=50 type=send: 443.42095
http/_chunky_http_client.js len=8 num=500 type=send: 547.35562
http/_chunky_http_client.js len=8 num=2000 type=send: 483.79544
http/_chunky_http_client.js len=16 num=5 type=send: 338.36248
http/_chunky_http_client.js len=16 num=50 type=send: 446.57086
http/_chunky_http_client.js len=16 num=500 type=send: 486.92901
http/_chunky_http_client.js len=16 num=2000 type=send: 584.45133
http/_chunky_http_client.js len=32 num=5 type=send: 340.79843
http/_chunky_http_client.js len=32 num=50 type=send: 460.50859
http/_chunky_http_client.js len=32 num=500 type=send: 519.05640
http/_chunky_http_client.js len=32 num=2000 type=send: 586.19401
http/_chunky_http_client.js len=64 num=5 type=send: 272.44610
http/_chunky_http_client.js len=64 num=50 type=send: 447.09721
http/_chunky_http_client.js len=64 num=500 type=send: 515.87747
http/_chunky_http_client.js len=64 num=2000 type=send: 559.26372
http/_chunky_http_client.js len=128 num=5 type=send: 297.09491
http/_chunky_http_client.js len=128 num=50 type=send: 471.53426
http/_chunky_http_client.js len=128 num=500 type=send: 542.51722
http/_chunky_http_client.js len=128 num=2000 type=send: 560.05282

http/simple.js
http/simple.js type=bytes length=4 chunks=0 c=50: 25226.23000
http/simple.js type=bytes length=4 chunks=0 c=500: 18151.43000
http/simple.js type=bytes length=4 chunks=1 c=50: 20577.27000
http/simple.js type=bytes length=4 chunks=1 c=500: 15549.25000
http/simple.js type=bytes length=4 chunks=4 c=50: 14388.59000
http/simple.js type=bytes length=4 chunks=4 c=500: 9539.76000
http/simple.js type=bytes length=1024 chunks=0 c=50: 20358.75000
http/simple.js type=bytes length=1024 chunks=0 c=500: 15976.36000
http/simple.js type=bytes length=1024 chunks=1 c=50: 13583.89000
http/simple.js type=bytes length=1024 chunks=1 c=500: 11073.13000
http/simple.js type=bytes length=1024 chunks=4 c=50: 13663.83000
http/simple.js type=bytes length=1024 chunks=4 c=500: 8996.71000
http/simple.js type=bytes length=102400 chunks=0 c=50: 512.10000
http/simple.js type=bytes length=102400 chunks=0 c=500: 498.84000
http/simple.js type=bytes length=102400 chunks=1 c=50: 299.35000
http/simple.js type=bytes length=102400 chunks=1 c=500: 288.03000
http/simple.js type=bytes length=102400 chunks=4 c=50: 4429.59000
http/simple.js type=bytes length=102400 chunks=4 c=500: 3312.56000
http/simple.js type=buffer length=4 chunks=0 c=50: 21341.12000
http/simple.js type=buffer length=4 chunks=0 c=500: 15809.36000
http/simple.js type=buffer length=4 chunks=1 c=50: 20624.46000
http/simple.js type=buffer length=4 chunks=1 c=500: 15265.08000
http/simple.js type=buffer length=4 chunks=4 c=50: 17654.69000
http/simple.js type=buffer length=4 chunks=4 c=500: 13598.87000
http/simple.js type=buffer length=1024 chunks=0 c=50: 21631.03000
http/simple.js type=buffer length=1024 chunks=0 c=500: 15588.24000
http/simple.js type=buffer length=1024 chunks=1 c=50: 19921.18000
http/simple.js type=buffer length=1024 chunks=1 c=500: 14350.79000
http/simple.js type=buffer length=1024 chunks=4 c=50: 16370.73000
http/simple.js type=buffer length=1024 chunks=4 c=500: 12104.33000
http/simple.js type=buffer length=102400 chunks=0 c=50: 14908.88000
http/simple.js type=buffer length=102400 chunks=0 c=500: 11917.31000
http/simple.js type=buffer length=102400 chunks=1 c=50: 13243.06000
http/simple.js type=buffer length=102400 chunks=1 c=500: 11359.34000
http/simple.js type=buffer length=102400 chunks=4 c=50: 12691.17000
http/simple.js type=buffer length=102400 chunks=4 c=500: 10403.64000

As you can tell, there is a small bump in the numbers (as should be expected). The question is: is the bump enough to justify the addition of the strictMode switch? Or should we just go ahead and always perform the token check?

@brendanashworth

This comment has been minimized.

Member

brendanashworth commented Sep 1, 2015

Would it be semver-major if it was turned on by default?

@thefourtheye

View changes

doc/api/http.markdown Outdated
@@ -493,6 +504,9 @@ Options:
- `Agent` object: explicitly use the passed in `Agent`.
- `false`: opts out of connection pooling with an Agent, defaults request to
`Connection: close`.
- `strictMode`: When `true`, enables additional strict standards conformance

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

Is the bullet aligned with the previous bullets?

@thefourtheye

View changes

lib/_http_common.js Outdated
@@ -195,3 +195,11 @@ function httpSocketSetup(socket) {
socket.on('drain', ondrain);
}
exports.httpSocketSetup = httpSocketSetup;
/** Verifies that the given val is a valid HTTP token

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

We don't use multiline comments, right?

This comment has been minimized.

@trevnorris

trevnorris Sep 1, 2015

Contributor

I don't understand. Are you asking if we don't use /* */?

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

@trevnorris Yup. I think I have never seen /* ... */ in our js code base.

This comment has been minimized.

@targos

targos Sep 1, 2015

Member

It doesn't mean that we cannot use it. And it is already used.

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

@targos Oh okay. Thanks :-)

This comment has been minimized.

@trevnorris

trevnorris Sep 1, 2015

Contributor

All lines should be prepended with *. If there's a comment on the first line please only use one *.

@thefourtheye

View changes

lib/_http_outgoing.js Outdated
@@ -299,6 +300,11 @@ function storeHeader(self, state, field, value) {
// minimize the performance impact in the common case.
if (/[\r\n]/.test(value))
value = value.replace(/[\r\n]+[ \t]*/g, '');
// Protect against invalid header field names... but only in strictmode
if (self.strictMode && !common._checkIsHttpToken(field, 'Header name')) {

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

What is the second parameter to _checkIsHttpToken doing?

This comment has been minimized.

@jasnell

jasnell Sep 1, 2015

Member

Ah, good catch. That was an artifact from a previous edit. I originally had the method raising the error itself but thought better of it.

@thefourtheye

View changes

test/sequential/test-http-strictmode.js Outdated
'use strict';
const common = require('../common');
const assert = require('assert');
const EventEmitter = require('events').EventEmitter;

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

The EventEmitter at the last is not necessary.

@thefourtheye

View changes

test/sequential/test-http-strictmode.js Outdated
const http = require('http');
var count = 3;
var ee = new EventEmitter();

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

This also can be a const.

@thefourtheye

View changes

test/sequential/test-http-strictmode.js Outdated
assert.doesNotThrow(function() {
res.setHeader('testing_123', 123);
});
assert.throws(function() {

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

Missing the expected exception.

@thefourtheye

View changes

test/sequential/test-http-strictmode.js Outdated
});
server.listen(common.PORT, function() {
http.get({port:common.PORT}, function(res) {

This comment has been minimized.

@thefourtheye

thefourtheye Sep 1, 2015

Contributor

res unused.

@rvagg

This comment has been minimized.

Member

rvagg commented Sep 1, 2015

I'm a bit concerned about the impending beating we may be in for wrt performance in v4. We don't have good numbers comparing the current codebase to 0.10, 0.12 or any version of io.js. We're good at microbenchmarking between increments but not as a broad view. So, without us providing numbers it's going to fall on others to do their own benchmarking, likely in ways that we don't think are necessarily valid reflections of true performance—and even where it is we may be in for some negative press with lower numbers particularly because of the Buffer changes.

So my inclination for now would be that if this lands it should stay behind a flag for v4 and when we have a chance to investigate the larger perf story (or hear it from others..) we can consider turning it on by default for v5 or later.

@trevnorris

View changes

lib/_http_outgoing.js Outdated
@@ -208,7 +210,6 @@ OutgoingMessage.prototype._storeHeader = function(firstLine, headers) {
field = key;
value = headers[key];
}

This comment has been minimized.

@trevnorris

trevnorris Sep 1, 2015

Contributor

Invalid whitespace change.

@jasnell jasnell force-pushed the jasnell:sanitize-headers branch Sep 1, 2015

@jasnell

This comment has been minimized.

Member

jasnell commented Sep 1, 2015

Yes, keeping this behind a flag for now is likely the best option. However, @rvagg, how do you feel about that flag being part of the API itself as opposed to a command line switch? The advantage of having the switch in code is that it is only enabled per individual request or response, it's not set globally, minimizing the impact, but it does mean an additional API we're going to have to track.

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Sep 1, 2015

I'm not sure this needs to be behind a flag. It's already an opt-in with response.strictMode = true; Otherwise the false conditional checks will add negligible performance overhead.

@rvagg

This comment has been minimized.

Member

rvagg commented Sep 2, 2015

@trevnorris I believe by "flag" he was referring to response.strictMode, i.e. more API we are responsible for maintaining.

However, if this does become default behaviour then that could be easily removed/ignored. If it doesn't become default behaviour then we're stuck maintaining this functionality, in a Domains kind of way.

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Sep 2, 2015

Oh. Missed that the hope was for this to be the default behavior.

@rvagg rvagg force-pushed the nodejs:master branch from 11c25c2 to ba02bd0 Sep 6, 2015

@jasnell jasnell force-pushed the jasnell:sanitize-headers branch Sep 22, 2015

@jasnell

This comment has been minimized.

Member

jasnell commented Sep 22, 2015

@brendanashworth @trevnorris @rvagg @indutny @ChALkeR ... I updated the commit to remove the strictMode check, making the default behavior to throw if the header field contains invalid characters. Please take another look.

@indutny

This comment has been minimized.

Member

indutny commented Sep 22, 2015

LGTM if CI is green. Good job @jasnell !

@brendanashworth

View changes

test/sequential/test-http-strictmode.js Outdated
@@ -0,0 +1,56 @@
'use strict';

This comment has been minimized.

@brendanashworth

brendanashworth Sep 22, 2015

Member

Is it possible to have this test go in /parallel/?

This comment has been minimized.

@jasnell

jasnell Sep 22, 2015

Member

Certainly. I also just realized I didn't rename it to reflect the fact that strictMode isn't actually there. I'll make both changes now.

@rvagg

This comment has been minimized.

Member

rvagg commented Sep 22, 2015

pretty sure this needs to be upgraded to semver-major, I'm fine with landing it for v5 / master

@ChALkeR

View changes

lib/_http_common.js Outdated
function checkIsHttpToken(val) {
return typeof val === 'string' && token.test(val);
}
exports._checkIsHttpToken = checkIsHttpToken;

This comment has been minimized.

@ChALkeR

ChALkeR Sep 22, 2015

Member

Why not just exports.checkIsHttpToken?

This comment has been minimized.

@brendanashworth

brendanashworth Sep 22, 2015

Member

Or if you wanted to introduce a new internal module (internal/http/util?) with this as the first one inside... that'd be great too.

This comment has been minimized.

@jasnell

jasnell Sep 23, 2015

Member

I'm not entirely convinced that's needed at this point. We already have _http_common.js, which is supposed to be (at least in theory) an internal module. Adding this method to that makes sense and if we want to pull bits of _http_common.js into an internal module later on, then so be it.

@thefourtheye

View changes

test/parallel/test-http-invalidheaderfield.js Outdated
const ee = new EventEmitter();
var count = 3;
var server = http.createServer(function(req, res) {

This comment has been minimized.

@thefourtheye

thefourtheye Sep 24, 2015

Contributor

minor: this also can be const

@brendanashworth

This comment has been minimized.

Member

brendanashworth commented Sep 24, 2015

Not to be argumentative, but I don't think a CLI flag is a good option for this. I'd say just leave it out of LTS and keep it as default=on for v5.

@ChALkeR

This comment has been minimized.

Member

ChALkeR commented Sep 24, 2015

default=on for v5.

That is what I meant, of course.

leave it out of LTS

I doubt that it is a good idea to leave that out of LTS completely. It means that LTS users wouldn't get this security fix, which is not nice. Having and CLI flag for LTS would leave give them the opportunity to have this patch while still being on LTS.

@rvagg

This comment has been minimized.

Member

rvagg commented Sep 24, 2015

mm, I'm not a fan of CLI flags for this kind of thing either, a compromise might be to introduce a global on http that could be set, http.strictMode = true, tho I'd rather it just be an option on an API(s), createServer() perhaps

@ChALkeR

This comment has been minimized.

Member

ChALkeR commented Sep 24, 2015

@rvagg Note that the flag is needed only for v4, it won't be there in v5. Is it a good idea to introduce a flag and make people update all their code, and then remove that flag in the next version? All the code would have to be patched then for v4 support.

@rvagg

This comment has been minimized.

Member

rvagg commented Sep 24, 2015

same argument works for a commandline flag

@ChALkeR

This comment has been minimized.

Member

ChALkeR commented Sep 24, 2015

@rvagg Command line flag wouldn't pollute all the code.

@ChALkeR

This comment has been minimized.

Member

ChALkeR commented Sep 24, 2015

@rvagg Btw, that's a bit of an off-topic discussion on how this should be merged in 4.x.
Let's merge this to master first as a semver-major for the default behavior, then discuss how this should be backported to v4.

@jasnell

This comment has been minimized.

Member

jasnell commented Sep 24, 2015

@rvagg ... if this is going to be the default behavior in v5 and forward, then I'd rather not introduce an API level switch in v4 that turns it on as doing so would obviously require a code change later to address.

jasnell added some commits Aug 24, 2015

http: add checkIsHttpToken check for header fields
Per: nodejs/node-convergence-archive#13

This adds a new check for header and trailer fields names and method
names to ensure that they conform to the HTTP token rule. If they do
not, a `TypeError` is thrown.

Previously this had an additional `strictMode` option that has been
removed in favor of making the strict check the default (and only)
behavior.

Doc and test case are included.

On the client-side
```javascript
var http = require('http');
var url = require('url');
var p = url.parse('http://localhost:8888');
p.headers = {'testing 123': 123};
http.client(p, function(res) { }); // throws
```

On the server-side
```javascript
var http = require('http');
var server = http.createServer(function(req,res) {
  res.setHeader('testing 123', 123); // throws
  res.end('...');
});
```
test: move header field validity test to parallel, rename
Rename the header field validity test to remove the `strictmode`
reference since the `strictMode` check does not exist.

Move the test to parallel instead of sequential

@jasnell jasnell force-pushed the jasnell:sanitize-headers branch to c0a660f Sep 24, 2015

@jasnell

This comment has been minimized.

Member

jasnell commented Sep 24, 2015

@thefourtheye @rvagg @ChALkeR ... ok, updated. Can I get a final review on this so I can get it landed in master

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Sep 24, 2015

commit msg nit: the top line of "Per: nodejs/node-convergence-archive#13" placed at the bottom and replace "Per:" with "Ref:".

Change looks good. The http benchmarks are generally inconsistent. Any variations I see there are within the margin of error for running the basic set. Several of them went really wrong. e.g.

http/simple.js type=bytes length=4 chunks=0 c=500: 3.88000
@jasnell

This comment has been minimized.

Member

jasnell commented Sep 24, 2015

@trevnorris I noticed that inconsistency also and believe it's an artifact of the machine I'm running on -- I've seen it running the benchmarks against master as well. If you're ok with landing this, we can get some more reliable benchmarks separately after it's landed and tweak from there.

@trevnorris

This comment has been minimized.

Contributor

trevnorris commented Sep 24, 2015

@jasnell I'm fine with the change going into v5. I think the only way to make a more reliable benchmark for http is to have a machine in a more reliable state. Best that can be done on our own boxes is to run fewer benchmarks with more iterations. Also cycling the http server in between small runs.

@jasnell

This comment has been minimized.

Member

jasnell commented Sep 24, 2015

@jasnell

This comment has been minimized.

Member

jasnell commented Sep 24, 2015

Two CI failures that look completely unrelated to this PR

@thefourtheye

This comment has been minimized.

Contributor

thefourtheye commented Sep 25, 2015

LGTM - Trevor's comment about the commit log.

jasnell added a commit that referenced this pull request Sep 25, 2015

http: add checkIsHttpToken check for header fields
Ref: nodejs/node-convergence-archive#13

This adds a new check for header and trailer fields names and method
names to ensure that they conform to the HTTP token rule. If they do
not, a `TypeError` is thrown.

Previously this had an additional `strictMode` option that has been
removed in favor of making the strict check the default (and only)
behavior.

Doc and test case are included.

On the client-side
```javascript
var http = require('http');
var url = require('url');
var p = url.parse('http://localhost:8888');
p.headers = {'testing 123': 123};
http.client(p, function(res) { }); // throws
```

On the server-side
```javascript
var http = require('http');
var server = http.createServer(function(req,res) {
  res.setHeader('testing 123', 123); // throws
  res.end('...');
});
```

Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Trevor Norris <trevnorris@nodejs.org>
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Rod Vagg <rod@vagg.org>
PR-URL: #2526
@jasnell

This comment has been minimized.

Member

jasnell commented Sep 25, 2015

Landed in 6192c98. Commit log nit fixed.

@jasnell jasnell closed this Sep 25, 2015

@rvagg rvagg referenced this pull request Sep 30, 2015

Closed

Release proposal: v4.1.2 #3128

@Fishrock123 Fishrock123 added this to the 5.0.0 milestone Oct 6, 2015

@rvagg rvagg referenced this pull request Oct 27, 2015

Merged

Release proposal: v5.0.0 #3466

rvagg added a commit to rvagg/io.js that referenced this pull request Oct 29, 2015

2015-10-29, Version 5.0.0 (Stable)
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) nodejs#2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) nodejs#3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) nodejs#3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) nodejs#3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) nodejs#3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) nodejs#3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) nodejs#3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) nodejs#3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) nodejs#2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) nodejs#3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) nodejs#2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) nodejs#3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) nodejs#3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) nodejs#3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) nodejs#2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) nodejs#2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) nodejs#1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) nodejs#3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) nodejs#3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) nodejs#3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) nodejs#2595.

PR-URL: nodejs#3466

rvagg added a commit that referenced this pull request Oct 29, 2015

2015-10-29, Version 5.0.0 (Stable)
Notable changes:

* buffer: (Breaking) Removed both 'raw' and 'raws' encoding types from Buffer,
  these have been deprecated for a long time (Sakthipriyan Vairamani) #2859.
* console: (Breaking) Values reported by console.time() now have 3 decimals of
  accuracy added (Michaël Zasso) #3166.
* fs:
  - fs.readFile*(), fs.writeFile*(), and fs.appendFile*() now also accept a file
    descriptor as their first argument (Johannes Wüller) #3163.
  - (Breaking) In fs.readFile(), if an encoding is specified and the internal
    toString() fails the error is no longer thrown but is passed to the callback
    (Evan Lucas) #3485.
  - (Breaking) In fs.read() (using the fs.read(fd, length, position, encoding,
    callback) form), if the internal toString() fails the error is no longer
    thrown but is passed to the callback (Evan Lucas) #3503.
* http:
  - Fixed a bug where pipelined http requests would stall (Fedor Indutny) #3342.
  - (Breaking) When parsing HTTP, don't add duplicates of the following headers:
    Retry-After, ETag, Last-Modified, Server, Age, Expires. This is in addition
    to the following headers which already block duplicates: Content-Type,
    Content-Length, User-Agent, Referer, Host, Authorization,
    Proxy-Authorization, If-Modified-Since, If-Unmodified-Since, From, Location,
    Max-Forwards (James M Snell) #3090.
  - (Breaking) The callback argument to OutgoingMessage#setTimeout() must be a
    function or a TypeError is thrown (James M Snell) #3090.
  - (Breaking) HTTP methods and header names must now conform to the RFC 2616
    "token" rule, a list of allowed characters that excludes control characters
    and a number of separator characters. Specifically, methods and header names
    must now match /^[a-zA-Z0-9_!#$%&'*+.^`|~-]+$/ or a TypeError will be thrown
    (James M Snell) #2526.
* node:
  - (Breaking) Deprecated the _linklist module (Rich Trott) #3078.
  - (Breaking) Removed require.paths and require.registerExtension(), both had
    been previously set to throw Error when accessed
    (Sakthipriyan Vairamani) #2922.
* npm: Upgraded to version 3.3.6 from 2.14.7, see
  https://github.com/npm/npm/releases/tag/v3.3.6 for more details. This is a
  major version bump for npm and it has seen a significant amount of change.
  Please see the original npm v3.0.0 release notes for a list of major changes
  (Rebecca Turner) #3310.
* src: (Breaking) Bumped NODE_MODULE_VERSION to 47 from 46, this is necessary
  due to the V8 upgrade. Native add-ons will need to be recompiled
  (Rod Vagg) #3400.
* timers: Attempt to reuse the timer handle for setTimeout().unref(). This fixes
  a long-standing known issue where unrefed timers would perviously hold
  beforeExit open (Fedor Indutny) #3407.
* tls:
  - Added ALPN Support (Shigeki Ohtsu) #2564.
  - TLS options can now be passed in an object to createSecurePair()
    (Коренберг Марк) #2441.
  - (Breaking) The default minimum DH key size for tls.connect() is now 1024
    bits and a warning is shown when DH key size is less than 2048 bits. This a security consideration to prevent "logjam" attacks. A new minDHSize TLS
    option can be used to override the default. (Shigeki Ohtsu) #1831.
* util:
  - (Breaking) util.p() was deprecated for years, and has now been removed
    (Wyatt Preul) #3432.
  - (Breaking) util.inherits() can now work with ES6 classes. This is considered
    a breaking change because of potential subtle side-effects caused by a
    change from directly reassigning the prototype of the constructor using
    `ctor.prototype = Object.create(superCtor.prototype, { constructor: { ... } })`
    to using `Object.setPrototypeOf(ctor.prototype, superCtor.prototype)`
    (Michaël Zasso) #3455.
* v8: (Breaking) Upgraded to 4.6.85.25 from 4.5.103.35 (Ali Ijaz Sheikh) #3351.
  - Implements the spread operator, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_operator
    for further information.
  - Implements new.target, see
    https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new.target
    for further information.
* zlib: Decompression now throws on truncated input (e.g. unexpected end of
  file) (Yuval Brik) #2595.

PR-URL: #3466
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment