Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update query parser options #5280

Merged
merged 1 commit into from
Mar 19, 2023
Merged

Conversation

nflaig
Copy link
Member

@nflaig nflaig commented Mar 17, 2023

Motivation

Follow up PR to #5268. Updates the parser options to make array query string parser more consistent/predictable.

Description

Update query parser options, drop support for array query strings like id[0]=1&id[1]=2&id[2]=3 as those are not required to be OpenAPI spec compliant and results are inconsistent, see ljharb/qs#331. The schema validation will catch this and throw an error as parsed query string results in an object.

Parsing of different formats

Here are just a few examples how array query strings will be parsed, most example query strings are from OpenAPI Parameter Serialization

"id[0]=3&id[1]=4&id[3]=5" => {"id":{"0":"3","1":"4","3":"5"}} // results in "should be array" error
"id=3&id=4&id=5"          => {"id":["3","4","5"]} // supported format
"id=3,4,5"                => {"id":["3","4","5"]} // supported format
"id=3|4|5"                => {"id":"3|4|5"} // results in a string (not supported)
"id=3%204%205"            => {"id":"3 4 5"} // results in a string (not supported)
"id=3%2C4%2C5"            => {"id":"3,4,5"} // results in a string (not supported)
"id"                      => {"id":""} // empty string here is expected

@@ -45,12 +45,12 @@ export class RestApiServer {
ajv: {customOptions: {coerceTypes: "array"}},
querystringParser: (str) =>
qs.parse(str, {
// defaults to 20 but Beacon API spec allows max items of 30
arrayLimit: 30,
Copy link
Member Author

@nflaig nflaig Mar 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed arrayLimit as it does not acutally work as expected, see ljharb/qs#294, in fact, completely removed array parsing (values like id[0]=1&id[1]=2&id[2]=3) to make the parsing more predictable. This kind of array query strings were also not supported before updating the query parser in #5268, so I would assume nobody relies on that functionality at the moment.

comma: true,
// default limit of 1000 seems unnecessarily high, let's reduce it a bit
parameterLimit: 100,
Copy link
Member Author

@nflaig nflaig Mar 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameter limit silently cuts off exceeding values which is not ideal, using the default value of 1000 here should be fine and is only relevant if someone sends array queries like this id=3&id=4&id=5.

The parameter limit is anyhow not what should be used to limit DoS vectors, instead we should rely on maximum header size of the http server which is 16KB.

Some downstream tooling such as rocketpool might require this limit to be increased as they send requests like this eth/v1/beacon/states/head/validators?id=<1000 validator pubkeys, comma-separated> which requires much higher max header size.

For now, this configurable using the --max-http-header-size node cli option, see http.maxHeaderSize

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lodestar now also has a CLI flag --rest.headerLimit to configure this

@github-actions
Copy link
Contributor

github-actions bot commented Mar 17, 2023

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: bfecbf4 Previous: 3e254d5 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 708.79 us/op 858.84 us/op 0.83
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 62.691 us/op 46.946 us/op 1.34
BLS verify - blst-native 1.3021 ms/op 1.2068 ms/op 1.08
BLS verifyMultipleSignatures 3 - blst-native 2.6789 ms/op 2.4537 ms/op 1.09
BLS verifyMultipleSignatures 8 - blst-native 5.8764 ms/op 5.2782 ms/op 1.11
BLS verifyMultipleSignatures 32 - blst-native 21.474 ms/op 19.090 ms/op 1.12
BLS aggregatePubkeys 32 - blst-native 30.821 us/op 25.492 us/op 1.21
BLS aggregatePubkeys 128 - blst-native 115.52 us/op 99.563 us/op 1.16
getAttestationsForBlock 84.192 ms/op 52.169 ms/op 1.61
isKnown best case - 1 super set check 280.00 ns/op 259.00 ns/op 1.08
isKnown normal case - 2 super set checks 269.00 ns/op 250.00 ns/op 1.08
isKnown worse case - 16 super set checks 269.00 ns/op 251.00 ns/op 1.07
CheckpointStateCache - add get delete 6.1080 us/op 4.8910 us/op 1.25
validate gossip signedAggregateAndProof - struct 2.9515 ms/op 2.7733 ms/op 1.06
validate gossip attestation - struct 1.3965 ms/op 1.3268 ms/op 1.05
pickEth1Vote - no votes 1.4453 ms/op 1.2532 ms/op 1.15
pickEth1Vote - max votes 13.829 ms/op 8.4823 ms/op 1.63
pickEth1Vote - Eth1Data hashTreeRoot value x2048 11.245 ms/op 8.9131 ms/op 1.26
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 19.912 ms/op 14.476 ms/op 1.38
pickEth1Vote - Eth1Data fastSerialize value x2048 917.98 us/op 657.28 us/op 1.40
pickEth1Vote - Eth1Data fastSerialize tree x2048 7.0942 ms/op 7.9892 ms/op 0.89
bytes32 toHexString 1.0220 us/op 505.00 ns/op 2.02
bytes32 Buffer.toString(hex) 530.00 ns/op 372.00 ns/op 1.42
bytes32 Buffer.toString(hex) from Uint8Array 726.00 ns/op 570.00 ns/op 1.27
bytes32 Buffer.toString(hex) + 0x 476.00 ns/op 359.00 ns/op 1.33
Object access 1 prop 0.20800 ns/op 0.17000 ns/op 1.22
Map access 1 prop 0.17800 ns/op 0.15700 ns/op 1.13
Object get x1000 7.5670 ns/op 7.0510 ns/op 1.07
Map get x1000 0.60900 ns/op 0.59000 ns/op 1.03
Object set x1000 76.369 ns/op 52.842 ns/op 1.45
Map set x1000 63.317 ns/op 44.831 ns/op 1.41
Return object 10000 times 0.28920 ns/op 0.24110 ns/op 1.20
Throw Error 10000 times 5.4945 us/op 4.2063 us/op 1.31
fastMsgIdFn sha256 / 200 bytes 4.0760 us/op 3.5030 us/op 1.16
fastMsgIdFn h32 xxhash / 200 bytes 358.00 ns/op 284.00 ns/op 1.26
fastMsgIdFn h64 xxhash / 200 bytes 525.00 ns/op 401.00 ns/op 1.31
fastMsgIdFn sha256 / 1000 bytes 12.608 us/op 11.777 us/op 1.07
fastMsgIdFn h32 xxhash / 1000 bytes 469.00 ns/op 412.00 ns/op 1.14
fastMsgIdFn h64 xxhash / 1000 bytes 556.00 ns/op 477.00 ns/op 1.17
fastMsgIdFn sha256 / 10000 bytes 111.56 us/op 105.98 us/op 1.05
fastMsgIdFn h32 xxhash / 10000 bytes 2.2020 us/op 1.9330 us/op 1.14
fastMsgIdFn h64 xxhash / 10000 bytes 1.5090 us/op 1.3590 us/op 1.11
enrSubnets - fastDeserialize 64 bits 1.9690 us/op 1.3000 us/op 1.51
enrSubnets - ssz BitVector 64 bits 725.00 ns/op 511.00 ns/op 1.42
enrSubnets - fastDeserialize 4 bits 228.00 ns/op 174.00 ns/op 1.31
enrSubnets - ssz BitVector 4 bits 680.00 ns/op 501.00 ns/op 1.36
prioritizePeers score -10:0 att 32-0.1 sync 2-0 165.18 us/op 105.73 us/op 1.56
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 175.43 us/op 135.90 us/op 1.29
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 225.32 us/op 171.82 us/op 1.31
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 404.20 us/op 307.35 us/op 1.32
prioritizePeers score 0:0 att 64-1 sync 4-1 511.04 us/op 370.46 us/op 1.38
array of 16000 items push then shift 1.8735 us/op 1.6496 us/op 1.14
LinkedList of 16000 items push then shift 11.741 ns/op 8.8790 ns/op 1.32
array of 16000 items push then pop 147.49 ns/op 79.403 ns/op 1.86
LinkedList of 16000 items push then pop 11.061 ns/op 8.5830 ns/op 1.29
array of 24000 items push then shift 2.7986 us/op 2.3888 us/op 1.17
LinkedList of 24000 items push then shift 12.120 ns/op 8.8320 ns/op 1.37
array of 24000 items push then pop 113.77 ns/op 77.324 ns/op 1.47
LinkedList of 24000 items push then pop 10.331 ns/op 8.8410 ns/op 1.17
intersect bitArray bitLen 8 14.745 ns/op 13.371 ns/op 1.10
intersect array and set length 8 127.01 ns/op 78.140 ns/op 1.63
intersect bitArray bitLen 128 47.572 ns/op 44.027 ns/op 1.08
intersect array and set length 128 1.4116 us/op 1.0776 us/op 1.31
Buffer.concat 32 items 3.0760 us/op 2.6710 us/op 1.15
Uint8Array.set 32 items 2.6770 us/op 2.8230 us/op 0.95
pass gossip attestations to forkchoice per slot 4.2703 ms/op 2.3027 ms/op 1.85
computeDeltas 3.4434 ms/op 3.3753 ms/op 1.02
computeProposerBoostScoreFromBalances 1.9909 ms/op 1.7818 ms/op 1.12
altair processAttestation - 250000 vs - 7PWei normalcase 3.9232 ms/op 2.1475 ms/op 1.83
altair processAttestation - 250000 vs - 7PWei worstcase 6.6715 ms/op 3.3676 ms/op 1.98
altair processAttestation - setStatus - 1/6 committees join 181.67 us/op 140.81 us/op 1.29
altair processAttestation - setStatus - 1/3 committees join 331.25 us/op 277.61 us/op 1.19
altair processAttestation - setStatus - 1/2 committees join 487.21 us/op 374.16 us/op 1.30
altair processAttestation - setStatus - 2/3 committees join 661.12 us/op 469.92 us/op 1.41
altair processAttestation - setStatus - 4/5 committees join 837.13 us/op 658.26 us/op 1.27
altair processAttestation - setStatus - 100% committees join 1.0253 ms/op 778.13 us/op 1.32
altair processBlock - 250000 vs - 7PWei normalcase 27.190 ms/op 14.333 ms/op 1.90
altair processBlock - 250000 vs - 7PWei normalcase hashState 34.944 ms/op 27.627 ms/op 1.26
altair processBlock - 250000 vs - 7PWei worstcase 62.678 ms/op 48.011 ms/op 1.31
altair processBlock - 250000 vs - 7PWei worstcase hashState 81.566 ms/op 67.582 ms/op 1.21
phase0 processBlock - 250000 vs - 7PWei normalcase 2.9804 ms/op 1.9881 ms/op 1.50
phase0 processBlock - 250000 vs - 7PWei worstcase 34.667 ms/op 28.336 ms/op 1.22
altair processEth1Data - 250000 vs - 7PWei normalcase 703.47 us/op 454.89 us/op 1.55
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 13.911 us/op 6.6820 us/op 2.08
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 38.048 us/op 19.643 us/op 1.94
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 17.658 us/op 8.3950 us/op 2.10
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 12.473 us/op 6.7540 us/op 1.85
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 105.93 us/op 84.786 us/op 1.25
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 662.64 us/op 663.64 us/op 1.00
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 930.02 us/op 892.92 us/op 1.04
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 921.79 us/op 862.73 us/op 1.07
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.9694 ms/op 2.3289 ms/op 1.28
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.6944 ms/op 1.6964 ms/op 1.00
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 4.8268 ms/op 3.9421 ms/op 1.22
Tree 40 250000 create 534.02 ms/op 304.47 ms/op 1.75
Tree 40 250000 get(125000) 200.48 ns/op 183.56 ns/op 1.09
Tree 40 250000 set(125000) 1.1684 us/op 902.80 ns/op 1.29
Tree 40 250000 toArray() 23.194 ms/op 17.727 ms/op 1.31
Tree 40 250000 iterate all - toArray() + loop 24.094 ms/op 17.360 ms/op 1.39
Tree 40 250000 iterate all - get(i) 78.226 ms/op 68.003 ms/op 1.15
MutableVector 250000 create 12.830 ms/op 9.2260 ms/op 1.39
MutableVector 250000 get(125000) 6.5820 ns/op 6.4110 ns/op 1.03
MutableVector 250000 set(125000) 299.40 ns/op 266.11 ns/op 1.13
MutableVector 250000 toArray() 3.9028 ms/op 2.7547 ms/op 1.42
MutableVector 250000 iterate all - toArray() + loop 4.3481 ms/op 2.8591 ms/op 1.52
MutableVector 250000 iterate all - get(i) 1.6040 ms/op 1.5153 ms/op 1.06
Array 250000 create 3.6907 ms/op 2.7013 ms/op 1.37
Array 250000 clone - spread 1.2860 ms/op 1.2354 ms/op 1.04
Array 250000 get(125000) 0.63300 ns/op 0.59600 ns/op 1.06
Array 250000 set(125000) 0.72800 ns/op 0.68800 ns/op 1.06
Array 250000 iterate all - loop 86.041 us/op 92.963 us/op 0.93
effectiveBalanceIncrements clone Uint8Array 300000 60.939 us/op 31.823 us/op 1.91
effectiveBalanceIncrements clone MutableVector 300000 413.00 ns/op 411.00 ns/op 1.00
effectiveBalanceIncrements rw all Uint8Array 300000 173.13 us/op 168.67 us/op 1.03
effectiveBalanceIncrements rw all MutableVector 300000 125.71 ms/op 87.191 ms/op 1.44
phase0 afterProcessEpoch - 250000 vs - 7PWei 123.45 ms/op 115.40 ms/op 1.07
phase0 beforeProcessEpoch - 250000 vs - 7PWei 47.398 ms/op 37.137 ms/op 1.28
altair processEpoch - mainnet_e81889 355.64 ms/op 301.23 ms/op 1.18
mainnet_e81889 - altair beforeProcessEpoch 62.333 ms/op 48.822 ms/op 1.28
mainnet_e81889 - altair processJustificationAndFinalization 23.898 us/op 17.628 us/op 1.36
mainnet_e81889 - altair processInactivityUpdates 6.7205 ms/op 5.6307 ms/op 1.19
mainnet_e81889 - altair processRewardsAndPenalties 70.193 ms/op 65.904 ms/op 1.07
mainnet_e81889 - altair processRegistryUpdates 3.9270 us/op 2.8740 us/op 1.37
mainnet_e81889 - altair processSlashings 493.00 ns/op 487.00 ns/op 1.01
mainnet_e81889 - altair processEth1DataReset 693.00 ns/op 501.00 ns/op 1.38
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2761 ms/op 1.2467 ms/op 1.02
mainnet_e81889 - altair processSlashingsReset 9.0220 us/op 4.2330 us/op 2.13
mainnet_e81889 - altair processRandaoMixesReset 4.9720 us/op 6.2410 us/op 0.80
mainnet_e81889 - altair processHistoricalRootsUpdate 897.00 ns/op 668.00 ns/op 1.34
mainnet_e81889 - altair processParticipationFlagUpdates 4.3930 us/op 2.5860 us/op 1.70
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0790 us/op 520.00 ns/op 2.08
mainnet_e81889 - altair afterProcessEpoch 145.53 ms/op 129.55 ms/op 1.12
phase0 processEpoch - mainnet_e58758 397.56 ms/op 371.22 ms/op 1.07
mainnet_e58758 - phase0 beforeProcessEpoch 169.23 ms/op 134.98 ms/op 1.25
mainnet_e58758 - phase0 processJustificationAndFinalization 25.304 us/op 15.599 us/op 1.62
mainnet_e58758 - phase0 processRewardsAndPenalties 70.249 ms/op 62.481 ms/op 1.12
mainnet_e58758 - phase0 processRegistryUpdates 12.457 us/op 8.3090 us/op 1.50
mainnet_e58758 - phase0 processSlashings 871.00 ns/op 478.00 ns/op 1.82
mainnet_e58758 - phase0 processEth1DataReset 796.00 ns/op 445.00 ns/op 1.79
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.4346 ms/op 1.0077 ms/op 1.42
mainnet_e58758 - phase0 processSlashingsReset 4.6120 us/op 4.2230 us/op 1.09
mainnet_e58758 - phase0 processRandaoMixesReset 7.5230 us/op 4.7300 us/op 1.59
mainnet_e58758 - phase0 processHistoricalRootsUpdate 716.00 ns/op 532.00 ns/op 1.35
mainnet_e58758 - phase0 processParticipationRecordUpdates 5.1770 us/op 3.7030 us/op 1.40
mainnet_e58758 - phase0 afterProcessEpoch 108.40 ms/op 98.054 ms/op 1.11
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.3952 ms/op 1.2029 ms/op 1.16
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.6936 ms/op 1.4684 ms/op 1.15
altair processInactivityUpdates - 250000 normalcase 28.990 ms/op 25.896 ms/op 1.12
altair processInactivityUpdates - 250000 worstcase 30.984 ms/op 24.849 ms/op 1.25
phase0 processRegistryUpdates - 250000 normalcase 9.8710 us/op 6.6270 us/op 1.49
phase0 processRegistryUpdates - 250000 badcase_full_deposits 283.85 us/op 253.15 us/op 1.12
phase0 processRegistryUpdates - 250000 worstcase 0.5 144.42 ms/op 127.44 ms/op 1.13
altair processRewardsAndPenalties - 250000 normalcase 71.328 ms/op 65.836 ms/op 1.08
altair processRewardsAndPenalties - 250000 worstcase 74.323 ms/op 64.252 ms/op 1.16
phase0 getAttestationDeltas - 250000 normalcase 7.1161 ms/op 6.6414 ms/op 1.07
phase0 getAttestationDeltas - 250000 worstcase 8.4324 ms/op 6.6750 ms/op 1.26
phase0 processSlashings - 250000 worstcase 3.7759 ms/op 3.6179 ms/op 1.04
altair processSyncCommitteeUpdates - 250000 194.02 ms/op 173.28 ms/op 1.12
BeaconState.hashTreeRoot - No change 294.00 ns/op 270.00 ns/op 1.09
BeaconState.hashTreeRoot - 1 full validator 51.231 us/op 52.661 us/op 0.97
BeaconState.hashTreeRoot - 32 full validator 614.58 us/op 534.33 us/op 1.15
BeaconState.hashTreeRoot - 512 full validator 6.5365 ms/op 5.2872 ms/op 1.24
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 66.568 us/op 66.813 us/op 1.00
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.0573 ms/op 903.00 us/op 1.17
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 13.857 ms/op 11.516 ms/op 1.20
BeaconState.hashTreeRoot - 1 balances 48.193 us/op 52.398 us/op 0.92
BeaconState.hashTreeRoot - 32 balances 480.75 us/op 454.43 us/op 1.06
BeaconState.hashTreeRoot - 512 balances 4.8873 ms/op 4.2946 ms/op 1.14
BeaconState.hashTreeRoot - 250000 balances 82.847 ms/op 83.609 ms/op 0.99
aggregationBits - 2048 els - zipIndexesInBitList 23.522 us/op 17.334 us/op 1.36
regular array get 100000 times 44.530 us/op 34.413 us/op 1.29
wrappedArray get 100000 times 35.343 us/op 34.628 us/op 1.02
arrayWithProxy get 100000 times 16.545 ms/op 16.518 ms/op 1.00
ssz.Root.equals 606.00 ns/op 568.00 ns/op 1.07
byteArrayEquals 592.00 ns/op 581.00 ns/op 1.02
shuffle list - 16384 els 7.2351 ms/op 7.0637 ms/op 1.02
shuffle list - 250000 els 106.14 ms/op 101.68 ms/op 1.04
processSlot - 1 slots 9.5070 us/op 8.4630 us/op 1.12
processSlot - 32 slots 1.4184 ms/op 1.3330 ms/op 1.06
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 218.36 us/op 194.13 us/op 1.12
getCommitteeAssignments - req 1 vs - 250000 vc 3.0329 ms/op 2.9218 ms/op 1.04
getCommitteeAssignments - req 100 vs - 250000 vc 4.2515 ms/op 4.1793 ms/op 1.02
getCommitteeAssignments - req 1000 vs - 250000 vc 4.5231 ms/op 4.5001 ms/op 1.01
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.3900 ns/op 4.4600 ns/op 1.21
state getBlockRootAtSlot - 250000 vs - 7PWei 830.34 ns/op 831.06 ns/op 1.00
computeProposers - vc 250000 12.462 ms/op 10.808 ms/op 1.15
computeEpochShuffling - vc 250000 109.61 ms/op 104.08 ms/op 1.05
getNextSyncCommittee - vc 250000 189.91 ms/op 187.72 ms/op 1.01

by benchmarkbot/action

@nflaig nflaig marked this pull request as ready for review March 17, 2023 15:29
@nflaig nflaig requested a review from a team as a code owner March 17, 2023 15:29
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nflaig nflaig merged commit ae4e442 into unstable Mar 19, 2023
@nflaig nflaig deleted the nflaig/update-query-parser-opts branch March 19, 2023 09:06
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.7.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants