Modify account search with bounded scan mechanics

[emmanueldenloye]

This PR extends the /txs/account endpoint by applying the BoundedScan machinery to it. This way, when the Chainweb-Execution-Strategy: Bounded header is passed to it, it will only scan up to a predetermined number of rows in the database, just like the /txs/events and /txs/search endpoints.

Note that under normal circumstances, the bounded scan should be unnoticeable for the client of the /txs/account endpoint, because unlike the search endpoints (/txs/{events,search}), /txs/account fetches is results almost directly from the respective account,height indexes. However, it still applies some extra conditions that aren't directly on the indexes (such as the token type and the optional chain ID predicates). For the vast majority of the practical cases, these predicates shouldn't filter out enough rows to make the bounded scan query produce less rows than the limit, that's why we're currently not trying to address this concern by extending our indexes (and incurring other costs). So, having the bounded scan machinery under this endpoint makes sure that specifically crafted /txs/account queries can't hit various corner cases that make the server do a lot of work per request.

[enobayram]

I've just tested this PR with the following requests:

Unbounded requests

$ time curl -v "$CWDURL/txs/account/k:303bdc29c2c1d640728ff215d792c98bcdcd9dc25bb695a126281f95bfead494?token=coin&limit=15" | jq '.[] | { height, requestKey, idx}'
< Chainweb-Next: KCgzMzYxMDM0LCJBMTFteXNSWm9kV01UcTJoYkNxMGh4VFNNaURmM1h0ZXVuYXEzTmhldXF3IiwwKSwwKQ

{ "height": 3361151, "requestKey": "v7wotFVaYAlbfAQE4HyQZHBm1XxqoMgXaAoKJuFmeIo","idx": 0 }
{ "height": 3361131, "requestKey": "JcPPhEqbbXoZDiz7z2W1qblhnWiEekYzNTibAS2ZAgc","idx": 0 }
{ "height": 3361121, "requestKey": "X-5GxVeFZAiaXyezoeqrtuetuZ3lMrR1gTzs-uqHRh4","idx": 0 }
{ "height": 3361109, "requestKey": "7cQQP3hT3WRqDP5YoUAmgFG8-AjReBV6lI-T2r3yWjA","idx": 0 }
{ "height": 3361103, "requestKey": "wZkIXuGNsVRH-rlLBcIdiFaK7BE2rlKZl-bm8pVjF3k","idx": 0 }
{ "height": 3361084, "requestKey": "WtbkhsNNsNI3pXZXNwpR5O5wRtgzwuzNnei4OCuDd5o","idx": 0 }
{ "height": 3361079, "requestKey": "N1d0wqb8TbaFkiUrEDGs7ptEck4MJoESAAJp0WttcOA","idx": 0 }
{ "height": 3361078, "requestKey": "rwPK0b9V2-iIB4VfSj55Ur8UqbRqHuSzUOTza_Xm0N8","idx": 0 }
{ "height": 3361069, "requestKey": "ub1-JmcoiV-2No9dxBejgoBeVCvyfTeHSW5DoZCpv5w","idx": 0 }
{ "height": 3361065, "requestKey": "J6LP3GoViUl25Kd1V70AMhnb9DvIa3q0QfVV9gzMZ-g","idx": 0 }
{ "height": 3361060, "requestKey": "EZ_pu6sYxXbg9QfSeRxVQ8cNA20j-TvcGzFkBDysLcM","idx": 0 }
{ "height": 3361059, "requestKey": "lUwZLxucVrvVcAkI__YFwgl35__SCqt1BRJlBKs_UI4","idx": 0 }
{ "height": 3361052, "requestKey": "xppuA6ChN3DoJhjDVAqp-AoCBLNCgubloGZ1kHWsMAE","idx": 0 }
{ "height": 3361041, "requestKey": "h5ogjZ6tcxiLsPi97jVIoC4yQs11ipi1GXwge9BsNF8","idx": 0 }
{ "height": 3361034, "requestKey": "A11mysRZodWMTq2hbCq0hxTSMiDf3Xteunaq3Nheuqw","idx": 0 }

real	0m0,271s

$ time curl -v "$CWDURL/txs/account/k:303bdc29c2c1d640728ff215d792c98bcdcd9dc25bb695a126281f95bfead494?token=coin&limit=5&offset=5" | jq '.[] | { height, requestKey, idx}'
< Chainweb-Next: KCgzMzYxMDY1LCJKNkxQM0dvVmlVbDI1S2QxVjcwQU1obmI5RHZJYTNxMFFmVlY5Z3pNWi1nIiwwKSwwKQ

{ "height": 3361084, "requestKey": "WtbkhsNNsNI3pXZXNwpR5O5wRtgzwuzNnei4OCuDd5o", "idx": 0 }
{ "height": 3361079, "requestKey": "N1d0wqb8TbaFkiUrEDGs7ptEck4MJoESAAJp0WttcOA", "idx": 0 }
{ "height": 3361078, "requestKey": "rwPK0b9V2-iIB4VfSj55Ur8UqbRqHuSzUOTza_Xm0N8", "idx": 0 }
{ "height": 3361069, "requestKey": "ub1-JmcoiV-2No9dxBejgoBeVCvyfTeHSW5DoZCpv5w", "idx": 0 }
{ "height": 3361065, "requestKey": "J6LP3GoViUl25Kd1V70AMhnb9DvIa3q0QfVV9gzMZ-g", "idx": 0 }

real	0m0,134s

$ time curl -v "$CWDURL/txs/account/k:303bdc29c2c1d640728ff215d792c98bcdcd9dc25bb695a126281f95bfead494?token=coin&limit=5&next=KCgzMzYxMDY1LCJKNkxQM0dvVmlVbDI1S2QxVjcwQU1obmI5RHZJYTNxMFFmVlY5Z3pNWi1nIiwwKSwwKQ" | jq '.[] | { height, requestKey, idx}'
< Chainweb-Next: KCgzMzYxMDM0LCJBMTFteXNSWm9kV01UcTJoYkNxMGh4VFNNaURmM1h0ZXVuYXEzTmhldXF3IiwwKSwwKQ

{ "height": 3361060, "requestKey": "EZ_pu6sYxXbg9QfSeRxVQ8cNA20j-TvcGzFkBDysLcM", "idx": 0 }
{ "height": 3361059, "requestKey": "lUwZLxucVrvVcAkI__YFwgl35__SCqt1BRJlBKs_UI4", "idx": 0 }
{ "height": 3361052, "requestKey": "xppuA6ChN3DoJhjDVAqp-AoCBLNCgubloGZ1kHWsMAE", "idx": 0 }
{ "height": 3361041, "requestKey": "h5ogjZ6tcxiLsPi97jVIoC4yQs11ipi1GXwge9BsNF8", "idx": 0 }
{ "height": 3361034, "requestKey": "A11mysRZodWMTq2hbCq0hxTSMiDf3Xteunaq3Nheuqw", "idx": 0 }

real	0m0,133s

Bounded requests

$ time curl -v -H 'Chainweb-Execution-Strategy:Bounded' "$CWDURL/txs/account/k:303bdc29c2c1d640728ff215d792c98bcdcd9dc25bb695a126281f95bfead494?token=coin&limit=15" | jq '.[] | { height, requestKey, idx}'
< Chainweb-Next: KCgzMzYxMDM0LCJBMTFteXNSWm9kV01UcTJoYkNxMGh4VFNNaURmM1h0ZXVuYXEzTmhldXF3IiwwKSwwKQ

{ "height": 3361151, "requestKey": "v7wotFVaYAlbfAQE4HyQZHBm1XxqoMgXaAoKJuFmeIo","idx": 0 }
{ "height": 3361131, "requestKey": "JcPPhEqbbXoZDiz7z2W1qblhnWiEekYzNTibAS2ZAgc","idx": 0 }
{ "height": 3361121, "requestKey": "X-5GxVeFZAiaXyezoeqrtuetuZ3lMrR1gTzs-uqHRh4","idx": 0 }
{ "height": 3361109, "requestKey": "7cQQP3hT3WRqDP5YoUAmgFG8-AjReBV6lI-T2r3yWjA","idx": 0 }
{ "height": 3361103, "requestKey": "wZkIXuGNsVRH-rlLBcIdiFaK7BE2rlKZl-bm8pVjF3k","idx": 0 }
{ "height": 3361084, "requestKey": "WtbkhsNNsNI3pXZXNwpR5O5wRtgzwuzNnei4OCuDd5o","idx": 0 }
{ "height": 3361079, "requestKey": "N1d0wqb8TbaFkiUrEDGs7ptEck4MJoESAAJp0WttcOA","idx": 0 }
{ "height": 3361078, "requestKey": "rwPK0b9V2-iIB4VfSj55Ur8UqbRqHuSzUOTza_Xm0N8","idx": 0 }
{ "height": 3361069, "requestKey": "ub1-JmcoiV-2No9dxBejgoBeVCvyfTeHSW5DoZCpv5w","idx": 0 }
{ "height": 3361065, "requestKey": "J6LP3GoViUl25Kd1V70AMhnb9DvIa3q0QfVV9gzMZ-g","idx": 0 }
{ "height": 3361060, "requestKey": "EZ_pu6sYxXbg9QfSeRxVQ8cNA20j-TvcGzFkBDysLcM","idx": 0 }
{ "height": 3361059, "requestKey": "lUwZLxucVrvVcAkI__YFwgl35__SCqt1BRJlBKs_UI4","idx": 0 }
{ "height": 3361052, "requestKey": "xppuA6ChN3DoJhjDVAqp-AoCBLNCgubloGZ1kHWsMAE","idx": 0 }
{ "height": 3361041, "requestKey": "h5ogjZ6tcxiLsPi97jVIoC4yQs11ipi1GXwge9BsNF8","idx": 0 }
{ "height": 3361034, "requestKey": "A11mysRZodWMTq2hbCq0hxTSMiDf3Xteunaq3Nheuqw","idx": 0 }

real	0m0,117s

$ time curl -v -H 'Chainweb-Execution-Strategy:Bounded' "$CWDURL/txs/account/k:303bdc29c2c1d640728ff215d792c98bcdcd9dc25bb695a126281f95bfead494?token=coin&limit=5&offset=5" | jq '.[] | { height, requestKey, idx}'
< Chainweb-Next: KCgzMzYxMDY1LCJKNkxQM0dvVmlVbDI1S2QxVjcwQU1obmI5RHZJYTNxMFFmVlY5Z3pNWi1nIiwwKSwwKQ


{ "height": 3361084, "requestKey": "WtbkhsNNsNI3pXZXNwpR5O5wRtgzwuzNnei4OCuDd5o","idx": 0 }
{ "height": 3361079, "requestKey": "N1d0wqb8TbaFkiUrEDGs7ptEck4MJoESAAJp0WttcOA","idx": 0 }
{ "height": 3361078, "requestKey": "rwPK0b9V2-iIB4VfSj55Ur8UqbRqHuSzUOTza_Xm0N8","idx": 0 }
{ "height": 3361069, "requestKey": "ub1-JmcoiV-2No9dxBejgoBeVCvyfTeHSW5DoZCpv5w","idx": 0 }
{ "height": 3361065, "requestKey": "J6LP3GoViUl25Kd1V70AMhnb9DvIa3q0QfVV9gzMZ-g","idx": 0 }

real	0m0,116s

$ time curl -v -H 'Chainweb-Execution-Strategy:Bounded' "$CWDURL/txs/account/k:303bdc29c2c1d640728ff215d792c98bcdcd9dc25bb695a126281f95bfead494?token=coin&limit=5&next=KCgzMzYxMDY1LCJKNkxQM0dvVmlVbDI1S2QxVjcwQU1obmI5RHZJYTNxMFFmVlY5Z3pNWi1nIiwwKSwwKQ" | jq '.[] | { height, requestKey, idx}'
< Chainweb-Next: KCgzMzYxMDM0LCJBMTFteXNSWm9kV01UcTJoYkNxMGh4VFNNaURmM1h0ZXVuYXEzTmhldXF3IiwwKSwwKQ

{ "height": 3361060, "requestKey": "EZ_pu6sYxXbg9QfSeRxVQ8cNA20j-TvcGzFkBDysLcM","idx": 0 }
{ "height": 3361059, "requestKey": "lUwZLxucVrvVcAkI__YFwgl35__SCqt1BRJlBKs_UI4","idx": 0 }
{ "height": 3361052, "requestKey": "xppuA6ChN3DoJhjDVAqp-AoCBLNCgubloGZ1kHWsMAE","idx": 0 }
{ "height": 3361041, "requestKey": "h5ogjZ6tcxiLsPi97jVIoC4yQs11ipi1GXwge9BsNF8","idx": 0 }
{ "height": 3361034, "requestKey": "A11mysRZodWMTq2hbCq0hxTSMiDf3Xteunaq3Nheuqw","idx": 0 }

real	0m0,106s

[enobayram]

Here's an example of how the bounded scan machinery protects the server: k:22fbc926b497bf3ecc7199b923039e74cdeda12726e1dc4f443953007cf8cc43 is a miner account with more than 2 million transfers (mostly due to coinbase events). The following unbounded query makes CW-D scan through all of the transfers belonging to this account:

$ time curl -v "$CWDURL/txs/account/k:22fbc926b497bf3ecc7199b923039e74cdeda12726e1dc4f443953007cf8cc43?token=notcoin" | jq '.[] | { height, requestKey, idx}'
(No Chainweb-Next header)
(Empty response)

real	0m1,923s

Note that it takes ~2 seconds and there's no Chainweb-Next token because the results were fully scanned. Here's the bounded version:

$ time curl -v -H 'Chainweb-Execution-Strategy:Bounded' "$CWDURL/txs/account/k:22fbc926b497bf3ecc7199b923039e74cdeda12726e1dc4f443953007cf8cc43?token=notcoin" | jq '.[] | { height, requestKey, idx}'
< Chainweb-Next: KCgzMzUxNDg0LCJjYiIsMCksMCk
(Empty response)

real	0m0,197s

It takes roughly as much as the other queries, and it has a Chainweb-Next token.

[enobayram]

LGTM!