tests: Speedup feature_pruning test and refactor big transaction logic

[conscott]

Description

Extended tests like feature_pruning.py create large blocks by making many large transactions with a lot of OP_RETURNS in a single tx. Under this model roughly 14 of these transactions make a full-ish block.

These transactions are already non-standard, so I have just modified the logic to instead just create 1 large output, making a single large transaction, mining a block of roughly the same size (947k). The reduction in tx creation / number of outputs seems to significantly improve performance.

Another optimization is to splice in the OP_RETURN output after transaction signing, since it reduces the copying of the large OP_RETURN portion during signing. All signatures are SIGHHASH_NONE, so splicing after is okay.

feature_pruning.py has about a 1.5x speedup on my machine (~12 minutes down to ~8 minutes), and mempool_limit, and feature_maxuploadtarget seem to benefit as well.

Refactoring

• The original OP_RETURN generating function is now just a string. If there is a better place to put this, I will move it. At the very least I just wanted to document the format because the previous function had little explanation for all the hex values.
• create_lots_of_big_transactions no longer requires providing the OP_RETURN splice, and uses the default from before, with an optional op_return_txout= keyword args for using the giant OP_RETURN mentioned above.

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #15282 (test: Replace hard-coded hex tx with class in test framework by stevenroose)
• #13541 (wallet/rpc: sendrawtransaction maxfeerate by kallewoof)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Coverage

Coverage	Change (pull 14691)	Reference (master)
Lines	+0.0395 %	87.0730 %
Functions	+0.0153 %	84.3717 %
Branches	+0.0151 %	51.5638 %

^{Updated at: 2018-11-08T22:02:58.107718.}

[fanquake]

Concept ACK

Haven't looked at the changes, but I'm seeing a ~1.4x speedup on my machine between feature_pruning.py in master (e70a19e) and c8ef522:

feature_pruning.py | ✓ Passed  | 819 s
ALL                | ✓ Passed  | 819 s (accumulated) 
Runtime: 819 s

feature_pruning.py | ✓ Passed  | 577 s
ALL                | ✓ Passed  | 577 s (accumulated) 
Runtime: 577 s

[DrahtBot]

Needs rebase

[fanquake]

@conscott I realise this hasn't seen much traction / concept ACKs, however are you interested in rebasing?

[conscott]

I almost forgot this was still alive :)

I will rebase when I get the chance, hopefully this week!

[sipa]

Concept ACK

[promag]

Concept ACK.

[promag]

Could simplify by ditching num parameter? This way below could be

return [create_big_transaction(node, utxo, fee, op_return_txout) for utxo in utxos]

[jamesob]

Concept ACK. Will review on rebase.

[jnewbery]

Is this still required after #15686?

[conscott]

@jnewbery - Looks like you got it covered more elegantly with OP_NOPs :) It was a fun learning experience anyhow.

I will go ahead and close this.