When creating new blocks, sort 'paid' area by fee-per-kb

[gavinandresen]

Modify CreateNewBlock so that instead of processing all transactions
in priority order, process the first 27K of transactions in
priority order and then process the rest in fee-per-kilobyte
order.

This is the first, minimal step towards better a better fee-handling
system for both miners and end-users; this patch should be easy
to backport to the old versions of Bitcoin, and accomplishes the
most important goal-- allow users to "buy their way in" to blocks
using transaction fees.

Four params give miners control over the transaction selection process:

-blockmaxsize : maximum size of blocks created (default: 250000)
-blockprioritysize : how much of the block to set aside for high-priority-maybe-free transactions (default: 27000)
-blockminsize : minimum size of blocks created (default: 0)
-mintxfee : transactions that pay below this many bitcoins-per-kilobyte are considered "free" (default: 0.0005)

Discussion thread in the Mining forums: https://bitcointalk.org/index.php?topic=95837

[luke-jr]

Does this include the improvements in #1240?

[gavinandresen]

@luke-jr : no, it doesn't include anything from #1240.

The steps towards better fee handling, as I see them, are:

1. Get miners to prefer including higher fee-per-kb transactions in their blocks. That's what this pull request does.
2. Once a significant number of miners are doing that, modify the client to estimate what fee (if any) is needed to get into the next block or six, and let the user include a fee on a transaction-by-transaction basis.
3. Make more improvements to the transaction selection algorithm, like considering dependent transactions' fees when selecting what to include in a block (part of Refactor CreateNewBlock transaction selection algorithm #1240), and roll that out to miners and clients.

[luke-jr]

#1240 does both 1 and 3, and is well-tested...

[gavinandresen]

Reworked a little bit after getting some feedback from the big mining pools (added options -maxblocksize and -maxfreesize).

@luke-jr : I spent most of today looking carefully at how your proposed priority mechanism (the age_weight1 + fee_weight2 formula) and this one actually behave with the transactions that are in the main network's memory pool right now (I implemented just the "change the priority calculation" part of #1240 and ran it).

And I came to the conclusion that with your scheme it will be hard to get the weights right, and miners will have no idea what they're trading off if we give them the ability to fiddle with the weights. I wrote the code, and couldn't predict how many free transactions would be included in a block if I made weight1=1 and weight2=1000.

I like this method better; it is clear what will happen if you set -maxblocksize=50000 and -maxfreesize=10000 (defaults are 250,000 and 27,000 which match the behavior of previous releases).

[luke-jr]

How can I, with this version, prioritize by fee alone (only using age when fee is identical) and still include as many free transactions as possible?

[gavinandresen]

@luke-jr : you can't. Prioritizing by fee alone is too easy to game (spammer includes a 1-satoshi fee with all their transactions and crowds out all the zero-fee transactions).

Sorting by significantly-greater-than-zero-fee first, and then filling up to -maxblocksize with free transactions would also be easy to game (just send lots of zero-priority zero-fee transactions to make the miner always create -maxblocksize blocks). Or all the SatoshiDice fee-paying transactions would crowd out old clients that expect that their high-priority free transactions will get into a block eventually.

[gavinandresen]

@luke-jr : I think we could safely get the behavior you want by introducing another parameter: lets call it -blockminsize.

The rules would then be:
Up to -blockprioritysize, insert transactions based on priority.
Then insert as many fee-paying transactions as will fit, to a max of -blockmaxsize.
If the block is less than -blockminsize big, fill up to -minblocksize with free transactions (where "free" is less-than-TX_MIN_FEE per kilobyte, to exclude 1-satoshi-fee attacks)

You'd set -blockprioritysize=0, and set -blockmaxsize and -blockminsize to the size of blocks you want to create, and you'll get always-sorted-by-fee, include-free-if-there-is-extra-room. Depending on your block size, I believe these days your blocks would contain no free transactions, though, due to all the SatoshiDice fee-paying return transactions.

My only concern is that three different sizes is a lot of knobs to explain.

[luke-jr]

Better to have undocumented options, than not have them at all because they're too hard to document. IMO.

[gavinandresen]

Implemented -blockminsize, and cleaned up the implementation (use std::heap for the priority queue).

Behavior with default values on the network right now looks reasonable; blocks have a dozen or so high-priority transactions, then are filled with fee-paying transactions. Blocks created are between 60 and 100K big

[jgarzik]

ACK