Drop fees by 10x due to the persistently higher exchange rate.

[mikehearn]

For discussion.

The last fee drop was by 5x (from 50k satoshis to 10k satoshis)
in the 0.8.2 release which was about 6 months ago.

The current fee is (assuming a $500 exchange rate) about 5 dollar
cents. The new fee after this patch is 0.5 cents.

Miners who prefer the higher fees are obviously still able to
use the command line flags to override this setting. Miners who
choose to create smaller blocks will select the highest-fee paying
transactions anyway.

This would hopefully be the last manual adjustment ever required
before floating fees become normal.

[luke-jr]

I think it makes sense to leave it alone until we have floating fees. (I don't feel strongly about this)

[laanwj]

@luke-jr I think that depends on whether floating fees make it into 0.9. If not, then this could be a stopgap until the 0.10 release...

[mikehearn]

Yeah. My gut sense is that smartfees might not really be widely deployed until EOQ1 next year (March/April time, perhaps). The existing code isn't even merged yet, then we need to find a solution for SPV clients, and then it will take a couple of months or more for most nodes and wallets to upgrade to the new protocol. Also smartfees would currently result in the fees going UP rather than down, which would make it a rather unpopular change.

So that's why I suggest this intermediate change. It could be deployed into a 0.8.x release.

I expect someone will bring up the orphan-fee-per-kb-cost thing. Fees used to be much lower than they are now, and yet Bitcoin has got faster and more efficient over time. So I'm not expecting some kind of minopocalypse from lower fees.

[petertodd]

@mikehearn Have you confirmed support from major pools? As eleuthria of BTC Guild pointed out earlier if transactions aren't being removed from the mempool that itself impacts block propagation and increases orphan rates: https://bitcointalk.org/index.php?topic=338452.msg3670185#msg3670185

"Fees used to be much lower than they are now" <- The orphan-cost is denominated in BTC not fiat; fees have always been higher in the past. If you don't understand this try re-deriving the equations Gronager and myself posted to the -dev email list - good homework problem.

[gavinandresen]

We can't just drop nMinTxFee, or early adopters will end up with stuck transactions before enough miners/relay nodes have upgraded.

Dropping nMinRelayTxFee for the 0.8.6 release is a good idea.

[wtogami]

Litecoin's ugly hack for dropping mintxfee was to temporarily suggest users addnode a decentralized group of interconnected supernodes. Within days of that deployment people were reasonably OK without it as long as they checked getpeerinfo for sufficient peer versions. With p2pool nodes hastening rollout it really wasn't too bad.

[mikehearn]

@gavinandresen - right, fixed.

[wtogami]

Is the "Merge branch 'master' of https://github.com/bitcoin/bitcoin" commit intended for this PR?

[mikehearn]

No, grr, git can be confusing sometimes. Fixed now. For some reason I have a little ghost next to my commits too - bizarre.

[BitcoinPullTester]

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/6a4c196dd64da2fd33dc7ae77a8cdd3e4cf0eff1 for binaries and test log.
This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/
Contact BlueMatt on freenode if something looks broken.

[wtogami]

Early in Litecoin 0.8.x dev we tested dropping minrelaytxfee in one release with the intent to allow that lower fee to be relayed safely before the next release that would reduce the mintxfee. To our surprise we found that insufficient fee tx were being included in the mempool and being mined despite an insufficient fee. Are you sure this won't happen here for 0.8.6?

[mikehearn]

Insufficient fee in what sense? The relay fee controls what makes it across the network and the mining code will simply order by whatever fees are attached. So that behaviour is what I'd expect to see. The other value controls what the client will produce.

[wtogami]

@mikehearn Oh! I just misunderstood how it is supposed to work.

[gavinandresen]

I'm having second thoughts on this.

The block-creation code uses CTransaction::nMinTxFee, aka -mintxfee, to decide when to stop adding transactions to the block.

So the likely result of this change will be nodes running with defaults will have memory pools that fill up with transactions paying between 1,000 and 10,000 satoshis/kilobyte.

The mining code should probably change to use CTransaction::nMinRelayTxFee. That will give the behavior Mike described (fill up the block with highest-fee transactions that meet the relaying criteria).

[mikehearn]

Right, txns with a fee less than nMinTxFee are treated the same as "free".

Is there a downside to simply removing that check entirely? Transactons < nMinRelayTxFee wouldn't enter the mempool in order to be considered for block inclusion at all, right?

Anyway - such a change is a different patch / pull req. @gavinandresen ACK for this change?

[petertodd]

@mikehearn Removing what check exatly?

[mikehearn]

Oh, never mind. Dumb idea, forget it. I keep forgetting that free transactions are special cased everywhere. The "min relay fee" is not really what it claims to be. More stuff that could be simplified.

[wtogami]

The underlying concern here is the chicken and egg problem of deploying a major fee reduction where the early adopters risk stuck transactions. Due to the random 8 outgoing connections, it really only works if a sufficiently large portion of the network allows relaying of the new lower fee.

Would this work to avoid this issue?

if (all 8 random outgoing connection peers are < 0.8.6) {
disconnect random peer;
try more random peers;
}

With this logic the network would naturally organize itself to better ensure the flow of the new lower fee tx's. The real question is would this open any DoS vector? I don't think so.

To better ensure that "0.8.6" or later peers are actually relaying the lower fee tx's, I further suggest that power-users who follow documentation know to manually addnode= BlueMatt's High Speed Relay network or other high capacity nodes. Manual addnode= for power users worked well for Litecoin's previous rollout of a major fee reduction especially with many p2pool node operators helping with the deployment of listening nodes. The addition of random outgoing connection churning logic would better the odds that users who use only defaults will eventually connect to a capable peer.

With these two measures I believe rolling out a major fee reduction can be relatively smooth, without any risk of changing the mempool or mining code paths.

(By "0.8.6" I also mean other known bad versions including 0.8.99.0. Bump master to 0.8.99.1 when it has the new lower fee so it can be successfully differentiated.)

[gmaxwell]

@wtogami Ah. No. Dropping peers based on version numbers creates potential network partitioning risks that are hard to analyze. When originally faced with the "wallet / relay / mining" behavior we split fee logic to have separate knobs to control wallet vs relay, so we could lower relay first. Doing that is completely smooth but not fast.

It's arguable if mining should follow the relay behavior or the wallet one, one or the other is sometimes convenient. Because of how centralized mining currently is there usually isn't a time to upgrade problem for mining. The point that the mempool will fill with junk we won't mine is pretty compelling though.

[wtogami]

@gmaxwell
The disconnect logic would be a special case. Yes, it can be sybiled, but given this logic only disconnects one peer until it finds an apparent 0.8.6+, is this any worse than the current situation where nodes have no chance of finding a capable peer if they had bad luck on their initial random selection?

This logic is an improvement to the fallback default. Power users need not rely on that at all. They can use the only assured solution, manual addnode= to supernode relays during initial rollout of the lower fee.

This is a way to upgrade to a lower fee without separating it into two version upgrades.

[mikehearn]

We've done fee drops like this before, so I don't think it needs to be complicated. Gavin is right that we should simplify the rules, but it's a separate commit/review thread.

[gavinandresen]

I spent a good chunk of time of yesterday thinking really hard about fees moving forward:
https://gist.github.com/gavinandresen/7670433

[mikehearn]

I didn't see drop fees under 0.8.6 changes - is that intentional? I really think we need an interim solution here. We're seeing more and more people choose deliberately to send transactions with no fee attached at all, and then get upset because their tx takes forever to confirm. Also I think we're going to need a solution for SPV nodes in 0.9, even if it's not a particularly amazing one (like just polling peers and averaging their answers). I've been thinking about better approaches too, but let's keep this thread for just this proposed change.

[gavinandresen]

Yes, not dropping default fees in 0.8.6 is intentional; it won't help if there is no room for lower-fee-transactions in blocks (and from what I see, there isn't, we've already got competition for the free/fee-paying space).

[gmaxwell]

We're seeing more and more people choose deliberately to send transactions with no fee attached at all

I'm confused by this statement. A low fee is never worse than no fee— its just sometimes no better.

[mikehearn]

But 0.8.6 is also going to make the default block size bigger (though not by any useful amount). So if the blocks got bigger then that should impose a downward pressure on fees. It won't do that if the network still has a hard-coded rule that's far too high.

Dropping relay fees is the right thing to do - wallets are free to still set higher fees if needed to get into blocks, and if miners wish to make bigger blocks and that puts downward pressure on prices wallets are free to respond to that.

For the rest of the proposal, I'm all for simplifying the rules (some of the most complex parts of bitcoinj are fee rule related). With regards to forcing miners to set block sizes themselves, here's an idea - why not just set the soft block size limit to 1mb in 0.8.6 and if that's too high and causes large orphan rates (which seems to be an open question given that Michael says he didn't see any difference when he increased his pools sizes) then that will provide an excellent alternative motivation for miners to learn about the settings. If it doesn't make orphan rates worse then great, we got lots of new capacity for actual users to use.

[gavinandresen]

"hard-coded rule that is far too high" -- according to all the calculations I've seen, and according to my watch-which-transactions-are-mined-fee-estimation-code, the hard-coded rule is too LOW right now.

Lowering the relay fee has two risks that I see: filling up memory with a bloated mempool of transactions that we'll never mine. And, because the 'dust' rule is computed based on the relay fee, opening ourselves up to TXOUT-spamming attacks.

I really don't want to start playing whack-a-mole with spamming DoS attacks, I'd much rather get to a floating fee system with NO hard-coded relay fee rule sooner...

[mikehearn]

I think we all agree floating fees are great and can't come soon enough .... the question is just one of what we do in the interim period.

These discussions are complicated because of the differing frames of reference. When I said the fees are far too high, my frame of reference is "what is a reasonable cost in dollars given what Bitcoin transactions used to cost". Given that the cost of bandwidth and CPU time hasn't changed a lot in the last two years, it stands to reason the cost of transactions should not have changed much either.

What you're comparing fees against is miners observed behaviour, but this is rather circular - we know that fees don't motivate miners, so they tend to ignore block sizes and spend all their time thinking about things like liquid cooling and stuff. ASICminer being a clear example of that (they still use the default 250kb soft limit). Because that's where the ROI is. So you say "fees are too low vs miner behaviour" and I say "fees are too high relative to what they used to cost/what we need to be competitive with other payment systems" and both views can be correct simultaneously, the push/pull in the middle being that miner behaviour is currently somewhat random.

The term "spam" is likewise undefined. One mans spam is another mans micropayment. We can always rely on Luke to remind us of this :) That's why I try to avoid using it.

If you don't bump the soft limits and lower the fees with 0.8.6, expect 4-6 months of complaints from users because I think that's how long it will take to get floating fees actually implemented and deployed everywhere.

[mikehearn]

Oh yeah, one other thing about 0.9/floating fees - if upgrading to a wallet that supports floating fees means suddenly transactions start costing a lot more, then a lot of users just won't upgrade if their existing transactions are "working". Because "working" means whatever merchants accept, and most merchants want to accept payment as fast as possible, this means in practice if a lot of users don't upgrade then payments will still appear to work for those who don't, because the pain of longer confirmation times and higher double spend rates will be pushed to the merchant side.

I know the theory is users will choose fees based on how fast they want confirmation, and I guess some sophisticated site operators will do that. But I can't see that happening for consumer wallets. For the common case of "I am buying a pizza and want it delivered in 30 minutes", I don't even know what kind of UI would be able to explain the fees and tradeoffs to people, I wouldn't even be able to explain it myself in person. BitPay doesn't wait for confirmations at all, because they're unexplainable and nobody wants to wait. The direction we're likely to go in with the Android wallet is to take out the block depth indicators entirely because they're just meaningless to anyone who isn't a protocol expert - which is now the majority of our users.

So for as long as the floating fee is calculated as higher than the min fee, I expect a lot of users to simply reject the whole notion and refuse to upgrade for as long as enough relay nodes exist to actually get the transaction to the merchant. Until they're forced by transactions completely failing to hit the network at all, that will slow down deployment even more.

So I see lots and lots of what we'd call "schedule risk" for floating fees. Just increasing block size and lowering the relay fee, on the other hand, can be done very quickly. Perhaps you could set a min block size at the same time to try and ensure the mempool empties out faster.

[jgarzik]

ACK - I think we need this short term for 0.9

[laanwj]

ACK

[gavinandresen]

ACK.

[sipa]

Is there any reason to assume this will help? My information is limited, but it was my impression that most fee-paying transactions already need to compete for block space?

[jgarzik]

Competing for space is fine... The transactions must first be relayed in order to compete for space. [vendor hat: on] BitPay definitely saw several examples of transactions which the network never relayed to our boundary nodes, yet we were able to get confirmed within 24 hours by directly pushing to some miner nodes (after manually getting the user to send support a hexdump).

The min-relay limit is there to prevent spam, not act as a miner price controls. We always want the min-relay limit lower than average.. ideally right around the fee price required to get confirmed after 144*2 blocks (ref #3722 and #3723)

[super3]

Although it would may unnecessary code complexity, part of me would like to see linearly reducing fees over a time period. If what @gavinandresen said about the mempool and TXOUT attacks happens, we could see that over time and fix it before any real problems occurred. If the drop is gradual may also address some the concerns that @wtogami posed.

@mikehearn Agree that higher floating fees might make the end user upset. Perhaps not making the drop fee so severe, so that so it can drop a little more when the floating fees come out?

[sipa]

I doubt this change will affect many transactions at all.

[petertodd]

@sipa Well, it'll affect all transactions given the obvious low-risk DoS attack it opens up.

[mikehearn]

After reading the 0.9rc2 release notes, I noticed that nobody ever adjusted the mining code to use nMinRelayFee for considering free transactions as Gavin suggested. This means miners will still consider transactions with the fee drop applied to be "free" which in turn would make them useless and mean it's not really possible to actually pay less fee, negating the point.

I suspect a communication screwup - I said that seemed like a separate pull req, but then I never made one, thinking Gavin would do it.

So I wonder if we should do a 0.9rc3 with nMinRelayFee being used in the mining code to choose the free tx cutoff. @gavinandresen WDYT?

[luke-jr]

@mikehearn Miners should not be running mainline code with defaults in any case. They are expected to set their own transaction acceptance policies.

[gavinandresen]

I think the mining code should use min(txfee, relaytxfee) as the "less than
this: treat it as free" amount.

And I agree, we should do a 0.9.0rc3 (when I'm back at a computer that can
codesign and gitian-build releases).

[pgrigor]

At the risk of cross-spamming this idea:

I've been thinking about the future direction for fees a lot and had the following idea: What if the amount of fees somehow affected, on a block-by-block basis, the difficulty for that block? This idea would require a hard fork but the more I think about it the more it appeals to me. I call it "weighted per-block difficulty."

The basic gist of it is:

1/ there will still be a "base" difficulty rate which would adjust approximately bi-weekly;

2/ the amount of fees included in a block, divided by the total input value for that block, would affect this "base" difficulty in a zero or positive way. Say the base difficulty is 1000 and the fees for the block comprise 10% of the total input value -- this would raise the difficulty to 1010. Of course the percentage would have to be thought out more than this. :)

A few ways this would affect miners' and users' that I've come up with:

1/ transactions which (probably, mistakenly) include huge fees will be shunned by miners as they will negatively affect their ability to solve a block;

2/ At present there is a perverse incentive for miners to include large fee transactions in a block over small fee incentives; they currently include small fee transactions out of the goodness of their heart because the block reward far outweighs any transaction fees they earn. This perverse incentive in the future will tend to, with a non-weighted difficulty rate, make miners include large fee transactions over small fee transactions — the fee pricing structure therefore, at present, is a POSITIVE FEEDBACK LOOP — when transaction fees are the lion’s share of the block reward the incentive will be to include higher and higher fees on transactions, ignoring lower-fee transactions. Needless to say this is a Bad Thing(tm).

3/ if the difficulty per block is reliant on the fees for that block the incentive then becomes to include the transactions with the lowest fees economically possible — however transactions with fees below a miner’s economic threshold will take longer to confirm, if they confirm at all;

4/ because the formula for adjusting difficulty would be based on the percentage of fees as they relate to the block’s total transaction inputs the idea is bitcoin-value agnostic -- always larger transactions with lower fees affect the difficulty less than small transactions with higher fees -- however fees would be stable as a percentage of the input(s);

5/ fees would be easily calculated by Bitcoin users by analyzing one or a few previous blocks; minimum, maximum, average and standard deviation of fees could be easily calculated;

6/ blocks could be just as easily confirmed by the reference implementation as the formula used by miners to adjust the base difficulty would be deterministic, based on the base difficulty, transaction input totals and fees included;

7/ Acceptance by the community would be a non-issue, as the algo favors a lower fee over a higher one;

I'd love to hear your thoughts...

[super3]

Certainly an interesting idea, but I personally don't feel like its worth the code complexity and effort. While it does provide an incentive the behavior we want, you loose me at "hard fork." I don't think its worth the disruption to the core Bitcoin network.

I would like, however, like to see an implementation of this in some coin.