Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

paytxfee behavior changed without warning #7633

Closed
mikegogulski opened this issue Mar 2, 2016 · 40 comments

Comments

Projects
None yet
@mikegogulski
Copy link

commented Mar 2, 2016

(updated with new information)
Calculation of the transaction fee when "paytxfee" is set has changed, without announcement or documentation.

Per @sipa :
In earlier versions however the size of a transaction was rounded up to a multiple of 1000, as that is how the mining code used to sort transaction years ago. In 0.12, it was finally changed to the accurate formula that works at an accuracy per byte. If the paytxfee value was always exactly what you saw as resulting fee, that must mean all your transactions were 1 kB in size.

In bitcoin.conf, I had

paytxfee=0.0001

In 0.11, that meant that BTC 0.0001 would be added to every outbound tx as a fee.

in 0.12, it means every tx will have added to it a fee of BTC 0.0001 per kilobyte

This is not documented in the release notes. As a result of this, I've got 3 days worth of spend transactions caught in unconfirmed limbo, having gone out with fees like 0.00002260 instead of 0.0001.

@sipa

This comment has been minimized.

Copy link
Member

commented Mar 2, 2016

@mikegogulski

This comment has been minimized.

Copy link
Author

commented Mar 2, 2016

Well that's pretty darned surprising:

$ bitcoin-cli listtransactions '*' 100|egrep '(confirmat|fee)'
    "fee": -0.00010000,
    "confirmations": 857,
    "fee": -0.00010000,
    "confirmations": 850,
    "fee": -0.00010000,
    "confirmations": 844,
    "fee": -0.00010000,
    "confirmations": 843,
    "fee": -0.00010000,
    "confirmations": 826,
    "fee": -0.00010000,
    "confirmations": 825,
    "fee": -0.00010000,
    "confirmations": 797,
    "fee": -0.00010000,
    "confirmations": 794,
    "fee": -0.00010000,
    "confirmations": 778,
    "fee": -0.00010000,
    "confirmations": 772,
    "fee": -0.00010000,
    "confirmations": 772,
    "fee": -0.00010000,
    "confirmations": 772,
    "fee": -0.00010000,
    "confirmations": 761,
    "fee": -0.00010000,
    "confirmations": 765,
    "fee": -0.00010000,
    "confirmations": 759,
    "fee": -0.00010000,
    "confirmations": 735,
    "fee": -0.00010000,
    "confirmations": 730,
    "confirmations": 699,
    "confirmations": 652,
    "fee": -0.00003740,      ### **Upgrade to 0.12**
    "confirmations": 639,
    "fee": -0.00003740,
    "confirmations": 616,
    "confirmations": 505,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 440,
    "confirmations": 453,
    "fee": -0.00003740,
    "confirmations": 0,
    "confirmations": 436,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 433,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 129,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 350,
    "fee": -0.00003740,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 322,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00005210,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00003730,
    "confirmations": 0,
    "fee": -0.00003730,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002250,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "fee": -0.00002260,
    "confirmations": 0,
    "confirmations": 9,
$ 
@sipa

This comment has been minimized.

Copy link
Member

commented Mar 2, 2016

@mikegogulski

This comment has been minimized.

Copy link
Author

commented Mar 2, 2016

I don't think so. It always sent fees exactly equal to the paytxfee config variable

@sipa

This comment has been minimized.

Copy link
Member

commented Mar 2, 2016

@mikegogulski

This comment has been minimized.

Copy link
Author

commented Mar 2, 2016

It's a breaking change. It should have been announced and documented. Very frustrating.

@mikegogulski

This comment has been minimized.

Copy link
Author

commented Mar 2, 2016

Put another way, it's not a breaking change. It's evidence of a missing set of tests.

@mikegogulski

This comment has been minimized.

Copy link
Author

commented Mar 2, 2016

Also, people need to be notified of this. I doubt I'm the only person affected, with transactions stuck in limbo hoping that the average block size drops a bit so that these get included. There's a 16,000 unconfirmed tx backlog right now.

@mikegogulski

This comment has been minimized.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

commented Mar 2, 2016

The description in issue is incorrect. Could you revise it based on the discussion here, since you've now promoted it?

@mikegogulski

This comment has been minimized.

Copy link
Author

commented Mar 2, 2016

Done!

@JokerCatz

This comment has been minimized.

Copy link

commented Mar 3, 2016

same issue with me #7634

@mikegogulski

This comment has been minimized.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Mar 3, 2016

@sipa is correct; paytxfee was always in BTC per kilobyte but bitcoin core 0.12 will no longer round up to the next kilobyte. And unfortunately, the release notes only implicitly mention that this changed: "pay a predefined fee rate by setting -paytxfee=<n>". (https://github.com/bitcoin/bitcoin/blob/v0.12.0/doc/release-notes.md#wallet-transaction-fees)

@mikegogulski

This comment has been minimized.

Copy link
Author

commented Mar 3, 2016

OK. Thank you @sipa and @MarcoFalke for clarifying the nature of this issue.

If either of you or @gmaxwell is unsatisfied with the current issue description, would you please suggest how to improve it?

@laanwj laanwj added the Wallet label Mar 3, 2016

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Mar 6, 2016

Is there any way to communicate this even though the release notes are impossible to change now? (E.g. add a section to the 0.12.1 release notes?)

@laanwj

This comment has been minimized.

Copy link
Member

commented Mar 8, 2016

I agree that this change of behavior should have been documented in the release notes, as it will result in - on average - in paying less fee if -paytxfee is used. This is good as rounding is wasteful and unnecessary, but I can see it result in confusion if it happens unexpectedly.

  • This is the commit in which Satoshi made paytxfee per kB, it has indeed been the case for a long time: 461764c (2010!)

I cannot find the commit/pull request that removed the rounding. Anyone?

Put another way, it's not a breaking change. It's evidence of a missing set of tests.

I agree that the wallet fee logic doesn't have very good testing.
This is a community effort, you're welcome to contribute more and better tests.

@laanwj

This comment has been minimized.

Copy link
Member

commented Mar 8, 2016

Is there any way to communicate this even though the release notes are impossible to change now? (E.g. add a section to the 0.12.1 release notes?)

What would help a lot, I think, would be to have documentation for the fee system - e.g. a user manual that describes how it is supposed to work, what the expectations should be for the fee computation. There are a lot of knobs, and it is far from clear from the --help output only how these are supposed to interact.

This causes both developers and users to make mistakes in that regard.

@laanwj laanwj added the Docs label Mar 8, 2016

@dooglus

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2016

@laanwj > I cannot find the commit/pull request that removed the rounding. Anyone?

Is this it?

commit c6cb21d
Author: Gavin Andresen gavinandresen@gmail.com
Date: Thu Apr 10 14:14:18 2014 -0400

Edit: incidentally I only yesterday noticed that I had settxfee=0.00025 in my bitcoin.conf instead of paytxfee=0.00025. I was wondering why every time I restarted bitcoind I had to manually use the RPC settxfee command again. That's just user error on my part, but the similarity of the config setting and the RPC command names probably contributed.

@OverlordQ

This comment has been minimized.

Copy link
Contributor

commented Mar 8, 2016

Or ecc7c82?

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Mar 8, 2016

I cannot find the commit/pull request that removed the rounding. Anyone?

In case you mean "removed the rounding in 0.12", it is a couple of pull request: #6649 (not merged), #6708 (not merged), #7103 (merged), #7096 (merged but not related to this issue, as it was rebased onto 7103)

What would help a lot, I think, would be to have documentation for the fee system

I tried to do exactly this in the release notes. Unfortunately I did not receive any feedback on the content, but I am happy to extend this. Maybe the website is a better place to curate such documentation?

@laanwj

This comment has been minimized.

Copy link
Member

commented Mar 11, 2016

In case you mean "removed the rounding in 0.12", it is a couple of pull request: #6649 (not merged), #6708 (not merged), #7103 (merged), #7096 (merged but not related to this issue, as it was rebased onto 7103)

Timeline:

  • 461764c: Added the rounding (back in 2010)
  • c6cb21d #3959: Removed the rounding (back in 2014). This was already in 0.10.
  • ed3e5e4 #5200: Added an option fPayAtLeastCustomFee which has, for <1kB transactions, the effect of rounding up. This was also in 0.10.
  • a775182 #7103: Default fPayAtLeastCustomFee to false. This made it into 0.12.

So all in all, rounding was already disabled in 0.10. It makes sense for this not to be mentioned in the release notes for 0.12. However what seems to have changed multiple times is the behavior for small transactions, due to fPayAtLeastCustomFee. This should have been mentioned.

I tried to do exactly this in the release notes. Unfortunately I did not receive any feedback on the content, but I am happy to extend this. Maybe the website is a better place to curate such documentation?

I don't really mind where, could just be some file in doc/ but having a document about the fee computation, anywhere, would be great.

laanwj added a commit that referenced this issue Mar 14, 2016

Revert "Default fPayAtLeastCustomFee to false"
This reverts commit 7d0a05f.

This change in fee behavior was unexpected to users and shouldn't
be in a backport release. See detailed timeline in
#7633 (comment)
for details.

People can upgrade to 0.12 if they want this new behavior.
@laanwj

This comment has been minimized.

Copy link
Member

commented Mar 14, 2016

Reverted these changes on the 0.11.x branch, so that they're not inadvertently imported there:

  • d49f724 - Revert "[wallet] Add rpc tests to verify fee calculations"
  • b6f5904 - Revert "Default fPayAtLeastCustomFee to false"

I agree that the wallet fee logic doesn't have very good testing.

I take this back, actually we did have to update the tests for this, see #7103 (a775182). So it isn't a testing problem, just a documentation one.

@tinspin

This comment has been minimized.

Copy link

commented Jul 15, 2016

Maybe a more permanent solution to the whole fee problem would be to simplify the transaction creation into preparesendfrom (because accounts are the only "off-chain" solution working today)/preparesendto that returns the transaction so that the user can see how much the total cost is going to be, then submittransaction that actually sends the transaction. All in easy to read JSON instead of raw hex. Hiding the transaction fees is confusing users and developers!

@sipa

This comment has been minimized.

Copy link
Member

commented Jul 15, 2016

@tinspin Well, you can already do that... with createrawtransaction + fundrawtransaction for preparesendto, and signrawtransaction + sendrawtransaction for submittransaction...

@tinspin

This comment has been minimized.

Copy link

commented Jul 15, 2016

Yes but the raw stuff is a bit, how can I put this, offputting. Specially when you are dealing with real money.

Edit: ok fundrawtransaction really helps alot... and this did to:

http://bitcoin.stackexchange.com/questions/44152/is-there-a-way-to-know-the-bitcoin-network-transaction-fee-before-actually-submi/44157

Even better:

http://bitcoin.stackexchange.com/questions/43532/transaction-fees-and-transactions-with-bitcoin-cli-wallet-address-balance-negat

But I'm surprised you can't know how large a transaction will be in bytes in advance?!

You can't control the network fee directly (as it is per byte, and you don't know the size beforehand)

You need to be able to put a fixed fee on a transaction in advance if the system is supposed to be usable?!

The exact size of a transaction can only be determined after it has been signed

Ok, I get it!

So it has to be createrawtransaction + fundrawtransaction + signrawtransaction for preparesendto, and sendrawtransaction for submittransaction... but how can I use raw stuff but associated with an account like sendfrom?

http://bitcoin.stackexchange.com/questions/24516/bitcoind-multisig-and-the-default-account

Ok, nm, I have to build my own account stuff or use sendfrom and suffer from the fee problem.

Or I hack my bitcoind to either take a fixed fee on sendfrom or I add a sendrawtransactionfrom method...

Edit; I tried looking at the code and it seems sendfrom uses a completely different implementation than sendrawtransaction?

Edit 2: #7729 but labels do not keep track of balances which is what we need accounts for.

@abelgo

This comment has been minimized.

Copy link

commented Nov 24, 2016

Dam. Is there a way to with out reverting back to v11 to set a fixed fee amount ?

@sipa

This comment has been minimized.

Copy link
Member

commented Nov 24, 2016

@abelgo

This comment has been minimized.

Copy link

commented Nov 24, 2016

Hmm that's just not an option right now, I have an upstream system written to work with what ever version of core we had 4 years ago. It deducts a set fee before sending the transaction and expects that to be the fee. I need a way to set a default fee, with out re-writing the upstream system to play whit transactions and its breaking a live system now by setting the fees too low... If I remember correctly back then there was a version of core when the fee was actually a fixed value.

@sipa

This comment has been minimized.

Copy link
Member

commented Nov 24, 2016

I'm telling that is not the case. It was pay per kilobyte, rounded up to kilobytes. Your upstream system makes incorrect assumptions, fix them.

@abelgo

This comment has been minimized.

Copy link

commented Nov 24, 2016

Yeah well, however it was doing. re writing the upstream system is just not a feasible option right now. Will have to push it back to v11 for the moment.

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Nov 24, 2016

@abelgo

This comment has been minimized.

Copy link

commented Nov 25, 2016

When I get the chance I will look as some other apis. Ideally you want to give your end user the option of deciding what fee they want to pay. But you have the difficulty of not being able to provide any certainty to your end user as you cant calculate that with out first building the transaction first. The other issue is dealing with the average end user. Some are not going to accept that there 1c transaction is going to cost more than there $1000 transaction because its ... "bigger" What you really want in terms of a API is to provide a simple, Send XXXX amount with fee of YYYY. An end users wants to know how much something will cost them before they do it.

@sipa

This comment has been minimized.

Copy link
Member

commented Nov 25, 2016

@abelgo

This comment has been minimized.

Copy link

commented Nov 25, 2016

Yeah I get that this is part of the nature of the design of the system. This problem is while it makes no sense to "the network" and is understandable to a developer. Its still something a novice end users wants and requests. The average user on my site wouldn't know the difference between a KB and a kb, and they don't know how many "blocks" they want there transaction confirmed in. They just want to know what it costs. This is one of the many disconnects that stops bitcoin becoming more mainstream. An API that thinks like the end user can help that. But yeah I get it.

@Aranad

This comment has been minimized.

Copy link

commented Nov 26, 2016

This looks like a really major problem, I have transactions from my local exchange have remained unconfirmed for nearly a week. The number of unconfirmed transactions is climbing at an alarming rate:
https://blockchain.info/unconfirmed-transactions (I just heard it peaked at 75K, then down to 28K yesterday, now 7K though, so perhaps things will pan out ok)

@jtimon

This comment has been minimized.

Copy link
Member

commented Nov 27, 2016

Some are not going to accept that there 1c transaction is going to cost more than there $1000 transaction because its ... "bigger" [...]

That's just how the system works: fees do not depend on amounts sent but on tx size. It may be counter-intuitive for people used to legacy systems, but it is actually an advantage in Bitcoin and we should simply educate users about it.

The average user on my site wouldn't know the difference between a KB and a kb, and they don't know how many "blocks" they want there transaction confirmed in. They just want to know what it costs.

You can present them only with the total fee before signing as suggested by sipa.
Regarding "how many blocks they want to wait" maybe you can explain them that a block takes approximately 10 minutes.

This looks like a really major problem, I have transactions from my local exchange have remained unconfirmed for nearly a week.

You should try higher fees, maybe consider using opt-in RBF. But this is not for technical support. I suggest IRC, maybe the bitcoin-discuss mailing list.

@unsystemizer

This comment has been minimized.

Copy link
Contributor

commented Nov 28, 2016

Regarding "how many blocks they want to wait" maybe you can explain them that a block takes approximately 10 minutes.

-txconfirmtarget=<n> allows them to do that, and on the API level per individual tx.
It's really just a doco issue, not a "major problem" as albego alleges. Once the UX creator is aware, he can let (or not) the user decide as they see fit.

@theochino

This comment has been minimized.

Copy link

commented Jun 7, 2017

Just got caught with my pants down. Created a transaction in a pinch;
Could the behavior be checked against the estimate fee and stop the TX if the fee smaller than it should ?
(Maybe adding a --force flag.)

@MarcoFalke

This comment has been minimized.

Copy link
Member

commented Mar 21, 2019

Closing this now. I think improved our process in the last two years to do a better job at documenting such changes in the release notes. Hopefully this will not happen again.

@MarcoFalke MarcoFalke closed this Mar 21, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.