Add 'subtractFeeFromAmount' option to 'fundrawtransaction'.

[dooglus]

I notice it is now possible to have the fee subtracted from the output amounts by specifying a subtractFeeFromAmount parameter in both sendtoaddress and sendmany, but not in fundrawtransaction.

This commit adds the option to fundrawtransaction.

[gmaxwell]

Concept ACK.

[jonasschnelli]

Nice. Concept ACK.
Needs test.

[dooglus]

@jonasschnelli I tried finding the fundrawtransaction tests but couldn't. Where are they?

src/test/rpc_tests.cpp seems like the natural place for them, but I see no 'fund' in there at all.

[jonasschnelli]

@dooglus
There is one at ./qa/rpc-tests/fundrawtransaction.py.
The tests should make sure that the subtractFeeFromAmount option work in conjunction with the custom feerate option (haven't look at your code so far).

[dooglus]

@jonasschnelli Thanks for pointing me at the qa/ directory. I hadn't noticed it before.

I have added tests for subtractFeeFromAmount, including checking that it works in combination with custom feerate.

[sipa]

Concept ACK

[jonasschnelli]

This will thrown an exception if one of the elements in the array is numeric. But I think this is okay.

[jonasschnelli]

Code Review ACK a979010.
Squash required.

[dooglus]

To 'squash' the commits do I just rewrite the same branch with a push --force? Or make a new branch and a new pull request?

[jonasschnelli]

@dooglus: Yes. I normally do a git rebase -I head~<amount-of-commits>, find the commit you'd like to squash to and mark all later commits with a s. Then git push --force.

[dooglus]

@jonasschnelli Thanks. The 'i' is lowercase and the 'HEAD' is uppercase but it was close enough.

I used git rebase -i HEAD~3 and it appears to have worked.

[morcos]

utACK

[mrbandrews]

nit: you can condense this to:
assert(result[0]['fee']==result[1]['fee']== result[2]['fee']==result[3]['fee'])

[dooglus]

Interesting. I didn't know Python did that. I will do as you suggest.

[dooglus]

Addressed in 6a41954.

[mrbandrews]

same

[dooglus]

Yes. Addressed in 6a41954.

[mrbandrews]

Should it throw an error if the given address is valid but is not among the outputs? (would have to check for this below, after retrieving the transaction). It seems like in this case, the user is trying to pay the fee with one of the outputs but has made an error.

[dooglus]

I wanted it to behave the same as it does in sendmany, where it doesn't complain if you include an address that isn't a recipient at all.

The user could have a list of addresses which should pay fees when sent to, and use that same list as their subtractFeeFromAmount parameter whichever addresses they are sending to.

[ryanofsky]

One difference between this and sendmany is that sendmany requires transaction outputs to be base58 addresses, and takes amounts and subtractfeefromamount arguments in base58 form, while fundrawtransaction allows outputs to be arbitrary scripts. This means with the PR in its current form, there may be no way for the new subtractFeeFromAmount argument to refer to certain outputs.

Instead of adding a subtractFeeFromAmount argument, I might suggest adding a subtractFeeFromPositions argument that takes a list of integer output indices. This would give callers the flexibility to refer to all outputs, be more consistent with the existing changePosition argument (which is also an integer output index), and also eliminate the need for ExtractDestination and CBitcoinAddress::ToString invocations in CWallet::FundTransaction.

[instagibbs]

@ryanofsky I like the idea but am a bit worried about the interaction of subtractFeeFromPositions and changePosition. It might not be clear to the user if the position marking is done before or after change output is added, or discount the wrong output by adding a changePosition argument.

[dooglus]

At the time of running fundrawtransaction there is no change output, and the user wouldn't know where the change will be inserted, so the position marking must be done before the change output is added.

I think since it is possible to add arbitrary hex output scripts which may not even have a corresponding address we need to be able to address the outputs by number rather than by address. It's also kind of ugly having to give the same address twice, once to createrawtransaction and then again to fundrawtransaction. I think using the output index (0 based) is cleaner.

[instagibbs]

@dooglus the user will "know" where change is going if they attempt to set the change index they're setting in the option, which is my point. It's not plainly clear how this should interact, unless you spell it out.

[dooglus]

Oh, I see. So I should spell it out...

I think it makes sense to use the position indices before the change output is added.

[dooglus]

Addressed @mrbandrews' nits. Should I re-squash now, or leave the 'nit' commit separate for a while?

[morcos]

re-utACK 6a41954

@dooglus good question, its not always clear. I personally think that if the prior code is not broken , then its ok not to squash.

[jonasschnelli]

utACK 6a41954
@dooglus IMO squashing is not required when the commits has a reason to be separated. If it's just a trivial change/overhaul of the previous commit(s) in the PR, it should probably be squashed.

[ryanofsky]

Could use list comprehension:

dec_tx = [self.nodes[3].decoderawtransaction(tx['hex'] for tx in result]

[ryanofsky]

Could use list comprehension (and similarly below):

output = [d['vout'][1 - r['changepos']]['value'] for d, r in zip(dec_tx, result)]

[ryanofsky]

Debug output will be a little better if you use assert_equal instead of assert here and below.

[dooglus]

It appears that assert_equal can only compare two things. For cases like assert(A == B == C == D) would you prefer 3 separate assert_equal() calls instead?

[MarcoFalke]

Makes sense. In case something fails we have the verbose output.

[ryanofsky]

I actually only meant to suggest using assert_equal for binary comparisons like the one on line 698. But if you wanted to use it more broadly, you could extend the function (in util.py) to accept more arguments:

def assert_equal(thing1, thing2, *args):
    if thing1 != thing2 or any(thing1 != arg for arg in args):
        raise AssertionError("!(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))

[dooglus]

Would it be better to extend assert_equal() to take an arbitrary number of parameters and have it compare them pairwise? Something like this would work:

def assert_equal(thing1, thing2, *other_things, depth=0):
    if thing1 != thing2:
        if depth or other_things:
            raise AssertionError("%s != %s (positions %d and %d)"%(str(thing1),str(thing2), depth, depth+1))
        else:
            raise AssertionError("%s != %s"%(str(thing1),str(thing2)))
    if other_things:
        assert_equal(thing2, *other_things, depth = depth + 1)

>>> assert_equal(4, 4, 5)
AssertionError: 4 != 5 (positions 1 and 2)

[dooglus]

I missed your last comment. Your solution is obviously much more elegant.

Is it acceptable to include a change like that in this pull request or should it be separate?

[MarcoFalke]

Fine to include it here.

[ryanofsky]

Could use dictionary comprehension:

outputs = {self.nodes[2].getnewaddress(): value for value in (1.0, 1.1, 1.2, 1.3)}

[dooglus]

Good idea, thanks.

[ryanofsky]

Could use list comprehension:

output = [[out[value] for i, out in enumerate(d['vout']) if i != r['changepos']]
          for d, r in zip(dec_tx, result)]]

[ryanofsky]

Maybe mention after This will not modify existing inputs, and will add one change output to the outputs above that no existing outputs will be modified either unless subtractFeeFromAmount is specified.

[ryanofsky]

Maybe s/If no addresses are specified here/If no addresses specified here are outputs in the transaction

[ryanofsky]

One difference between this and sendmany is that sendmany requires transaction outputs to be base58 addresses, and takes amounts and subtractfeefromamount arguments in base58 form, while fundrawtransaction allows outputs to be arbitrary scripts. This means with the PR in its current form, there may be no way for the new subtractFeeFromAmount argument to refer to certain outputs.

Instead of adding a subtractFeeFromAmount argument, I might suggest adding a subtractFeeFromPositions argument that takes a list of integer output indices. This would give callers the flexibility to refer to all outputs, be more consistent with the existing changePosition argument (which is also an integer output index), and also eliminate the need for ExtractDestination and CBitcoinAddress::ToString invocations in CWallet::FundTransaction.

[ryanofsky]

Would suggest changing the new set<string> argument to set<int> to be consistent with the existing nChangePosInOut argument which refers to an output by integer index instead of base58 address string. This would give callers more flexibility in referring to outputs and also simplify handling of the new argument below.

[dooglus]

I've addressed all the review comments, rebased, squashed, and pushed the resulting commit.

I'm wondering whether there's a potential issue with using integers to select which outputs to subtract the fee from, since the outputs are specified by a JSON dictionary, and dictionary keys are inherently unordered. Are we guaranteed when we createrawtransaction '[]' '{"a0":1,"a1":1,"a2":1}' that a<n> will be output <n>?

[instagibbs]

@dooglus good point, I don't think so. Recently ran into this writing extended rpc tests for something.

[dooglus]

@instagibbs me too:

$ python3
Python 3.4.2

>>> [k for k in {'a':1,'b':2,'c':3,'d':1,'e':1,'f':1}]
['c', 'd', 'f', 'b', 'e', 'a']

Since the input to fundrawtransaction is a raw transaction with its outputs already serialized this is less of an issue. But I tend to string my RPC calls together and expect the outputs to be serialized in the order I type them to createrawtransaction. They always do seem to be in the correct order except when using Python.

[sipa]

I don't think objects in JSON are assumed to have a meaningful ordering.

[MarcoFalke]

@dooglus For python you'd have to import OrderedDict (see #7980) but I don't think there is an ordered dict for json, so we should not rely on the order.

[dooglus]

So we are saying that it's OK to use a list of integer indexes into the list of outputs because:

1. by the time we're running fundrawtransaction the output list already has its order fixed (it's a raw transaction already, not a JSON object)
2. we have no other way to refer to general outputs, since they can be arbitrary hex strings and may not even have a base58 address
3. Bitcoin Core's use of univalue means that the JSON output list provided to createrawtransaction is always interpreted as an ordered dictionary anyway

Right?

(I tested point 3 as follows:

addr1=$(for x in {1..32}; do bitcoin-cli --testnet getnewaddress; done)
raw=$(bitcoin-cli --testnet createrawtransaction '[]' '{"'$(echo $addr1 | sed 's/ /":1,"/g')'":1}')
addr2=$(bitcoin-cli --testnet decoderawtransaction $raw | grep -E '^ {10}"' | cut -d'"' -f2)
echo $addr1 | sha1sum
echo $addr2 | sha1sum

and found that the outputs appear in the raw transaction in the same order as listed in the input to createrawtransaction)

[sipa]

I think reason (1) is enough to make position based indexing ok, and (2) strengthens it.

I don't think (3) is a good reason or something we should ever rely on. The only reason this is brought up is because createrawtransaction accepts an object to list the outputs. If strict ordering is expected there, perhaps we should change that argument from {"addr1":val1, "addr2":val2} to [["addr1",val1],["addr2",val2]] instead (in another PR).

[dooglus]

@sipa I'll look into making such a change in a separate PR. Using an object for the outputs not only means we cannot guarantee the order of the outputs but also having the addresses as dictionary keys means we can't have multiple outputs with the same address, which I sometimes like to do to (example).

I assume it would be best to allow the current object format in addition to the new array format for backwards compatibility.

[morcos]

+1 on address reuse being a sometimes valuable tool

[dooglus]

What happens next? I addressed all the comments. Is there something else I need to do?

[ryanofsky]

Change looks good as is, just left a few possible suggestions.

Lightly tested ACK 56ea974.

[ryanofsky]

This and the 5 following comments are basically just describing the asserts without adding any information. Could maybe remove the comments and condense the asserts.

[dooglus]

Agree.

[ryanofsky]

Maybe add comment describing output, could be # Nested list of non-change output amounts for each transaction

[dooglus]

OK.

[ryanofsky]

Maybe add comment like # List of difference in output amounts between normal and subtractFee transactions.

[dooglus]

OK.

[ryanofsky]

Since it is python not c, maybe replace ! with not

[dooglus]

I can't imagine who might have written that! ;)

Will change ! to not.

[ryanofsky]

Maybe just say an array instead of a json array, since the whole data structure is json.

[dooglus]

Agree, but it appears that everywhere else we refer to 'json array' (see sendmany, addmultisigaddress, lockunspent, listunspent...). Nowhere (in rpcwallet.cpp at least) do we simply say 'an array'.

Will leave as 'json array' for the sake of consistency.

[ryanofsky]

Little unusual to use %s for the main string instead of strprintf("Invalid parameter, duplicated position: %d", pos)

[dooglus]

Absolutely.

[ryanofsky]

New argument looks like it could be const reference

[dooglus]

Indeed.

[ryanofsky]

Little better to use size_t here instead of unsigned int.

[dooglus]

OK.

[dooglus]

Addressed @ryanofsky nits, rebased, squashed.

[ryanofsky]

Lightly tested ACK 453bda6

[dooglus]

Can this be merged now?