Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
Already on GitHub? Sign in to your account
[rpc] Allow fetching tx directly from specified block in getrawtransaction #10275
Conversation
fanquake
added
the
RPC/REST/ZMQ
label
Apr 25, 2017
| + uint256 hash; | ||
| + int64_t blockHeight = -1; | ||
| + bool inMainChain = true; | ||
| + auto p = request.params[0].get_str().find(':'); |
laanwj
Apr 25, 2017
•
Owner
This functionality could come in useful.
As for the API I prefer to not do any string combining/parsing here, this makes the API less clean to work with at least in my experience. I'd prefer to add an optional (can be null or missing) fromblock argument.
gmaxwell
reviewed
Apr 25, 2017
Concept ACK. I've really wanted this before.
Allowing it work on orphan blocks is an interesting idea. I'm a little less sure about that-- I think it could allow it to return transactions that have never been validated, which would be somewhat surprising.
|
Nice feature! |
|
Fair enough, I'll add a blockhash argument instead. I was kind of toying with the idea of a new standard for referencing transactions which included the block height (not hash) so everyone could always find a tx presuming they had the block in question, and that thought sort of seeped in here. |
I just realized my logic was flawed on this. I am passing the block height only to the GetTransaction method, which means it will always pick the active chain at the given height. Either I throw when the block is not in the main chain (i.e. no support for orphaned blocks) or I move the height determine logic over to validation.cpp. I am leaning towards the latter, but feedback welcome. Edit: passing CBlockIndex to GetTransaction seems like a great way to do this. Going with that. |
|
The code now works as advertised (see updated OP).
|
| - fVerbose = true; | ||
| - } | ||
| + fVerbose = (request.params[1].get_int() != 0); | ||
| + } else if (request.params[1].isBool()) { |
jnewbery
May 2, 2017
Member
nit: You can replace this and the next three lines with:
} else {
fVerbose = (request.params[1].get_bool());
}since get_bool() does the type testing for you and throws the JSONRPCError if the type isn't a VBOOL.
Up to you whether you think that's clearer.
kallewoof
May 4, 2017
•
Member
Ohh, good point! Thanks.
Edit: I don't want it to throw for null values though.
kallewoof
May 4, 2017
Member
So I end up with
if (request.params[1].isNum()) {
fVerbose = (request.params[1].get_int() != 0);
} else if (!request.params[1].isNull()) {
fVerbose = (request.params[1].get_bool());
}|
utACK, but I think this deserves a new functional test case. |
|
@jnewbery I agree. Will get to work on that.
|
jnewbery
reviewed
May 15, 2017
tested ACK the integration test in a6b8461 with a couple of nits.
| + # make a tx by sending then generate 2 blocks; block1 has the tx in it, | ||
| + # presumably | ||
| + tx = self.nodes[2].sendtoaddress(self.nodes[1].getnewaddress(), 1) | ||
| + [ block1, block2 ] = self.nodes[2].generate(2) |
jnewbery
May 15, 2017
Member
nit: no need for brackets here. The following should do:
block1, block2 = self.nodes[2].generate(2)
| + [ block1, block2 ] = self.nodes[2].generate(2) | ||
| + self.sync_all() | ||
| + # We should be able to get the raw transaction by providing the correct block | ||
| + assert self.nodes[0].getrawtransaction(tx, True, block1) |
jnewbery
May 15, 2017
Member
nit: I think it's better to assert on the actual value here (ie verify that the getrawtransaction returned the correct transaction rather than returned anything). The following should do that:
assert_equal(self.nodes[0].getrawtransaction(tx, True, block1)['txid'], tx)
|
Is this still needed after #8704? |
|
@kallewoof #8704 does not require indexing either |
|
Yeah, sorry, I meant that this is a way to get a specific transaction if you know the block hash, whereas #8704 shows you all transactions in the entire block. You get the info, but you have to wade through stuff to get it. |
|
In both cases the whole block has to be loaded from disk, and parsed, and searched linearly. The difference is whether the linear search step happens on the server or client. I think both #8704 and this can be useful, but have to agree there's only superficial difference. |
|
The difference between the two is the serialization and transmission and parsing of ~5MB of JSON data vs a few hundred bytes of hex encoded data. That's a 1,000x difference on the client side, and the same absolute improvement on the server -- although as a multiplier it'd be less since as you note the server still has to parse the block from disk. That's a nontrivial performance difference. (Also, this would have greatly helped me in the past so another +1 from me.) |
|
Agree with @maaku - simply encoding the data into 5MB of json could be time-consuming, during which time the cs_main lock is held. Having a command to return just a single transaction from a block seems very useful. |
jnewbery
referenced this pull request
May 23, 2017
Closed
RPC: getblock returns coinbase scriptsig #10385
| "\nNOTE: By default this function only works for mempool transactions. If the -txindex option is\n" | ||
| - "enabled, it also works for blockchain transactions.\n" | ||
| + "enabled, it also works for blockchain transactions. If the block hash is known, it can be provided\n" | ||
| + "for nodes without -txindex.\n" |
| "\nResult (if verbose is not set or set to false):\n" | ||
| "\"data\" (string) The serialized, hex-encoded data for 'txid'\n" | ||
| "\nResult (if verbose is set to true):\n" | ||
| "{\n" | ||
| + " \"inMainChain\": b, (bool) Whether transaction is in the main chain or not. Only visible when specifying block hash\n" |
luke-jr
Jun 3, 2017
Member
Whether the block specified is in the main chain or not... This could be false with the tx being still in the main chain!
| + if (!blockHash.IsNull()) { | ||
| + BlockMap::iterator it = mapBlockIndex.find(blockHash); | ||
| + if (it == mapBlockIndex.end()) { | ||
| + throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "Block hash not found in chain"); |
added a commit
to luke-jr/bitcoin
that referenced
this pull request
Jun 3, 2017
added a commit
to luke-jr/bitcoin
that referenced
this pull request
Jun 3, 2017
added a commit
to luke-jr/bitcoin
that referenced
this pull request
Jun 3, 2017
added a commit
to luke-jr/bitcoin
that referenced
this pull request
Jun 3, 2017
added a commit
to luke-jr/bitcoin
that referenced
this pull request
Jun 3, 2017
|
Suggested message fix on my |
|
Needs rebase. |
|
@luke-jr Thanks for the review! I cherry-picked your commit.
@sipa Rebased. |
TheBlueMatt
reviewed
Jun 7, 2017
You may wish to squash "[rpc] Fix fVerbose parsing (remove excess if cases and ensure null is…" and "[test] Updated rawtransactions.py to assert for adjusted exception.".
Generally we try to make sure that after each individual commit, at least it builds and all tests pass.
| "\nNOTE: By default this function only works for mempool transactions. If the -txindex option is\n" | ||
| - "enabled, it also works for blockchain transactions.\n" | ||
| + "enabled, it also works for blockchain transactions. If the block hash is known, it can be provided\n" | ||
| + "for nodes without -txindex, in which case the transaction will only be found if it is in that\n" |
TheBlueMatt
Jun 7, 2017
Contributor
grammar nit: this reads funny to me, and could be a bit more explicit. Maybe:
"If the block which contains the transaction is known, its hash can be provided even for nodes without -txindex."
"Note that if a blockhash is provided, only it will be searched and if the transaction is in mempool, other blocks, or if this node does not have the given block available, the transaction will not be found."
kallewoof
Jun 8, 2017
Member
Thanks, that looks better yeah. Adding with minor tweaks.
"\nNOTE: By default this function only works for mempool transactions. If the -txindex option is\n"
"enabled, it also works for blockchain transactions. If the block which contains the transaction\n"
"is known, its hash can be provided even for nodes without -txindex. Note that if a blockhash is\n"
"provided, only that block will be searched and if the transaction is in the mempool or other\n"
"blocks, or if this node does not have the given block available, the transaction will not be found.\n"| "\nResult (if verbose is not set or set to false):\n" | ||
| "\"data\" (string) The serialized, hex-encoded data for 'txid'\n" | ||
| "\nResult (if verbose is set to true):\n" | ||
| "{\n" | ||
| + " \"inMainChain\": b, (bool) Whether specified block is in the main chain or not\n" |
TheBlueMatt
Jun 7, 2017
Contributor
Hmm, maybe say "if blockhash is specified" or otherwise mention this wont appear unless a blockhash is provided. Even better, fill it out if GetTransaction returns the blockhash cause it found it via UTXO/txindex.
kallewoof
Jun 8, 2017
Member
I was sure I did, but guess not:
" \"inMainChain\": b, (bool) Whether specified block is in the main chain or not (only present with explicit \"blockhash\" argument)\n"I like the idea of including when able but will keep it out of this PR for now.
| + } | ||
| + | ||
| + if (request.params.size() > 2 && !request.params[2].isNull()) { | ||
| + uint256 blockHash = ParseHashV(request.params[2], "parameter 3"); |
TheBlueMatt
Jun 7, 2017
Contributor
Hmm? Shouldn't we use the parameter's name here instead of "parameter 3"?
kallewoof
Jun 8, 2017
Member
The general tendency seems to be to identify the parameter index so I stuck with that. I agree it may be better to be more descriptive though...
| - if (!GetTransaction(hash, tx, Params().GetConsensus(), hashBlock, true)) | ||
| - throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, std::string(fTxIndex ? "No such mempool or blockchain transaction" | ||
| + if (!GetTransaction(hash, tx, Params().GetConsensus(), hashBlock, true, blockIndex)) | ||
| + throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, std::string(fTxIndex || blockIndex ? "No such mempool or blockchain transaction" |
TheBlueMatt
Jun 7, 2017
Contributor
Maybe further update this error message, eg if (blockIndex) "No such transaction found in the provided block".
kallewoof
Jun 8, 2017
•
Member
Yeah, I wanted to avoid ?:?:. Rewritten as:
if (!GetTransaction(hash, tx, Params().GetConsensus(), hashBlock, true, blockIndex)) {
std::string errmsg;
if (blockIndex) {
errmsg = "No such transaction found in the provided block";
} else {
errmsg = fTxIndex
? "No such mempool or blockchain transaction"
: "No such mempool transaction. Use -txindex to enable blockchain transaction queries";
}
throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, errmsg + ". Use gettransaction for wallet transactions.");
}|
@TheBlueMatt Thanks for the review!
We try to keep tests as separate commits though, so that would assume tests and code changes come in pairs (tests will fail before test commit or after test commit and before change commit, obv). That was my intention with the split here. I may have screwed up. I'll double check and/or squash as appropriate. Edit: I noticed the order was off (two fixes then two tests). Rearranged them. The commit/test pairs now pass make check individually (i.e. a43ec61 and 8f2ce52). Edit 2: [...]:
|
added a commit
to bitcoinknots/bitcoin
that referenced
this pull request
Jun 15, 2017
added a commit
to bitcoinknots/bitcoin
that referenced
this pull request
Jun 15, 2017
added a commit
to bitcoinknots/bitcoin
that referenced
this pull request
Jun 15, 2017
added a commit
to bitcoinknots/bitcoin
that referenced
this pull request
Jun 15, 2017
|
utACK 8f2ce52 |
|
Needs rebase. |
|
Rebased. |
|
a) Is there a reason why mainchain height is not supported as alternative for the blockhash? Eventually with a security of only accepting heights of a hundred blocks below the tip as a reorganisation protection (but I'd prefer to not add this protection). b) @kallewoof the idea about the standard for a transaction reference has already been worked into a BIP: bitcoin/bips#555 (maybe we can support this – if we agree on that BIP to be worth implementing – also via |
|
@jonasschnelli Regarding height, I chose not to include it as it could potentially cause unexpected results when a reorg happens, but if people don't think that's an issue it should be fairly straightforward to allow for both. Edit: as for the standard, that looks exciting for sure. If it matures enough and this PR isn't merged already I may take a stab at it. |
| "\nResult (if verbose is not set or set to false):\n" | ||
| "\"data\" (string) The serialized, hex-encoded data for 'txid'\n" | ||
| "\nResult (if verbose is set to true):\n" | ||
| "{\n" | ||
| + " \"inMainChain\": b, (bool) Whether specified block is in the main chain or not (only present with explicit \"blockhash\" argument)\n" |
jtimon
Jul 17, 2017
•
Member
This may be confused with Params().NetworkIDString() == "main" (kind of like we do with "testnet" in getinfo).
Can we rename to "inActiveChain" or something of the sort?
| - txOut = ptx; | ||
| - return true; | ||
| - } | ||
| + if (!blockIndex) { |
jtimon
Jul 17, 2017
Member
Perhaps the diff can be less disruptive by moving everything inside if (!blockIndex) {...} to a new static function defined right above instead of indenting all of it?
kallewoof
Jul 18, 2017
Member
I'm not a huge fan of changing code just to make diffs smaller, and as I mention above, you can put ?w=1 to get diff without indentation changes.
|
utACK 3ec2d28 besides small nits. |
|
Addressed @jtimon nits. |
|
I'm not sure about allowing the height. |
jtimon
referenced this pull request
Jul 18, 2017
Closed
[pull request idea] addressindex, spentindex, timestampindex (Bitcore patches) #10370
|
@jtimon It feels like a convenience thing that, IMO, getblock should have as well (i.e. allow both height number and block hash). I don't have a strong opinion on the subject though, and will drop the last 2 commits (height support) whenever this is ready for merging unless someone argues for block height support. |
|
Well, don't have a strong opinion on adding the block height either, if more people like it, let's keep it. |
|
concept ACK, needs rebase/squash, would review |
|
@instagibbs Rebased & squashed; also removed block height option for now. Will push as separate PR if desirable. |
instagibbs
suggested changes
Aug 24, 2017
Current behavior of GetTransaction means that even if you specify a block index, it still does all the other searches. This means that a user could specify an incorrect blockhash, and still return a valid result, with "inActive" == False(though it may actually be in the valid chain!).
As a user I would expect any call to fail if I simply gave the wrong blockhash, but that's simply my opinion.
| "\nResult (if verbose is not set or set to false):\n" | ||
| "\"data\" (string) The serialized, hex-encoded data for 'txid'\n" | ||
| "\nResult (if verbose is set to true):\n" | ||
| "{\n" | ||
| + " \"inActiveChain\": b, (bool) Whether specified block is in the active chain or not (only present with explicit \"blockhash\" argument)\n" |
| ); | ||
| LOCK(cs_main); | ||
| + bool inActiveChain = true; |
instagibbs
Aug 24, 2017
Member
in_active_chain to conform to new style guide and make easier to read.
| - if (fAllowSlow) { // use coin database to locate block that contains transaction, and scan it | ||
| - const Coin& coin = AccessByTxid(*pcoinsTip, hash); | ||
| - if (!coin.IsSpent()) pindexSlow = chainActive[coin.nHeight]; | ||
| + if (fAllowSlow) { // use coin database to locate block that contains transaction, and scan it |
kallewoof
Aug 25, 2017
Member
This is inside the if (!blockIndex) block so it should never execute if a block hash was provided, no?
|
I fail at counting brackets. utACK 440123f |
jnewbery
referenced this pull request
Sep 1, 2017
Closed
[trivial] [rpc] Fix getrawtransaction help for per-txout chainstate db #11213
jnewbery
reviewed
Sep 1, 2017
A few nits inline.
I think you need to squash the first two commits to make the code and test changes atomic (otherwise the test fails on the intermediate 90e1408 commit)
| "\nResult (if verbose is not set or set to false):\n" | ||
| "\"data\" (string) The serialized, hex-encoded data for 'txid'\n" | ||
| "\nResult (if verbose is set to true):\n" | ||
| "{\n" | ||
| + " \"in_active_chain\": b, (bool) Whether specified block is in the active chain or not (only present with explicit \"blockhash\" argument)\n" |
kallewoof
Sep 4, 2017
Member
" \"in_active_chain\": b, (bool) Whether specified block is in the active chain or not (only present with explicit \"blockhash\" argument)\n"
" \"hex\" : \"data\", (string) The serialized, hex-encoded data for 'txid'\n"becomes
"in_active_chain": b, (bool) Whether specified block is in the active chain or not (only present with explicit "blockhash" argument)
"hex" : "data", (string) The serialized, hex-encoded data for 'txid'
(two \ vs four \ means indentation diff in code)
jnewbery
Sep 4, 2017
Member
Yes, of course you're correct. The size, vsize, version, ... fields below aren't aligned by the same logic, but that's beyond the scope of this PR. Please disregard my previous comment.
| + if (!GetTransaction(hash, tx, Params().GetConsensus(), hash_block, true, blockindex)) { | ||
| + std::string errmsg; | ||
| + if (blockindex) { | ||
| + errmsg = "No such transaction found in the provided block"; |
jnewbery
Sep 1, 2017
Member
I think we would also hit this error if the block had been pruned. In that case, this error is misleading.
Is there any way to test above for whether the block has been pruned and error out before doing the transaction lookup?
kallewoof
Sep 4, 2017
Member
Good point!
I added code (3c0cb43) that checks if pruning is enabled, and if it is, checks if the block is actually available by reading it from disk and setting the error message based on the results of that.
| + ############################ | ||
| + | ||
| + # make a tx by sending then generate 2 blocks; block1 has the tx in it, | ||
| + # presumably |
| + # We should be able to get the raw transaction by providing the correct block | ||
| + assert_equal(self.nodes[0].getrawtransaction(tx, True, block1)['txid'], tx) | ||
| + # We should not get the tx if we provide an unrelated block | ||
| + assert_raises_jsonrpc(-5, "No such", self.nodes[0].getrawtransaction, tx, True, block2) |
jnewbery
Sep 1, 2017
Member
You need to make the matching error text longer to be non-ambiguous. This test will still pass if the node returns the error No such mempool or blockchain transaction (which would be incorrect behaviour)
|
@jnewbery Thanks for the review! I think I've addressed all your comments. Edit: Actually I forgot about the commit merge. I try to keep tests and code changes as separate commits. Why would you want it to succeed between the test update? (Or am I confused about what you're asking?) |
I think the principal is that each individual commit should be at least internally consistent (should build and pass its own tests). Your first commit updates the product code so the returned error changes, and your second commit updates the test code to pass with the new error string. That means that if I run the tests after the first commit they'll fail. I understand the desire to structure the commits in a way that tells a story, but this breaks tools like git bisect that rely on commits to be internally consistent. |
|
@jnewbery You can |
|
I agree with @jnewbery, one commit should not break the code, tests including. If you revert that commit, codes stays green. IMO makes reviewing easier. |
| + std::string errmsg; | ||
| + if (blockindex) { | ||
| + bool block_available = true; | ||
| + if (gArgs.GetArg("-prune", 0)) { |
| + if (gArgs.GetArg("-prune", 0)) { | ||
| + // we are pruning; see if block is available at all | ||
| + CBlock block; | ||
| + block_available = ReadBlockFromDisk(block, blockindex, Params().GetConsensus()); |
luke-jr
Sep 5, 2017
Member
Use !(pblockindex->nStatus & BLOCK_HAVE_DATA) && pblockindex->nTx > 0 (see getblock)
|
I agree individual commits should never break the tests ideally. That doesn't mean that you can't introduce new functionality and its new tests in separated commits when that's not the case. |
|
Last two commits look good. For the pruning error, I think you can error out earlier by re-using the pruned test from
ok. I think both ways are valid, and I've come across both before. I'm not sure if this project has a preferred style. |
|
I'm merging those commits as it's not new functionality but just a fix that changes an error message. |
|
@luke-jr Nice! |
| + if (blockindex) { | ||
| + bool block_pruned = fHavePruned && !(blockindex->nStatus & BLOCK_HAVE_DATA) && blockindex->nTx > 0; | ||
| + errmsg = block_pruned | ||
| + ? "Block not available (pruned data)" |
jnewbery
Sep 6, 2017
Member
supernit: Not finding a block because it's been pruned is not a RPC_INVALID_ADDRESS_OR_KEY error. It would more accurately be a RPC_MISC_ERROR.
My suggestion would be to do the pruned check above, immediately after the throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "Block hash not found"); line
|
re-utACK 300a5f1 |
kallewoof commentedApr 25, 2017
•
Edited 1 time
-
kallewoof
Apr 26, 2017
[Reviewer hint: use ?w=1 to avoid seeing a bunch of indentation changes.]
Presuming a user knows the block hash of the block containing a given transaction, this PR allows them to fetch the raw transaction, even without
-txindex. It also enables support for getting transactions that are in orphaned blocks.Note that supplying a block hash will override mempool and txindex support in
GetTransaction. The rationale behind this is that a transaction may be in multiple places (orphaned blocks) and if the user supplies an explicit block hash it should be adhered to.