Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

show scriptSig signature hash types in transaction decodes. fixes #3166 #5264

Merged
merged 1 commit into from Sep 25, 2015

Conversation

@mruddy
Copy link
Contributor

@mruddy mruddy commented Nov 12, 2014

show scriptSig signature hash types. fixes #3166

Please give this a look and let me know if you'd like it changed or if I mis-understood the issue request.

The fix basically appends the scriptSig signature hash types, within parentheses, onto the end of the signature(s) in the various "asm" json outputs. That's just the first formatting idea that came to my mind.

Added some tests for this too.

@petertodd
Copy link
Contributor

@petertodd petertodd commented Nov 12, 2014

NACK

This will definitely break things, and I don't see a need given how easy the SIGHASH flags are to remember. All the standard ones can be interpreted by thinking in terms of upper and lower nibbles:

Upper nibble == 0:

01 - ALL
02 - NONE
03 - SINGLE

Upper nibble == 8:

81 - ANYONECANPAY | ALL
82 - ANYONECANPAY | NONE
83 - ANYONECANPAY | SINGLE

That's it.

@laanwj
Copy link
Member

@laanwj laanwj commented Nov 12, 2014

@petertodd That's not really fair. Why was this an open issue, if this is not desirable?

Where is the risk of breakage? It doesn't affect consensus code. Do people rely on the exact format of the dumped script format?

@laanwj laanwj added the RPC/REST/ZMQ label Nov 12, 2014
@petertodd
Copy link
Contributor

@petertodd petertodd commented Nov 12, 2014

@laanwj They sure do! Granted, maybe we don't want that, in which case we should delibrately break it. (make a note in the release notes please)

Why was this an open issue, if this is not desirable?

Well, I'm telling people why I think it's not desirable. If I'm outnumbered on this, then I'll at least ask if we could drop the 'SIGHASH_' prefix on this; having the full string is kinda long if it's not meant to be parsed by anyone other than humans.

@laanwj
laanwj reviewed Nov 12, 2014
View changes
src/script/script.cpp Outdated
if (vch.size() <= 4)
return strprintf("%d", CScriptNum(vch, false).getint());
else
return HexStr(vch);
return HexStr(vch) + (CheckSignatureEncoding(vch, SCRIPT_VERIFY_STRICTENC) ? ("(" + mapSigHashTypes[vch.back()] + ")") : "");

This comment has been minimized.

@laanwj

laanwj Nov 12, 2014
Member

Any special reason to use SCRIPT_VERIFY_STRICTENC instead of SCRIPT_VERIFY_DERSIG here?

This comment has been minimized.

@mruddy

mruddy Nov 12, 2014
Author Contributor

the reason why i went with SCRIPT_VERIFY_STRICTENC was because in CheckSignatureEncoding, SCRIPT_VERIFY_STRICTENC will get us some checking through IsDefinedHashtypeSignature as well in order to make sure that the hash type is one of the defined types. if it's not a defined hash type, then no decode gets appended to the hex encoding of the signature.

@laanwj
Copy link
Member

@laanwj laanwj commented Nov 12, 2014

@petertodd Yes, let's see what the others think here. I generally like it when people solve an actual open issue :)

Agree on dropping the SIGHASH_ prefix and adding mention to doc/release-notes.md.

@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Nov 12, 2014

@petertodd @laanwj thanks for the feedback.

I have removed the "SIGHASH_" from the text due to your feedback that it was too verbose. I was back and forth on that before I pushed it up, so I'm happy for your opinions.

I can see @petertodd's concern about this being a breaking change for people scripted against it. It's something to weigh. I don't know that adding more flags is the answer. From working on another pull request, my understanding is that the JSON is equivalent to "verbose".

FWIW, I kind of like the extra decoding output myself because it's one less thing I have to think about. It's a simple decode like @petertodd said, but having it just there in plain sight means I don't have to remember, or go look it up.

I will add some release notes comments in doc/release-notes.md in a little bit after waiting to hear what other feedback is.

@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Nov 12, 2014

Added some release notes.

@petertodd
Copy link
Contributor

@petertodd petertodd commented Nov 12, 2014

@mruddy Mind squashing those commits?

@mruddy mruddy force-pushed the mruddy:iss3166 branch Nov 12, 2014
@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Nov 12, 2014

sure thing, done

@sipa
Copy link
Member

@sipa sipa commented Nov 12, 2014

I don't think we should be adding this to the existing 'asm' output, but perhaps an alternate decoding that goes into lower level detail.

@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Nov 12, 2014

ok, thanks. if nobody pops up and says that they want this within a day or so, i'll go ahead and close this. it probably fits better somewhere else.

@laanwj
Copy link
Member

@laanwj laanwj commented Nov 13, 2014

@sipa What if the 'asm' format grew a way to specify comments?

@laanwj
Copy link
Member

@laanwj laanwj commented Nov 13, 2014

@gmaxwell can you comment here? You created #3166, so maybe you can illuminate how it should work

@gmaxwell
Copy link
Contributor

@gmaxwell gmaxwell commented Nov 13, 2014

Hm. I don't have a strong opinion about how it would work. I would have normally assumed some annotation on the asm output, or an additional lower level asm view.

My goal was mostly that it would be clearly indicated in some manner stronger than squnting at bytes. In particular, squinting fails when you only barely know there is something you need to squint at. An expliact flag list would be much more obvious and discoverable.

WRT compatiblity of the ASM output, we've certantly never promised to hold that encoding constant... it's fine if things are reading it, but they'll need to be revised if it changes.

@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Nov 18, 2014

Closing because there doesn't seem to be much demand for this fix with the possibility that it breaks things scripted against the "asm" output value.

@mruddy mruddy closed this Nov 18, 2014
@petertodd
Copy link
Contributor

@petertodd petertodd commented Nov 18, 2014

@mruddy Actually, I was just asking around, and it looks like people are getting the message and not depending on the asm output format as much as before. As an example Counterparty switched to using python-bitcoinlib for that on my advice.

Maybe we should reopen this for v0.11 and simultaneously drop the 'OP_' prefix from the opcode names in the asm output? That'd likely break the remaining stuff pretty thoroughly in a clear way.

@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Nov 19, 2014

@petertodd sure, re-opened. I have not gone through and made the "OP_" prefix changes yet. I figure that I'll have to do a bunch of reference checking to see what all I'm impacting, and then update or make some unit tests when I do that. So, more changes to come for that.

@mruddy mruddy reopened this Nov 19, 2014
@petertodd
Copy link
Contributor

@petertodd petertodd commented Nov 20, 2014

@mruddy Great, thanks!

FWIW a "OP_" prefix dropping change should definitely be in a separate commit so it can be debated separately. Also, if changing stuff like that doesn't break any tests, keep in mind it's a sign that maybe depending on the exact format of the asm output is a bad idea. :)

(yeah, the more I think about this, the more I think my original objection was wrong...)

@mruddy mruddy force-pushed the mruddy:iss3166 branch 3 times, most recently Nov 28, 2014
@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Nov 28, 2014

@petertodd sorry for my delayed response... i just made #5392 for the "OP_" prefix changes. also, i rebased this request's commit so that it'll merge into the upstream once again.

@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Dec 11, 2014

Eh, not important enough to keep open and keep re-basing the commit.

@mruddy mruddy closed this Dec 11, 2014
@laanwj
Copy link
Member

@laanwj laanwj commented Dec 11, 2014

I still think this makes sense, but after 0.10 obviously.
If this is closed unfixed with "don't bother" then so should #3166.

@mruddy mruddy force-pushed the mruddy:iss3166 branch Jul 28, 2015
@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Jul 28, 2015

@jtimon that's a lot of useful input, thanks! I took that and actually refactored these changes twice.

I put each of the two ways into the last two commits.

The first commit refactored into a couple of formatter classes. It worked, but the second/next idea seemed a little cleaner.

The second commit does away with the formatter class idea of the previous commit and just moves CScript::ToString out into a function in core_write.cpp.

I think this last commit converges your input with sipa's input. I think the last one loops in everyone's input, doesn't layer violate, and reduces code from the consensus library.

I purposefully didn't choose to sighhash decode some scriptSigs that were just going to be logged since those usages were tangential to what i was trying to actually update for this ticket.

Let me know what you think of the latest. Thanks!

@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Jul 28, 2015

... or not. I see the travis windows builds failed because primitives/transaction.cpp is still part of libconsensus and references the ScriptToAsmStr for ToString'ing CTxIn and CTxOut objects. Guess I'll have to move those around too.

@jtimon
Copy link
Member

@jtimon jtimon commented Jul 28, 2015

You are welcomed. I'm fine with putting new things in core_io as @sipa suggested, just not in libconsensus. CTransaction::ToString now calls ScriptToAsmStr so instead of putting script/standard in libconsensus, now you're putting core_io instead. The important thing is this: independently of where you chose to put the new code outside of libconsensus, you cannot include that new code from libconsensus. That's why I keep posting a link to the list of files that are part of libconsensus: https://github.com/bitcoin/bitcoin/blob/master/src/Makefile.am#L365
If you include more things in any of those files, travis will complain (unless you adapt the makefile properly to include the new file, in which case we will complain).

So, either don't adapt CTransaction::ToScript to the new code (seems the simpler solution) or also take CTransaction::ToString out of consensus (seems better but also more work).

@jtimon
Copy link
Member

@jtimon jtimon commented Jul 28, 2015

Another possibility is to leave the new code in libconsensus like you had at the beginning (not my preference), but it still couldn't be in script/script since you need script/interpreter and that would cause a circular dependency. So even leaving the new code in libconsensus, you always need to rename CScript::ToString and move it our of script/script.

@mruddy mruddy force-pushed the mruddy:iss3166 branch Jul 28, 2015
@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Jul 28, 2015

It turns out that resolving that last dependency was pretty easy.
I simply stopped using that script assembly formatting in the CTxIn and CTxOut ToString methods.
Those ToString's were only called by very low value debug and log print statements and the formatted scripts were being truncated immediately too. So, converting to hex instead in those cases isn't losing much, if anything at all, in my opinion. Doing it this way allowed the removal of the original CScript::ToString from libconsensus.

@jtimon
jtimon reviewed Jul 29, 2015
View changes
src/primitives/transaction.cpp Outdated
@@ -5,6 +5,7 @@

#include "primitives/transaction.h"

#include "core_io.h"

This comment has been minimized.

@jtimon

jtimon Jul 29, 2015
Member

This is not necessary anymore, right?

@jtimon
jtimon reviewed Jul 29, 2015
View changes
src/bitcoin-tx.cpp Outdated
@@ -12,6 +12,7 @@
#include "primitives/transaction.h"
#include "script/script.h"
#include "script/sign.h"
#include "script/standard.h"

This comment has been minimized.

@jtimon

jtimon Jul 29, 2015
Member

This is core_io now.

@jtimon
jtimon reviewed Jul 29, 2015
View changes
src/test/script_tests.cpp Outdated
@@ -9,6 +9,7 @@
#include "key.h"
#include "keystore.h"
#include "script/script.h"
#include "script/standard.h"

This comment has been minimized.

@jtimon

jtimon Jul 29, 2015
Member

This is core_io now.

@jtimon
Copy link
Member

@jtimon jtimon commented Jul 29, 2015

This looks like a reasonable solution.
Apart from the latest little nits, consensus-safe-review ACK (I also slightly reviewed the new functions, but didn't look much at the tests).

@mruddy mruddy force-pushed the mruddy:iss3166 branch Jul 29, 2015
@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Jul 29, 2015

@jtimon You are correct about those latest nits. I made the updates and squashed+rebased everything back down to the one latest commit. I also updated the release notes description at the same time. Should be all good now. Thanks for the help, it was good working with you on this!

@jtimon
Copy link
Member

@jtimon jtimon commented Jul 29, 2015

Thanks to you for your patience when improving this. Everything looks good, re-utACK.

@sipa
sipa reviewed Jul 30, 2015
View changes
src/core_write.cpp Outdated
if (vch.size() <= static_cast<vector<unsigned char>::size_type>(4)) {
str += strprintf("%d", CScriptNum(vch, false).getint());
} else {
if (fAttemptSighashDecode) {

This comment has been minimized.

@sipa

sipa Jul 30, 2015
Member

Nit: the !script.IsUnspendable() check can move here, I believe.

@sipa
Copy link
Member

@sipa sipa commented Jul 30, 2015

Untested ACK, but the tests look convincing.

These changes decode valid SIGHASH types on signatures in assembly (asm) representations of scriptSig scripts.
This squashed commit incorporates substantial helpful feedback from jtimon, laanwj, and sipa.
@mruddy mruddy force-pushed the mruddy:iss3166 branch to af3208b Jul 31, 2015
@mruddy
Copy link
Contributor Author

@mruddy mruddy commented Jul 31, 2015

@sipa nit addressed. I also added a new test case just to cover a specific low probability case related to the nit to make sure it was covered.

@jgarzik
Copy link
Contributor

@jgarzik jgarzik commented Sep 15, 2015

ut ACK - looks ready to merge

@laanwj laanwj merged commit af3208b into bitcoin:master Sep 25, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
laanwj added a commit that referenced this pull request Sep 25, 2015
af3208b Resolve issue 3166. These changes decode valid SIGHASH types on signatures in assembly (asm) representations of scriptSig scripts. This squashed commit incorporates substantial helpful feedback from jtimon, laanwj, and sipa. (mruddy)
@mruddy mruddy deleted the mruddy:iss3166 branch Sep 29, 2015
zkbot added a commit to zcash/zcash that referenced this pull request Apr 17, 2018
Upstream serialization improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#5264
- bitcoin/bitcoin#6914
- bitcoin/bitcoin#6215
- bitcoin/bitcoin#8068
  - Only the `COMPACTSIZE` wrapper commit
- bitcoin/bitcoin#8658
- bitcoin/bitcoin#8708
  - Only the serializer variadics commit
- bitcoin/bitcoin#9039
- bitcoin/bitcoin#9125
  - Only the first two commits (the last two block on other upstream PRs)

Part of #2074.
@@ -36,7 +36,7 @@ std::string CTxIn::ToString() const
if (prevout.IsNull())
str += strprintf(", coinbase %s", HexStr(scriptSig));
else
str += strprintf(", scriptSig=%s", scriptSig.ToString().substr(0,24));
str += strprintf(", scriptSig=%s", HexStr(scriptSig).substr(0, 24));

This comment has been minimized.

@arielgabizon

arielgabizon Apr 18, 2018

was there a reason HexStr wasn't used here before?

This comment has been minimized.

@mruddy

mruddy Apr 18, 2018
Author Contributor

It was just more verbose (someone's personal preference, I guess), see what was removed CScript::ToString: af3208b#diff-f7ca24fb80ddba0f291cb66344ca6fcb

zkbot added a commit to zcash/zcash that referenced this pull request Apr 18, 2018
Upstream serialization improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#5264
- bitcoin/bitcoin#6914
- bitcoin/bitcoin#6215
- bitcoin/bitcoin#8068
  - Only the `COMPACTSIZE` wrapper commit
- bitcoin/bitcoin#8658
- bitcoin/bitcoin#8708
  - Only the serializer variadics commit
- bitcoin/bitcoin#9039
- bitcoin/bitcoin#9125
  - Only the first two commits (the last two block on other upstream PRs)

Part of #2074.
zkbot added a commit to zcash/zcash that referenced this pull request Apr 19, 2018
Upstream serialization improvements

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#5264
- bitcoin/bitcoin#6914
- bitcoin/bitcoin#6215
- bitcoin/bitcoin#8068
  - Only the `COMPACTSIZE` wrapper commit
- bitcoin/bitcoin#8658
- bitcoin/bitcoin#8708
  - Only the serializer variadics commit
- bitcoin/bitcoin#9039
- bitcoin/bitcoin#9125
  - Only the first two commits (the last two block on other upstream PRs)

Part of #2074.
furszy added a commit to PIVX-Project/PIVX that referenced this pull request May 29, 2020
249cc9d Avoid -Wshadow errors (random-zebra)
8e1ec9e Use fixed preallocation instead of costly GetSerializeSize (random-zebra)
9b801d0 Add optimized CSizeComputer serializers (random-zebra)
0035a54 Make CSerAction's ForRead() constexpr (random-zebra)
9730a3f Get rid of nType and nVersion (random-zebra)
25ce2bb Make GetSerializeSize a wrapper on top of CSizeComputer (random-zebra)
1b479db Make nType and nVersion private and sometimes const (random-zebra)
35f1755 Make streams' read and write return void (random-zebra)
a395914 Remove unused ReadVersion and WriteVersion (random-zebra)
52e614c [WIP] Remove unused statement in serialization (random-zebra)
82a2021 Add COMPACTSIZE wrapper similar to VARINT for serialization (random-zebra)
13ad779 add bip32 pubkey serialization (random-zebra)
9e9b7b5 [QA] Update json files with sig hash type in ASM for bitcoin-util-test (random-zebra)
3383983 Resolve issue bitcoin#3166 (random-zebra)

Pull request description:

  -Based on top of
  - [x] #1629

  Backports the following serialization improvements from upstream and adds the required changes for the 2nd layer network and the legacy zerocoin code.

  - bitcoin#5264
    > show scriptSig signature hash types. fixes bitcoin#3166
    >
    > The fix basically appends the scriptSig signature hash types, within parentheses, onto the end of the signature(s) in the various "asm" json outputs. That's just the first formatting idea that came to my mind.
    >
    > Added some tests for this too.

  - bitcoin#6215
    > CExtPubKey should be serializable like CPubKey.
    > This would allow storing extended private and public key to support BIP32/HD wallets.

  - bitcoin#8068 (only commit 5249dac)
     This adds COMPACTSIZE wrapper similar to VARINT for serialization

  - bitcoin#8658
    > As the line
    > ```
    > nVersion = this->nVersion;
    > ```
    > seems to have no meaning in READ and also in WRITE serialization op, let's remove it and see what our tests/travis will tell us. See bitcoin#8468 for previous discussion.

  - bitcoin#9039
    > The commits in this pull request implement a sequence of changes:
    >
    > - Simplifications:
    >   - **Remove unused ReadVersion and WriteVersion** CDataStream and CAutoFile had a ReadVersion and WriteVersion method that was never used. Remove them.
    >   - **Make nType and nVersion private and sometimes const** Make the various stream implementations' nType and nVersion private and const (except in CDataStream where we really need a setter).
    >   - **Make streams' read and write return void** The stream implementations have two layers (the upper one with operator<< and operator>>, and a lower one with read and write). The lower layer's return values are never used (nor should they, as they should only be used from the higher layer), so make them void.
    >   - **Make GetSerializeSize a wrapper on top of CSizeComputer** Given that in default GetSerializeSize implementations we're already using CSizeComputer(), get rid of the specialized GetSerializeSize methods everywhere, and just use CSizeComputer. This removes a lot of code which isn't actually used anywhere. In a few places, this removes an actually more efficient size computing algorithm, which we'll bring back in the "Add optimized CSizeComputer serializers" commit later.
    >   - **Get rid of nType and nVersion** The big change: remove the nType and nVersion as parameters to all serialization methods and functions. There is only one place where it's read and has an impact (in CAddress), and even there it does not impact any of the member objects' serializations. Instead, the few places that need nType or nVersion read it directly from the stream, through GetType and GetVersion calls which are added to all streams.
    >   - **Avoid -Wshadow errors** As suggested by @paveljanik, remove the few remaining cases of variable shadowing in the serialization code.
    > - Optimizations:
    >   - **Make CSerAction's ForRead() constexpr** The CSerAction's ForRead() method does not depend on any runtime data, so guarantee that requests to it can be optimized out by making it constexpr (suggested by @theuni in bitcoin#8580).
    >   - **Add optimized CSizeComputer serializers** To get the advantages of faster GetSerializeSize implementations back, reintroduce them in the few places where they actually make a difference, in the form of a specialized Serialize implementation. This actually gets us in a better state than before, as these even get used when they're nested inside the serialization of another object.
    >   - **Use fixed preallocation instead of costly GetSerializeSize** dbwrapper uses GetSerializeSize to compute the size of the buffer to preallocate. For some cases (specifically: CCoins) this requires a costly compression call. Avoid this by just using fixed size preallocations instead.
    >
    > This will make it easier to address @TheBlueMatt's comments in bitcoin#8580, resulting is a simpler and more efficient way to simultaneously deserialize+construct objects with const members from streams.

ACKs for top commit:
  furszy:
    Long and nice PR 👌 , code review ACK 249cc9d .
  Fuzzbawls:
    ACK 249cc9d
  furszy:
    tested ACK 249cc9d and merging.

Tree-SHA512: 56b07634b1e18871e7c9a99d412282c83b85f77f1672ec56330a1131fc7c234cd1ba3a053bdd210cc29f1e636ee374477ff614fa9a930329a7f8f912c5006232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

9 participants