Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Add normalized transaction hash #3656

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
Owner

sipa commented Feb 12, 2014

This is just a very basic implementation for exposure/review, with lacking documentation, and no functionality to look up transactions by normalized txid.

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/e7853a91cf646a6a4701158d148f036924575a97 for binaries and test log.
This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/
Contact BlueMatt on freenode if something looks broken.

Member

gmaxwell commented Feb 12, 2014

I note that this patch does what MtGox appeared to be asking for in their press release. In the >48 hours since its creation no one from MtGox has commented on it.

If people think a normalized ID would be useful, then great. Personally I'm pretty meh about it, as it doesn't address most of the interesting consequences of malleability, and some people might think they don't have to worry about malleability once they're using this in some place or another.

jMyles commented Feb 12, 2014

I agree that it's telling that MtGox hasn't weighed in, here specifically or with a solution proposal in general.

I generally lean toward wontfix on malleability, so I'm a little biased, but I agree that this solution may provide a false anaesthesis.

Owner

laanwj commented Feb 12, 2014

Maybe we should rename it to 'mtgox helpdesk id' :-)

I genuinely think that a normalized transaction hash standard can be useful. Not only for helpdesks but also for associating information to a transaction outside the block chain. Wallet metadata for example, we store this mapped to txid right now, which we explicitly recommend against using!

Member

gmaxwell commented Feb 12, 2014

@laanwj I don't think we've thought through the implications of all the cases where different transactions would get the same ID or wouldn't but would be expected to... different applications may need different levels of normalization.

E.g. consider a pubkey requiring signatures 1 of {A,B}. (or some more complicated scriptPubKey with an or like property). A transaction singing with one key may be very different from the user's perspective, and which of A or B signed may have substantial contractual or legal implications.

Or flipping the other way, consider transactions made ANYONE_CAN_PAY purposefully so that more inputs can be added to boost fees without resigning (but could be changed by other people too). These are still effectively the same transaction— pay the same people, achieve the same purpose, yet their normalized IDs would change. (I understand a large bitcoin company is planning on using these soon). "Oh no, malleability is back!"— if you wrote your wallet assuming the normalized IDs won't change out from under you and you start getting lots of payments from the aforementioned unmentioned company, you may be in for a surprise.

Just some food for though. It would be good to think through

I am unsure about ANYONE_CAN_PAY transactions as I haven't seen these yet.

Note that normalized transaction ids are meant to be used for transactions issued by the exchange, not received transactions.

Finally even if nobody from MtGox commented here before, this doesn't mean we have not been involved.

Contributor

wtogami commented Feb 12, 2014

I heard from another exchange that they had accounting problems for deposits. Apparently their implementation was confused between the 0-conf txid and the mutated 1-conf txid that came later. While that was an implementation issue for them that was easily fixed, this normalized txid would make it easier to see which tx are the same.

Owner

sipa commented Feb 12, 2014

@MagicalTux Absolutely, this is about transactions you send yourself, so if you're sure you stay within the confines of what this guarantees to catch as malleability, you're fine. Sighash flags just limit the usefulness of this normalized txid, not its correctness.

@laanwj Wallets internally have much better ways of dealing with this. The problem is wider - you probably want to deal with double spends the same way, which a normalized txid cannot catch. However, it's trivial to detect: whenever a transaction is found accepted that spends an input another transaction in your wallet already spends, it is either a mutated version of it, or a double spend thereof.

Contributor

coblee commented Feb 13, 2014

At Coinbase, we went with our own normalized tx id. I think having a standard recommended normalized tx id would be useful. Sites like Coinbase and MtGox can tell their users this tx id, and the users can check for the transaction on blockchain.info.

There's also no need to add an index for this txid. Only sites like blockchain.info will need to have an index for it.

Could also consider have transactions that are intentionally malleable (ANYONE_CAN_PAY) not have a normalized transaction id. So people won't accidentally use it thinking they are non-malleable.

Member

gmaxwell commented Feb 13, 2014

@coblee Sounds great.

If people want this to be something standardized, you're going to need to comment on it. If you are waiting on something to just happen here then you are going to be waiting a very long time.

Alternative implementations would be good— so we can make sure they agree (e.g. on fuzzed versions of all tx in the blockchain) and that there isn't corner-case behavior which is too difficult to correctly reproduce.

@gmaxwell Are you saying that we should just come up with our own standard and push it rather than waiting for the bitcoin core devs to confirm what standard would work best?
If not I believe most of the people here agree that having this "normalised id" (call it maybe a "helpdesk id" or whatever) being the SignatureHash makes most sense.

Contributor

coblee commented Feb 13, 2014

@gmaxwell Yeah, we were just waiting for a decision from the devs, figuring you guys will have a better idea on how to solve this problem.

@MagicalTux If the devs aren't going to move forward with this, we should just talk to piuk (blockchain.info) and other exchanges and come up with a standard.

I think having a standard normalized transaction id is important for the user experience when doing withdrawals from exchanges.

Owner

laanwj commented Feb 13, 2014

a) Restricting the scope to 'accounting id' or 'helpdesk id' is helpful here. Transactions that are never issued by an exchange for withdraws fall outside the scope here (like ANYONECANPAY).

b) What we need here is a BIP standard, not a 'bitcoind client feature'! So that means different implementations should be made, test vectors for cross-validating them, and so on.

c) 'The devs' (@sipa in this case) already gave their proposal and input. If you accept the proposal there is no need to wait. Documents can be written, implementations can be made, etc. If not - this needs further discussion and the best place would be the mailing list, not a github issue. I think @gmaxwell means that your input/vote is necessary not that you have to come up with your own standard.

Member

gmaxwell commented Feb 13, 2014

@MagicalTux We've proposed an approach,. In the three days that it has been posted you have not provided any substantive comment on it. Will this fit your needs? Are you able to implement it? have you attempted to implement it? If so can we test agreement between this implementation and yours? Do you have an alternative proposal?

I don't believe there is anything more that the core devs should be doing here or are planning on doing at this time. What exactly are you expecting from us?

@gmaxwell The implementation here fits what I discussed with @sipa and on IRC on the chatroom dedicated to this issue. It is still missing ways to get this id while sending coins (for example "normsendtoaddress" rpc method?) but apart from this, this seems to works exactly as we suggested.
I am unsure about a single point: bitcoin usually reverse bytes in hashes for txs and blocks, I do guess the same should happen for this. I believe we should keep this standard consistent but I am unsure if SignatureHash does that or not.

Member

luke-jr commented Feb 13, 2014

This proposal doesn't gracefully scale to newer protocols such as CoinJoin. Using a hash of scriptPubKey (if address reuse must be supported, the change scriptPubKey fits all current uses of txid) should be fairly compatible and also known only to the transacting parties (sender, in case of change output) and those whom they reveal it.

Member

luke-jr commented Feb 13, 2014

@coblee IMO any solution that proposes users depend on a centralised service is unacceptable. So indexes are needed.

Owner

laanwj commented Feb 13, 2014

@luke-jr That's beside the point if this is an id for exchanges to track their outgoing transactions.

@MagicalTux getting this id after sending would be a matter of calling getnormalizedtxid on the transaction created by the send. I do think we need a getrawwallettransaction RPC as there is currently no way to get raw transactions from the wallet.

Member

luke-jr commented Feb 13, 2014

@laanwj There's no reason to single out exchanges as a special case that gets to use a new feature. Furthermore, exchanges should not be forced to avoid (or even discouraged from) using improvements like CoinJoin.

Owner

laanwj commented Feb 13, 2014

@luke-jr The point is that this cannot be everything to everyone! As @gmaxwell explains in his second post it may not be possible to come up with a generic 'normalized txid' that satisfies everyone and every use case. So the focus has to be narrowed to something practical. They want an ID to track a specific network transaction.

If you want to be invariant under coinjoin and other input/output mangling you should use the destination address for tracking. That's a different use-case though.

@laanwj having to do two rpc calls can increase the risk of an error occurring, and may not be something exchanges want to do. Maybe alternatively we could use an argument to sendtoaddress to specify that the returned hash should be this new hash (or maybe both could be returned at the same time via another method that would provide more details on the generated tx such as fees/etc)

@luke-jr actually this works just fine with coinjoin, just that the hash issuance is delayed to the point at which the final tx is agreed to, and will be shared with someone else, as is the main transaction hash.

Member

gmaxwell commented Feb 13, 2014

@luke-jr we don't normally have an index for txid in the default configuration, even the wallets internal indexing of txids is pretty much just for internal use. E.g. no way to get to it in the GUI as far as I know.

@MagicalTux To get this ID when sending coins you call the additional RPC to look it up. Sadly, we can't use this as a general replacement for txids because some (currently unusual) transaction types just don't work as expected with it (they're still malleable, or you have application relevant different transactions which have the same ID)— these aren't issues with this proposal they're inherent in any proposal to produce a canonical ID by masking.

@gmaxwell While I do not think using this as a replacement (an api needs to stay consistent across updates), but having a different method or a parameter allowing to obtain this id directly will make the implementation of this solution that much easier for services using the bitcoind client.

Member

gmaxwell commented Feb 13, 2014

Unfortunately, as I pointed out, it is not viable to use the normalized ID exclusively to track transactions— not just for API stability reasons, but because the IDs are not very useful if you start making transactions with sighash flags other than ALL or do things which depend on which way a script with branches was satisfied, so I believe we're not going to do that at this time.

I wouldn't see a problem with handing out both— I think I suggested that above, though the only benefit would be saving an API round trip, and I think that sounds like an optimization which could wait for later. Or even wait just for someone saying they'd actually use it.

Owner

laanwj commented Feb 13, 2014

@MagicalTux we need to seperate two concerns here:

  1. Standarization of the 'helpdesk id'
  2. Implementation in the bitcoind client

Let's focus on 1 first. 2) can wait. Remember that @sipa explicitly mentions that he didn't flesh out the entire API yet and just wants to propose a way to compute the id so that we can at least agree on that.
As you use custom software (as I guess many exchanges do for their wallet) I can't suppose 2 isn't even that urgent.

@laanwj I agree, standardization is the most important point. Since we use our own client we can easily implement that on our side, and I believe it'll be the same for blockchain.info

Owner

sipa commented Feb 13, 2014

So, to clarify: I don't think this normalized transaction id as such is a required part of solving the malleability problem in wallets - those have better (potential) ways of dealing with it (which would equally help with double-spending in general).

However, it turns out that due to how many services work, I believe it is useful to have a standard for this at a higher level. As MtGox asked for this, I published my proposal for that on the mailing list, and now here as a pull request to simplify testing. Someone will have to come up with test vectors, compare implementations, check that it satisfies all requirements, and maybe write a BIP (sorry, I currently don't have time for this myself).

Also, I don't think the implementation as such (or it being accepted in the reference client codebase) is that important. The purpose was to set a standard. I don't think we can force one, especially as it is not something that affects the protocol or the reference client directly. If people want to see it deployed, ask that services adopt it.

Back to this implemnentation: I do believe that (IF the reference client adopts this standard, which can be independent from whether other services do), we need to be able to look up wallet transactions by it. The opposite would be encouraging a feature that only works with the presence of centralized block chain indexing services.

@sipa Sorry I'm new here but, can you maybe explain what the normalizing function should do?

@Corelianer It will create something similar to the TXID that will not be malleable. However it won't be useful for other things like creating transactions. Its simply an identifier which you can use to label each transaction.

@MagicalTux Is there any particular reason that you use a client written in PHP? While I can understand that it would make some aspects easier, couldn't you create a wallet that pushes relevant transactions to your sever that uses something that isn't PHP?

Owner

laanwj commented Feb 13, 2014

Guys, please, let's not let this devolve this into a generic Q&A forum for Mtgox or malleability issues. This is for technical discussion only.

@MagicalTux yep, this implementation suffices to validate alternate implementations.

@sipa: isn't where an existing rule to deal with TXID collisions in the blockchain? That 2 transactions in the blockchain can have the same TXID if and only if the first transaction is spent by the time the second transaction is added in a new block? (can't find the reference now)

How do you deal with normalized TXID collisions? Now, in the long run, you may end up with 2 unrelated transactions in the blockchain with different TXID (common) but with the same normalized TXID (very very improbable).

Or said in another way: very very improbable, but a new transaction can be issued and latter confirmed, which could have the same normalized TXID than an unrelated previous transaction in the blockchain (that previous transaction having a different TXID and either be spent or not). The normalized TXID gives the impression of a unique ID that seams misleading.

If this is not dealt properly then normalized TXID is king of a cheap fix and should only be used as an internal tool for individuals (or exchanges) at their own risk of getting confusing info (again very improbable, but then again, the blockchain deals with this).

@llongeri If you're worried about that, use the whole transaction as the unique ID. Honestly nothing else will prevent a possible collision. It is just too unlikely there'll ever be a collision to worry about it.

Identical NTXID will cause a transaction to seem confirmed when it isn't yet. If that should ever happen (it won't, I bet) it will likely not affect whatever process it is in, as it is merely to prevent malicious users. And there'll always be the actual TXID!

This fix might be cheap, but that doesn't mean it's not high quality.

@lodewijkadlp you are right, it is just that dealing with two IDs that are not unique either of them is just more confusion.
And yes, if ever a NTXID collision happens it most probably will go unnoticed and will only involve 2 transactions.
But I think this is just making a more confusing system, and it could be useful until there is a real solution to malleability is widely adopted. I would very much support eliminating NTXID once that happens.

Owner

sipa commented Feb 13, 2014

Two transactions with the same ntxid will necessarily have the same inputs, so conflict with eachother. This guarantees they cannot enter the same block chain together, since BIP30 and BIP34.

The name should avoid including "id", because that's going to cause problems down the road when people will make false assumptions about its properties.

I think "txhint" would result in fewer future problems.

@sipa BIP30 guarantees that 2 unspend transactions in the blockchain cannot have the same TXID. And BIP34 introduces uniqueness of coinbase transactions.

I understand there is no guaranty for unique TXID, and much less guaranty for NTXID uniqueness since is not even treated by miners or ever consider in the mining process.

But NTXID can be useful until a solution to malleability is widely spread. One thing though, I would not use the same format that it is used for TXID. This is going to confuse people, and that is why different identifications in bitcoin are recognizable by looking at them (addresses, TXIDs, blockhashes).

I would suggest at least add a prefix (like 'N') to make it noticeable and recognizable by any software.

Owner

sipa commented Feb 13, 2014

BIP34 guarantees that the same txid cannot reoccur for a new transaction, as that requires all its dependencies to be equal as well, up to and including all coinbases it inherits from.

But even independently from that, two transactions that spend the same inputs will always conflict with eachother. The only way it could occur is if the unspent outputs were recreated in the mean time, and that is not possible anymore due to BIP34.

Member

luke-jr commented Feb 13, 2014

@laanwj AFAIK hash of change scriptPubKey works in every use case?

@MagicalTux CoinJoin does not provide a final transaction form until it is mined in a block (or at least all parties have signed the transaction and it is broadcast). You also lose the ability to reissue with higher fees should it not get mined in a timeframe you are happy with.

@gmaxwell The wallet should have indexes by txid, or gettransaction would be very slow if it had to scan over everything in the wallet.. ;)

Owner

sipa commented on e7853a9 Feb 13, 2014

This is a not a serious proposal for inclusion into bitcoind.

This is a simple implementation of a proposed standard to verify compatibility.

@sipa, BIP34 guarantees that 2 coinbase transactions cannot have the same content (whole transaction content, by adding the block's height in its scriptSig). Thus stopping the trivial reappearance of the same TXID in the blockchain for coinbase transactions (and also for subsequent transactions in their spending chains).

But it is important to clearly state that TXIDs are not guarantied to be unique (I believe that there is no validation of this in the code, but maybe I am mistaken and would like to be clarified). Now with BIP34 it is highly improbable for 2 transactions to have the same TXID. No one can trivially duplicate TXIDs as it was possible before, but that does not guaranties uniqueness (And yes, there maybe not enough time in the universe to come up with a duplicated TXID now, I know).

Even if enforcement of unique TXID is introduced in the blockchain, it would not guarantee NTXID uniqueness, it would just continue to be very improbable and not trivial.

Things should be addressed as they are so no misconceptions get spread so this patch and other alternatives can be better compared/evaluated.

Again, if NTXID is introduced, I would suggest it should have its own format in order to differentiate it from other elements. A prefix could be used.

Owner

sipa commented Feb 13, 2014

I'm sorry, I don't have the time to explain this. But the network rules current guarantee that any two transactions in the block chain have a different txid. This is an indirect result of BIP34. By extension, as the ntxid depends on the txids of the outputs being spent, any two transactions in the blockchain will have a different ntxid (as otherwise they would be spending the same outputs twice).

Also, this is irrelevant, IMHO. Nobody should compute ntxid of blockchain transactions. They are only relevant to your own wallet, and in that sense, they are designed to collide.

Adding a prefix to distinguish it, is fine to me.

Owner

sipa commented Feb 13, 2014

Re-reading your comment, you're just referring to SHA256 collisions resulting in txid collisions?

Sure, that is possible. If you're going to care about this, you should stop using Bitcoin, though. We rely on much weaker cryptographic guarantees already.

Will NTXID be validated by nodes or will it be susceptible to malicious intentional collisions?

EDIT: I thought the ntxid was going to be broadcast with the transaction. Since it is client side only, there's no risk of intentional spoofing.

Owner

sipa commented Feb 13, 2014

NTXID is just a service-level thing. It does not exist at the level of the blockchain, the protocol, the peer to peer network, or the consensus rules. There is no validation on it.

It just exists to help humans quickly find identical transactions.

@sipa, sorry, don't want to waste your time. Yes, I was referring to SHA256 collisions (not just being paranoid, but wanted to clarify things).
But BIP30 already deals with this for transactions.

Anyway this implementation does not alter current processes (mining, block validation, etc), so the patch is much easier to validate and without possible unforeseen side effects that other patches could have.

@tmairegasnighto making collisions (if you mean duplicating an the hash as in SHA256 collision) is practically impossible to do.

Updates have been today as to how blockchain.info handles mutable transactions which are relevant to this discussion.

We are now using Sipa's normalized hash scheme to detect transaction conflicts internally. If multiple transactions are received with the same NTXID only one will be kept in the memory pool.

  • If searching for the conflicting hash which wasn't kept in the memory pool it will transparently resolve to the other transaction matching the same NTXID.
  • When one of the conflicting transactions confirms you will always be able to resolve to the matching NTXID using either hash.
  • The NTXID is never displayed to the user and you can't query by NTXID.

When two unconfirmed transactions with the same NTXID are detected the following note will be displayed "A conflicting transaction has been detected in our memory pool. The transaction hash may change."

When the conflict is resolved and you search for hash which failed to confirm a note will be displayed "The transaction hash you searched for was not found. However a replacement for the transaction is listed below."

Service providers, such as Mt.Gox, therefore do not need to be using the same NTXID scheme as blockchain.info but can still use transaction hash links on blockchain.info as "receipts" guaranteeing that even if the transaction is replaced the link is still valid.

@zootreeves:
Hmm sounds good however i see one problem in your plan (correct me if i am wrong).
Imagine i send transaction, somebody modifies its ID. Modified/mutated transaction gets into the block before original transaction gets into your bitcoin client memory pool. This way you will have no link to ID of original issued transaction and no way to find it. So MtGox still should use NTXID for searching not TXID.

Owner

sipa commented Feb 13, 2014

@zootreeves Not sure why you need this normalized hash for that. The mempool (at least in recent bitcoind releases) already guarantees consistency and prevents two transactions that consume the same input from entering the mempool at the same time.

@underhood It will not make it into the memory pool, but we will still record a link from hash to NTXID regardless, so searching by either hash will work. Edit: If we don't see it on the network at all then yes the link will not resolve.

@sipa Blockchain.info displays double spends, so we allow conflicts in the memory pool. Before this manipulated transactions were displayed with a big "double spend" warning.

@zootreeves: I think you don't get me.
In case i described (original transaction not reaching your mempool before malled gets into block):
you can do malled transaction hash --> NTXID
but not original transaction hash --> NTXID (as original transaction with original hash never reached you)

so exchange cannot use original transactinon hash links as you suggested

@underhood Yes I see. In that case a standardised hash is needed.

My only reluctance is allowing searching by NTXID for all transactions means maintaining another large, ever expanding, index. Using the previous technique we only need to keep a link from hash to NTXID while waiting for the transaction to confirm, only if conflict actually occurs then we need to keep the link permanently.

lnovy commented Feb 13, 2014

I don't think this should be considered as standard. My reasoning follows. I would strongly prefer this to be considered as best-practice only as there are clearly some benefits of using this, but this is only limited to specific scenarios and even this is limited to current situation where malleability of transactions is possible. This is a this temporally solution to a design flaw of Bitcoin protocol and by making this "hot-fix" a standard we are accepting this flaw as something correctly designed.

  • Standards are here to make cooperation between different parties easier and in many cases even possible.
  • Proposed approach does not brings any benefit to cooperation between different parties because it is by design created to be used by sending party only
  • The primary goal of normalized transaction hashes is to allow sending party to identify their own transaction and match it to possible muttated versions of it. There is a obvious benefit for sending party to know if that concrete transaction made it into the blockchain and this can be done only by matching muttated versions.
  • For the receiving party to make sure that a concrete transaction made it to the blockchain there already exist a way to do so and using normalized transaction id for this purpose therefor does not bring any benefit. In fact using normalized transaction hash is more difficult than current available methods as a new hash need to be obtained from the sender. If the receiving party chooses to it can simple do this tracking by using different receiving address for every transaction.
  • Normalized transaction hash cannot be used by receiving party for accounting purposes contrary to usage by the sending party.
  • As there is not a single usage of normalized transaction hash by receiving party that cannot be done by already available means there isn't actually any reason for receiving party to bother about it or know it.
  • As there is no need to publish this new hash by sending party because there is no external benefit for doing so, this hash should be kept internal.
  • As normalized transaction hash is only used internally by sending party, it's computation is not dependent on any internal structures or inner working of bitcoin client software, it is computable from a standalone transaction, this computation can be done by external application and it's usage is applicable strictly outside the bitcoin software, we should not "enforce" a way to compute this hash and let this to a decision of sending party.
  • When/if malleability of transactions is properly resolved this new hash will be creating no other benefit for the sending party compared to using regular transaction id. this duplication should be avoided in front and making this a standard would make it much harder in future to get rid of.

@zootreeves As your business revolves fairly tightly around keeping indexes for the blockchain that nobody else does, it's not like other people are ever going to encounter a similar problem. I can't imagine you'll ever be in the situation where your revenue is less than the cost of adding additional drives to a ZFS pool from time to time.

Contributor

maaku commented Feb 13, 2014

I didn't know this had gotten to be a pull request. I'm working on a fork of sipa's branch which adds error corrective capabilities and makes the normative ID visually distinctive from the protocol transaction ID in the following ways:

  1. 64 base32 characters instead of base16 (hex).
  2. Automatic correction of up to two errors in input, and highly probable detection of non-recoverable errors.
  3. Normative ids begin with the characters "tx"
  4. Visually distinctive grouping of 8

The purpose of these changes is not to bring additional security, but rather to accomplish goals which will save time and money for customer-service operations:

  1. Make normative IDs and protocol transaction IDs visually distinctive so as to prevent user confusion. "I just read you the hash, what other number are you looking for?"
  2. Correct small transcription errors, of the sort which inevitably result from reading a string of 64 digits/numbers over the phone in a customer support scenario.
  3. Provide visual grouping to assist in human transcription from paper or 2nd device, or over the phone.

The result is a normative ID that would look like the following:

tx5x14j6-5cqqkytc-tq646pss-2hq3sm6k-pw1vxz1m-r7jkkjsy-ts0rx1dx-f79mv3jw

I'm expecting a reviewable pull request to be done today, if all goes well.

"NTXID is just a service-level thing. It does not exist at the level of the blockchain, the protocol, the peer to peer network, or the consensus rules. There is no validation on it.

It just exists to help humans quickly find identical transactions."

A good reason to call it a hint instead of an id, because if you call it an id people will mistake it for something validated in the protocol.

@maaku your ID example does not pass a 'copy-paste' test. A double click on
it doesn't fully select it in a browser. All other bitcoin's identification encodings were designed to be easy to select. You should get rid of the dashes at least.

@justusranvier I agree with you.

@maaku I'm having trouble getting enthusiastic about this one. What possible scenario would involve a user being forced to read out 64 characters of information to a companies support? The original point of a "normalized ID/hint" is for a service to be able to identify their own outgoing transactions, and that's it.

Contributor

maaku commented Feb 13, 2014

@b733686a-14ea-4048-aa94-44c2cb029830, if you scroll up in this discussion you'll find representatives from Coinbase talking about sending normative IDs to customers which they can use to lookup transactions in their wallet or some centralized indexing service.

And, believe it or not, a large segment of the non-tech community actually likes to do things like arrange transactions over the phone. Old skool, but that's how people are. "Please remain on the line so I can give you your transaction's helpdesk identifier..."

If is is just a service level thing why not use base58 and trim the hash to 16 or 20 bytes.

Member

gmaxwell commented Feb 13, 2014

From experience base58 is remarkably hard to read without errors due to mixed case, FWIW.

@maaku Right, I'd missed that part, my mistake.

I'd work on your encoding though. I'm in the same school of thought as @gmaxwell in that it's extremely hard to read, and with @llongeri that keys with hyphens in them are difficult to work with due to copy-paste not seeing it as a "word". If anybody else, this novelty account proves that working with GUID-like formats is a right pain in the ass from both an identification and usability standpoint.

Contributor

maaku commented Feb 13, 2014

@b733686a-14ea-4048-aa94-44c2cb029830, gmaxwell was commenting on the suggestion to use base58. I think that is a bad idea for the same reasons. My encoding is base32 to avoid case sensitivity issues (also a problem for over-the-phone situations). I'm fine with removing hyphens as copy-and-paste seems like a more common use case than the need for visual groupings. The decoder should be insensitive to hyphens anyway.

Unitech commented Feb 13, 2014

@MagicalTux Ca fait maintenant bientôt deux semaines que MtGox bloque les retraits de Bitcoin. Pour les gens et les développeurs comme moi c'est une triste histoire, qui nous enlève des possibilitées d'entreprendre certains projets... Est-ce qu'un jour nous allons pouvoir accéder à notre petite fortune qu'on a, pour certains, mis beaucoup de temps a accumuler ?...

Contributor

maaku commented Feb 13, 2014

@Unitech, totally the wrong place for MtGox support.

T4Truth commented Feb 13, 2014

don't you guys see what is going on here? MtGox is trying to cover up for the gross incompetence that left them insolvent: they paid people multiple times with totally separate transactions and didn't notice until their wallet was empty.

Now they are holding customer's BitCoin trying to extort some free labour from the volunteer developers here but the labour isn't really what they want. By conning you into adding this sham fix they hope to shift the legal liability for their incompetence onto some unpaid open source developers.

Do not fall for it. The coins aren't there, and they don't even run BitCoin-qt so nothing here will help them.

Member

gmaxwell commented Feb 13, 2014

@T4Truth This normalized hash stuff doesn't have anything to do with coin loss— the kind of doubled payments you're talking about can only be prevented by making sure the reissued transaction conflicts with the original, giving you atomic exclusion. Nothing else is safe, since nothing else prevent both txn from going in (even if the inputs were already conflicted by another transaction, due to the possibility of a reorg).

What this is supposed to help with is the user confusion related to a giving the user one txid and having another show up. Since some transaction types are inherently malleable (as a feature!) this can't work for all transaction types, but for the common, boring ones, it could. For internal use, there are better things that you can do, like finding the conflicting transaction and actually comparing it (allowing you to also handle ordinary double-spends).

I don't think any of the core developers here view this particular piece of software as a high priority right now, relative to the DOS bugfixes, however. Probably the discussion for this should move to the mailing lists like other ecosystem standards stuff. Regardless, a second independent implementation will really be needed to move it forward. @maaku You weren't planning on writing a second implementation, by any chance?

il--ya commented Feb 13, 2014

@zootreeves "If is is just a service level thing why not use base58 and trim the hash to 16 or 20 bytes." 4 byte ID is more than enough for identifying transaction in internal database. Any collisions can be resolved using destination address or user ID etc. There is no need to standardize "normalized ID" hash or include it into reference implementation, I agree with @lnovy on that. Too much time wasted, IMO, on this red herring thrown in by MtGox.

Member

gmaxwell commented Feb 13, 2014

interesting point— indeed the classic way to identify a payment is that you're supposed to use a virgin scriptpubkey for every payment, and then you know you've been paid when it shows up— but that doesn't match how many people use Bitcoin today. Maybe [[8byte ID]-address-value] would be a much better normalized identifier, in fact.

lnovy commented Feb 13, 2014

Another thing that I'm not ready to accept is that even with the way Bitcoin is used today the probability of to the customer once given out transaction id changing in the future can minimized to negligible, if one just waits for the transaction to be confirmed. In that case only reorgs could/would be causing this id to change and exchanges would have their freedom of choice when deciding how to implement their accounting and muttated transactions detection. Putting a trust into a transaction with no confirmation is not very smart anyway.

Possibility of grouping transactions and using sendmany comes as a free bonus here.

@gmaxwell 8 bytes is overkill as well. Even if you were looking at a Satoshi Dice level of address reuse you're unlikely to ever get into the situation where you have a collision. 4 bytes of the signature (assuming 0.8 clients that enforce encoding style) + a few bytes of the HASH160 the transaction output was to and you'd be home free.

@lnovy Agreed, it's unlikely that a withdraw from a service like Mt Gox wouldn't be included in the next block (especially with their agreement with Eligius, does that still exist?) as they pay more than the required fee. I imagine any support situation mentioning a particular transaction could be confirmed 10 times over before any support ticket was answered. Once it's in a block the chance of it being altered by a reorg is completely insignificant, I doubt you could find a single instance of that ever occurring.

lnovy commented Feb 13, 2014

You could have a split which each chain having different version, but this is just a corner case, is solvable by 1 proactive email and support desk would be already prepared for that potential ticket.

jfollas commented Feb 13, 2014

The unmalleable transaction/reference id could be based on deterministic data, like a Merkle tree made up of hashes + index of the transaction's TXINs.

Member

gmaxwell commented Feb 13, 2014

@jfollas Please don't comment without even reading the patch.

This is getting us nowhere. There is indeed so many things we could do on our own even at MtGox, however this merge request's purpose is to define a standard others using Bitcoin can agree to and that can be implemented.
Getting rid of malleability would be the ideal idea, but this won't happen anytime soon no matter how everyone tries. It can be made harder, but not impossible.

On the other hand, I believe keeping track of what is actually signed when a transaction is issued makes sense, and it seems that some others here agree to that.

Re-issuing of transactions should be done using the same inputs, but as of today this poses a problem as said txs won't be relayed as easily as clean txs. There is also the fact that people using regular bitcoind as backend won't be able to do that either.

Keeping only a service-specific ID stored in the transaction is not a solution either, as it would be difficult to index globally (ie. we have to find the right txs from the whole blockchain).

Anyway we came up with this new hash solution because:

  • it requires no change to the protocol and will not cause any forks or issue to people not using it
  • because bitcoind already has facilities to exclude the txin script from a tx to hash it, it is very easy to implement
  • because it's a hash of what is actually signed, anything causing this hash to change would also invalidate the signature

As for CoinJoin, I believe the client will have a view of the transaction before it is actually sent (to ensure it includes its outputs and to sign it). As such it is perfectly possible to fetch this hash at that stage, even before knowing the actual tx hash (and it could remove the need for the client to see the actual tx to know its hash as they potentially wouldn't need it anymore)

Member

gmaxwell commented Feb 13, 2014

@MagicalTux Reissuing a transaction without reusing the inputs is unsafe. You cannot do this and be confident you will not be robbed, because there is nothing preventing both transactions from going through. No amount of checking can provide protection in that case, because the prior transaction may not have gone through yet but still may go through for as long as it has not been conflicted— or even after it has been, if the conflict can be removed in a reorg. It may be slow, it may not work super well, but it is the only way to prevent transactions from being paid twice. This has nothing to do with malleability: You simply cannot pay someone twice and not risk them actually getting paid twice, unless their original payment is impossible or the transactions are mutually exclusive.

If including this patch is going to make even one service believe they can safely reissue in this manner than we must not do it.

Pull requests here are not used to define standards, normally "standards"— in so far as we have them at all in the Bitcoin community— are defined using BIPs. The GitHub page was a quick way to get code up so that people could start commenting on it— and get it if they needed something to use internally to help sort things out.

It is not our responsibility to run your service, so if you're waiting on anything on this pull req you are doing something wrong. You already issue several kinds of mtgox proprietary ID with your transactions.

anything causing this hash to change would also invalidate the signature

This isn't true for all transactions, it's true for the conventional ones you issue— indeed. Some transaction types are inherently malleable and no ID can be both unique and non-malleable for them.

Keeping only a service-specific ID stored in the transaction is not a solution either, as it would be difficult to index globally

I am not a fan of putting an extra ID in transactions because it bloats them, it's bad for privacy, and can potentially be simulated in other transactions (unless you go through the heroic effort of making it a signature)... But there is no reason why it should be any more difficult to index than any other data from a transaction. Instead of running the normalized ID extraction, you run the service ID extraction, and index that if there is one.

@gmaxwell
I can't imagine that at this time anyone would suggest adding any more data to the transactions of the block chain, considering the size of it is already considered a high priority issue and that it would most likely require a hard fork and complete reindex of everyone's clients.

Member

luke-jr commented Feb 13, 2014

Indeed, the change-scriptPubKey-hash proposal only suggests using existing data.

Contributor

maaku commented Feb 13, 2014

@MagicalTux If you re-issue a transaction which does not double-spend at least one of the original transaction's inputs, you will lose money. With the way that the protocol works, double-spending is the only option to ensure that two transactions remain exclusive of each other and only one makes it on the chain. Otherwise the original you issued remains valid and might be confirmed, in which case you'll find yourself in the position of asking for your money back.

lnovy commented Feb 14, 2014

There are actually 2 different things that exchanges are trying to address here.

  • The need of some id of their own transaction that is invariant under transaction malleation.
    • this should be some internal token, bitcoin client nor network nor customer is interested in that.
    • the only reason why regular txid cannot be used here and for this type of transactions is the potential transaction malleation in the time before the transaction is locked by the blockchain.
  • Exchanges assert that their customers need a similar way to identify withdrawal transaction issued to them by exchange and also assert that their customers need this token even before the transaction is locked by the blockchain.

From the exchange point of view this need of having invariant hash even before the transaction gets mined is understandable, from the point of view of their customers I just can't see any need for that.

The simple knowledge of whether the transaction was broadcasted to the network by exchange is not of any value for customers, for them transactions are interesting only after they are included in the blockchain and trustable. At this point any malleation of this transaction is of very low probability and therefore the normal txids can be used.

I need to ask you this once again, is there any real benefit for customer here or could all useful properties that ntxid is trying to provide to customer be provided by currently implemented means? Is there any real benefit for a standardization of some purely internal service provider process?

lnovy commented Feb 14, 2014

Giving the transaction hash to the customer before the transaction is confirmed only brings complications on every side even when this not potentially useful for anyone. It is not even more difficult or expensive for service provider to delay this giveaway as it need to track the transaction progress anyway.

@invony Assuming that the method of creating that token is standardized, and possibly built into future clients without being included in the blockchain it should be fine. The users would still be able to search for the transaction on blockchain.info and if they update their wallet they can do so in their client.

This way it would be useful for the client, and the exchange without requiring a fork or putting any more strain on the network.

lnovy commented Feb 14, 2014

If the customer is given the final txid in first place there is no need for this to be in client nor this need to be implemented by blockchain.info. Why should we implement something that is not needed and is duplicating a functionality of txid but only for smaller subset of possible transaction types?

il--ya commented Feb 14, 2014

@gmaxwell @maaku It's quite worrying and real pain in the backside that failed transactions may potentially be included at some later point into the blockchain. It seems at the moment the only way to properly "cancel" previous transaction which failed to be confirmed is to spend at least one of it's inputs. But what if that fails as well? Then you have yet another stuck transaction to worry about and keep track of.. Would it be possible to add something opposite to nLockTime, time or block height after which transaction becomes expired and should not be included into a block?

@lnovy While I agree with you, it also provides a useful index for companies that use 0 confirmation transactions to build up their databases.

I would agree with you fully if it were to be included in the blockchain, but if it is calculated on the client side (and therefore something that could be turned off) it will simply provide another measure against malleability until/if it is fixed.

@il-ya unless I am mistaken, this could be done simply by creating a fresh wallet and sending the entire balance to it, which would take a matter of minutes.

Contributor

maaku commented Feb 14, 2014

@il--ya off-topic. Short answer: yes, with a fork, but it introduces edge cases where entire chains of transactions can be invalidated. I personally think the pros outweigh the cons, but there is not consensus on this. Please take follow-up discussion to the forum, mailing list, or IRC (not here).

lnovy commented Feb 14, 2014

Companies that need to build these indexes need them strictly for internal needs of speed benefit when fighting the malleability, but contrary to bitcoind they can actually affort this as they can limit the type of transactions they are issuing. It is not possible to do this hashing generally in bitcoind and we are fighting the malleability differently. The point is that the method of achieving this speed benefit is strongly bound to business category and these categories nor methods hasn't been even enumerated yet. This proposal is a best-practice for one of this category and one method but is presented as been a magical standard solution for everybody. Finally as I've stated before if this is used internally and not for enhancement of different parties collaboration/communication it should not be called a standard.

To make the use case clear:

I am company X. The customer Y wants to withdraw.

  • Y requests payment (out of band, http for example)
  • I perform a payment.

Now either I can CLAIM to have made the transaction, and the user has to
wait 10*x where x >= 1 to know if I'm right. OR I can give him the txid.

If X gives him the txid it might get malleaded and fail and cause a big
fuzz. This is not a neat choice, but might be the best we can do.

X could give Y metainfo, like which inputs were spent. This is pretty
complicated. X could give the whole transaction. That's not at all easier.

X could issue a proprietary ID. Then the only place to check on the tx
would be X's service or those allied with X. Y likely doesn't have any
capacity for checking.

Thus: Mt Gox asks for a unique ID for unverified transactions. The use case
being pre-confirm communication. This is considered to be important.

Such an ID Y can dump into his client and see "transaction in network",
"transaction might be next in block", "transaction confirmed once" and
after a few "transaction confirmed" (when cost-of-falsify > value of tx).
Even if Y unknowing.

Internally this ID might help X. Especially interesting will be the
confirmed once then reverted in another, longer, chain with a different
txid bugs.

On the flip side I have to say it is disappointing to see Mtgox crash our
fine currency like this. It has a lot of people very worried about what's
really happened. Extra so as it is very much solvable by Mtgox itself,
although the solution would not be as pretty.

Bottom line I am buying and hoping Mtgox is secretly working in a fix akin
to Coinbase's fix, to reopen withdrawals ASAP.

Furthermore I would like to see a txid be something that can not be
changed, like an output could not be changed. As this malleability thing is
hard to spell and confusing.

I think Mtgox should reopen and this problem must be given its fair
attention.

For now this is not standardized in a BIP and is not fixable. I wouldn't
mind a "practical standard" for now, but this is not my decision.

Cheers.

lnovy commented Feb 14, 2014

Until the transaction is confirmed it enjoys exactly the same level of trust as any other claim.
Compare:

  • User Y requests a payment, is given a ntxid, puts it into blockchain.info, transaction might appear, warning might appear that the transaction was malleated, needs to wait avg. 5 minutes until transaction is confirmed to trust you.
  • User Y requests a payment, is given a message that withdrawal is been processed. needs to wait avg. 5 minutes until is given txid, puts it into blockchain.info, transaction is always confirmed at this stage, trusts you.

@justusranvier +1 on the hint vs id

As a developer new to bitcoin this has been very confusing, it seems small but with documentation only in the git logs txid has been little devil to work with from the start even without the current issues.

Just to update this thread, it seems that this discussion is mostly stale now. We (at MtGox) will implement this new hash index in our transactions database and start working with it (we will announce a maintenance as we will have to stop bitcoin deposits too during the database schema update) and will start providing this new hash when customers are withdrawing bitcoins, litecoins, or any other coin based on Bitcoin we may support in the future.

We will also provide an API that will allow our customers to use this hash to retrieve the transaction hash as seen in the blockchain once the transaction is confirmed, and will hope others (blockchain.info?) will index this value one day.

We also invite other exchanges and businesses which may need to keep track of bitcoins they send to use this same method, since dealing with multiple variations of the same thing wouldn't be very productive.

If nobody does it, we will also post some test vectors for regular (in=>out) transactions in the near future.

Contributor

maaku commented Feb 14, 2014

I've finished the encoder and a verification program which exhaustively demonstrates the error-correction capabilities of the CRC polynomial used. It's available at this gist:

https://gist.github.com/maaku/8996338

You can test it out like this:

$ ./encode-ntxid 4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
txr5munp7vx5jcsv6omwyhht56yq74pp75pi6ggmekrrsmutpcxk5s4bpjj6ifr6

I'm still reworking the decoder which requires slightly different logic, but hope to have that posted shortly.

lnovy commented Feb 14, 2014

@MagicalTux this is OT, but I hope you will keep displaying the regular txid along with it, maybe adding some flag (confirmation count?) to let customer know that this txid is final? Also ping me on freenode to discuss some potentially troublesome properties of your implementation I'm concern of...

Definitelly ACK on this... I think however format of this NTXID should be discussed. Should be reasonably concise and adding error correction is good to have(against mistype or errors in manual copying from paper).
Also good point is being able to distinguish TXID from NTXID by format of it to avoid confusion.
Selectability by double click in browser is nice thing to have but not critical for me.

So to sum this I think patch should be accepted. What we should discuss is only format of this hash then Finally merge it with main branch so that merchants, services, blockchain.info, etc. can start their work on common ground.

Preliminary support for searching by normalized hash is now available on blockchain.info. Links can be resolved using /ntxid/* path e.g.

https://blockchain.info/ntxid/3c0b247b0f9107309c603441f0411b1e8a54dbbb86c892d0aced6aa285804c7f

Two new query tools:

/q/hashtontxid/$txhash - Convert a transaction hash to ntxid
/q/ntxidtohash/$ntxid - Convert an ntxid to transaction hash

Only recent transactions are currently available, historical transactions will be indexed over the weekend. The ntxid format can be updated according if a standard is agreed. For the email notifications etc we will still be using tx hash links.

@MagicalTux as a long time MtGox user I'm happy to hear we can finish this black chapter soon.

@zootreeves I greatly respect the speed at which new features have always appeared on blockchain.info. Thank you.

I hope that with this we can move on a little bit.

@gmaxwell @sipa @otherdevs I think this forum is outlasting its utility. Keeping it open for further updates on progress might be a good idea, but discussion should shift towards discussing the format and a BIP.

lnovy commented Feb 14, 2014

I found another lethal problem with this. Normalized transaction hashes try but are not by definition unique and sender needs to put out large effort to assert that transaction which he issues has non-colliding ntxid with another one already included in the blockchain if he wishes to use this hash for accounting purporses or eg. as a relation databases' key.

From BIP0030:

Blocks are not allowed to contain a transaction whose identifier matches that of an earlier, not-fully-spent transaction in the same chain.

Reverting this it is possible and legal for a transaction with identical txid to by included in the blockchain in multiple copies.

BIP0030 s enforcing the uniqueness only on the set of transactions which have each one or more unspend output. Proposed ntxid is not reducing this set of transactions in any way as this global uniqueness in the chain is here one of the main desired properties. Making sure that this uniqueness is not broken requires for every client to maintain an index of every transaction ntxid in the blockchain and to check for potential colision on every signing of transaction. It is possible and probable, that such cases breaking this uniqueness are already included in the blockchain and together with high cost this brings to assert this uniqueness this proposal has high potential of causing hard-forks.

I now personally declare this proposal as broken by design.

@inovy, so you're also in favor of calling it a hint, not an identifier?

lnovy commented Feb 14, 2014

I would consider substantial rework from scratch, the only usable part of this is it's ability of detecting mutant versions of your own transaction of specific type in the time window before this transaction is confirmed in the blockchain. In this case it's actually an id and process of obstaining it should be put out as a best-practice to mitigate some of the current malleability issue at condition where the method used by bitcoind is hard to use or when bitcoind's approach is too general and this brings time&cost benefits. Because of it's non-uniqueness the proposed planed use of this as an accounting handle is also not possible.

Owner

sipa commented Feb 14, 2014

As I've said before: since BIP34, transaction ids are unique.

lnovy commented Feb 14, 2014

Mae culpa. I need to lower my declaration from broken by design back to "not useful for anyone else than the sender internally when not duplicating properties already available" :)

EDIT: It seems I owe you all a big apology as I can no longer stand behind most of the claims I've written here and in other places.

Contributor

maaku commented Feb 15, 2014

My gist has been updated with an error correcting decoder:

https://gist.github.com/maaku/8996338

I also have most of the code necessary to integrate this into bitcoind on top of @sipa's commits. That will be the next task. You can use the gist C++ programs like so:

$ ./encode-ntxid 4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b
txr5munp7vx5jcsv6omwyhht56yq74pp75pi6ggmekrrsmutpcxk5s4bpjj6ifr6
$ ./decode-ntxid txr5munpavx5jcsv6omwyhht56yq74pp75pi6ggmekrrsmutpcxk5s4bpjj6ifre
4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b nExtra=0 {8:'7', 63:'6'}

Note that the value passed for decoding had two errors: one at position 8 (an 'a' instead of a '7'), and another at position 63 (an 'e' instead of a '6'). These errors were corrected and reported as part of the decoding process.

I'm not entirely clear on what "normalized transaction hash" is a hash of. Is it a hash of everything but the signature? If someone has a spare minute could you explain...
conventional: txid -- sha256 hash of whole transaction (right?)
"normalized transaction hash": ntxid -- hash of ???

Contributor

maaku commented Feb 15, 2014

It is that thing which is signed by the holder of the coins in order to authorize the transaction, for standard transactions under normal circumstances.

Transactions are "malleable" because there are portions which are not covered by the signature and therefore can be altered without invalidating the signature. A normative / canonical ID excludes these parts of the transaction as well, so the "view" of the transaction when creating this ID and when signing it are the same, therefore anything which changes the ntxid would invalidate the signature, and anything which invalidates the signature would change the ntxid.

Now having said all that, I urge you to look at the patch itself before commenting. It's rather straightforward and the answer to your question is in the very first lines:

sipa/bitcoin@e7853a9#diff-9c8594b107c5d10b42944ad46a2f1a76R82

Owner

laanwj commented Feb 15, 2014

@boborsomeoneelse The code that actually serializes the data that will be hashed is in https://github.com/bitcoin/bitcoin/blob/master/src/script.cpp#L1054
(txTo=transaction, scriptCode=CScript(), nHashType=SIGHASH_ALL, nIn=0).

This hashes (w/ SHA256):

  • nVersion
  • vIn:
    • Effectively all input scripts are blanked out before serialization
      • Blanks out input scripts except for input 0 by serializing CScript() (nInput != nIn). The script for input 0 is replaced with scriptCode=CScript() (the empty script) (nInput == nIn)
    • That leaves: prevout, CScript(), nSequence for every input
  • vOut:
    • Outputs serialized in the normal way
  • nLockTime

@MagicalTux MagicalTux added a commit to MagicalTux/gentoo that referenced this pull request Feb 15, 2014

@MagicalTux MagicalTux bitcoin-0.8.6-r1: backported pull request bitcoin/bitcoin#3656 0712aaa
Contributor

gavinandresen commented Feb 15, 2014

Rough consensus seems to be this is a good idea. I don't see any consensus for an error-correcting or more complicated version of this.

RE: looking up by normalized transaction ID: automatically creating a multimap<normtxid,txid> index violates a design principle of "don't add features that cost something even when they are not being used, unless that feature will be used by the vast majority of users." (the cost here would be memory used for that index) I don't think lookup-transactions-by-normalized-txid will be used by the vast majority of bitcoind RPC users.

I suppose a "gettransactionsbynormalizedid" RPC call could build the index the first time it was called, but that might take a while on a big wallet.

(UPDATED quick thought: perhaps running with -txindex=1 should turn on this feature....)

In any case: @MagicalTux , do you need to be able to take the normalized txid and get back a list of wallet transaction ids that correspond to it? Note that with latest patches (and this patch), gettransaction will return a conflicts array that will tell you the mutant version of the transaction that ended up getting mined.

Owner

sipa commented Feb 15, 2014

@gavinandresen Looking up by normalized txid, you're talking about just indexing the wallet, not the whole blockchain I hope (if so, why does -txindex have anything to do with it?).

Within the wallet... I'm not sure, it may encourage people to use it for more than it is meant to. But not being able to look up wallet transactions by ntxid would be implementing a feature that's really only useful when centralized indexes exist for it. On the other hand, even for 100000 wallet transactions, it would just be few MB.

Contributor

dexX7 commented Feb 15, 2014

blockchain.info implemented a lookup (see comment from @zootreeves). From my point of view it appears to be appealing for application developers to simply replace any usage of txids with ntxids and as long as this is no universal replacement for txids I suggest to discourage the usage of it as such as much as possible.

@dexX7 I agree, I don't see what can go wrong with its usage although I don't think ntxid is a good name as it may confuse some people. It should have a completely separate name.

Contributor

maaku commented Feb 15, 2014

I've pushed the patch using base-32 encoding and CRC error correction polynomials to my own repository. It works, but still needs some test vectors. Any code review would be appreciated:

https://github.com/maaku/bitcoin/tree/normtxid

@ryanxcharles ryanxcharles referenced this pull request in bitpay/bitcore Feb 16, 2014

Closed

Support normalized txid #64

Maybe someone from the devs could answer the following question on bitcoin.SE about what the normalized tx hash is:
https://bitcoin.stackexchange.com/questions/22168/what-is-the-normalized-transaction-hash-of-a-transaction

I think that would be very helpful towards other parties trying to implement/use them.

@rebroad rebroad commented on the diff Feb 18, 2014

src/rpcrawtransaction.cpp
@@ -801,3 +801,25 @@ Value sendrawtransaction(const Array& params, bool fHelp)
return hashTx.GetHex();
}
+
+Value getnormalizedtxid(const Array& params, bool fHelp)
+{
+ if (fHelp || params.size() != 1)
+ throw runtime_error("blah");
@rebroad

rebroad Feb 18, 2014

Contributor

blah?

@lnovy

lnovy Feb 18, 2014

Read the whole thread.
This is a not a serious proposal for inclusion into bitcoind.
This is a simple implementation of a proposed standard to verify compatibility.

Contributor

maaku commented Feb 18, 2014

I added unit tests, and the ability to encode/decode strings which are not a multiple of 128 bits in length (to support applications other than ntxid):

https://github.com/maaku/bitcoin/tree/normtxid

Contributor

maaku commented Feb 20, 2014

I've pushed a hopefully final version of the error correction coded normative transaction ID branch to my public repository:

https://github.com/maaku/bitcoin/tree/normtxid

This version breaks behavior from what is described in this pull-request by using the regular (not SignatureHash) transaction ID for coinbase transactions. The patch in this pull request has the potential to result in duplicate ntxid's for coinbase transactions since the coinbase string which contains the BIP-34 block height is a scriptSig and therefore stripped from the normative data structure. At this time I cannot imagine why you would need a normalized transaction ID for a coinbase transaction. But just because I can't imagine a use doesn't mean there isn't one, and services indexing the block chain need to make a decision about this edge case.

Additionally, the code now provides error correction coding for arbitrary length base32 strings, and complete coverage with a suite of unit tests. It also corrects two bugs that were found in the encoding algorithm, which were uncovered in the process of writing the tests. All that remains to be done is to write a BIP documenting its inner workings.

Owner

laanwj commented Mar 10, 2014

Time to close this?

This does not provide a solution to transaction malleability abuse but only a workaround with very limited applicability. This is basically only good for identifying your own transaction to users for helpdesk usage. There are many other ways to do that, such as providing some internal database ID or even giving the whole transaction hex.

Wallets have better ways of being robust against malleated transactions, see for example the handling of conflicts in 0.9.

Also it appears the 'malleability issue' was only the tip of the iceberg of problems at MtGox. This feels like it was a diversion.

All in all, I'd recommend closing this without merging.

Contributor

gavinandresen commented Mar 10, 2014

I agree with @laanwj -- this should not be merged.

@ghost

ghost commented Mar 10, 2014

Agreed wholeheartedly, this should not be merged.

@sipa sipa closed this Mar 10, 2014

@wtogami wtogami referenced this pull request in slickage/baron Mar 22, 2014

Closed

Notes on normalized txid #2

@wtogami wtogami added a commit to litecoin-project/litecoin that referenced this pull request Mar 27, 2014

@sipa @wtogami sipa + wtogami Add normalized transaction hash
Rebased-from: e7853a9
Rebased-by:   Warren Togami <wtogami@gmail.com>

Original code from bitcoin#3656

Warning
=======
This patch was rejected from Bitcoin Core and must be considered experimental.
Theoretically it is compatible with the de facto standard as utilized by
blockchain.info and a few vendors.
149487d

@wtogami wtogami added a commit to litecoin-project/bitcoinomg that referenced this pull request Mar 27, 2014

@sipa @wtogami sipa + wtogami Add normalized transaction hash
Rebased-from: e7853a91cf646a6a4701158d148f036924575a97
Rebased-by:   Warren Togami <wtogami@gmail.com>

Original code from bitcoin/bitcoin#3656

Warning
=======
This patch was rejected from Bitcoin Core and must be considered experimental.
Theoretically it is compatible with the de facto standard as utilized by
blockchain.info and a few vendors.
2c5dc6e

@wtogami wtogami added a commit to wtogami/bitcoin that referenced this pull request Apr 4, 2014

@sipa @wtogami sipa + wtogami Add normalized transaction hash
Conflicts:
	src/rpcserver.cpp
	src/rpcwallet.cpp

Rebased-from: e7853a9
Based-on:     bitcoin#3656
b1b7452

@wtogami wtogami added a commit to litecoin-project/litecoin that referenced this pull request Apr 4, 2014

@sipa @wtogami sipa + wtogami Add normalized transaction hash
Rebased-from: e7853a9
Rebased-by:   Warren Togami <wtogami@gmail.com>

Original code from bitcoin#3656

Warning
=======
This patch was rejected from Bitcoin Core and must be considered experimental.
Theoretically it is compatible with the de facto standard as utilized by
blockchain.info and a few vendors.
bc7bc9d

@matiu matiu referenced this pull request in bitpay/bitcore Apr 11, 2014

Merged

add normalized hash for Txs #249

@wtogami wtogami added a commit to wtogami/bitcoin that referenced this pull request Apr 24, 2014

@sipa @wtogami sipa + wtogami Add normalized transaction hash
Conflicts:
	src/rpcserver.cpp
	src/rpcwallet.cpp

Rebased-from: e7853a9
Based-on:     bitcoin#3656
d809936

@MathyV MathyV added a commit to reddcoin-project/reddcoin that referenced this pull request May 9, 2014

@sipa @MathyV sipa + MathyV Add normalized transaction hash
Rebased-from: e7853a9
Rebased-by:   Warren Togami <wtogami@gmail.com>

Original code from bitcoin#3656

Warning
=======
This patch was rejected from Bitcoin Core and must be considered experimental.
Theoretically it is compatible with the de facto standard as utilized by
blockchain.info and a few vendors.
98829dd

@icanprogram icanprogram pushed a commit to MidasPaymentLTD/midascoin that referenced this pull request May 17, 2014

@sipa @wtogami sipa + wtogami Add normalized transaction hash
Rebased-from: e7853a91cf646a6a4701158d148f036924575a97
Rebased-by:   Warren Togami <wtogami@gmail.com>

Original code from bitcoin/bitcoin#3656

Warning
=======
This patch was rejected from Bitcoin Core and must be considered experimental.
Theoretically it is compatible with the de facto standard as utilized by
blockchain.info and a few vendors.
30e5e56

@dcousens dcousens referenced this pull request in bitcoinjs/bitcoinjs-lib Feb 12, 2015

Closed

Transaction: get normalized hash / id #355

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment