Can serialize and deserialize transaction with witnesses #1171

NicolasDorier · 2016-01-25T22:33:59Z

First step for implementation of segwit, transaction serialization and deserialization

NicolasDorier · 2016-01-25T22:41:39Z

You can cross check transactions on https://blockchainprogramming.azurewebsites.net/checktx

There is also another problem which I bumped into with NBitcoin: It is impossible to deserialize which those 2 conditions : No inputs, One or More Output.

In practice this is not a big deal because transaction without input does not exist... But sometimes, during internal processing, you may serialize and deserialize for cloning your transaction for example. And I've encountered case in NBitcoin where it happened without input.

The way I dealt with it is by fitting a bit in the tx version which means "I really have no witness, don't read the next byte as flag".

I have not implemented that here.

schildbach · 2016-01-25T22:43:52Z

Thanks, that's great! I think this PR misses the code for protobuf-serialization of witnesses. So a wallet that saves to disk and back will miss the witnesses.

Also please watch the code style. Though I can fix that while merging.

schildbach · 2016-01-25T22:44:27Z

core/src/main/java/org/bitcoinj/core/Transaction.java

-            byte[] bits = bitcoinSerialize();
+            ByteArrayOutputStream stream = new UnsafeByteArrayOutputStream(length < 32 ? 32 : length + 32);
+            try{
+            bitcoinSerializeToStream(stream, TransactionOptions.NONE);


indentation

NicolasDorier · 2016-01-25T22:45:10Z

I tried to be careful with code style did I missed something ?
Where is handled protobuf serialization ? (I think it is better in a separate PR though)

schildbach · 2016-01-25T22:45:21Z

core/src/main/java/org/bitcoinj/core/Transaction.java

+    }
+
+    private void readWitness()
+    {


'{' at end of line

NicolasDorier · 2016-01-25T22:49:42Z

Fixing formatting, I did not knew how to do it on android studio :p

NicolasDorier · 2016-01-25T23:19:07Z

@schildbach ok I reformatted, some other things has been catched up that were not properly formatted.

NicolasDorier · 2016-01-26T04:45:09Z

@schildbach Actually, there is a second way of doing, which I did with NBitcoin.

In this PR, each transaction have "outputs", "inputs" and "witnesses" where witnesses is either null OR of exactly the size of inputs. It matches the serialization model on a bitcoin transaction. (witnesses are not serialized in the "TxInput").

In NBitcoin I initially took this approach, but found it bothersome to develop against. So I changed and added an accessor "Witness" into the "TxInput" class. This does not match the serialization model, however, it is conceptually more correct.

schildbach · 2016-01-26T09:21:42Z

Protobuf serialization is done in the WalletProtobufSerializer class and the structure will be extended in wallet.proto. Don't forget to generate fresh bindings – see comment in wallet.proto. It's fine to do it in a separate commit. Separate PR feels weird though, at least it depends on your changes already done.

About the witnesses structure I have no preference right now. Usually it makes sense to look at how Bitcoin Core does things. Also, maybe it makes the decision easier if we anticipate there can be multiple witnesses in future.

If you use a formatter, please make sure to format only your code. Correcting already existing format problems should go into a different PR (don't bother about it now). It may sound a bit finicky, but keeping changes to the minimum helps with reviewing and backporting your code. If you're unsure with formatting, I can do it at merge time.

schildbach · 2016-01-26T09:23:24Z

core/src/main/java/org/bitcoinj/core/TransactionWitness.java

+
+public class TransactionWitness {
+    public TransactionWitness(int pushCount)
+    {


'{' to the EOL please

NicolasDorier · 2016-01-26T11:29:09Z

Ok now it is perfectly formatted, I'll be more careful next commit.
I am wondering if serialization of the witness is really useful for the wallet. The advantage of segwit is precisely that you don't need to store the witness part. Do you prefer I do it still ?

There is other things to do for segwit, one of which is the new hash function for signature, do I do that in the same PR as well ?

schildbach · 2016-01-26T18:00:39Z

core/src/test/java/org/bitcoinj/core/TransactionTest.java

@@ -278,6 +278,33 @@ public void testCLTVPaymentChannelTransactionRefund() {
    }

    @Test
+    public void canParseTransaction() {


Maybe rename this test to something more meaningful, e.g. witnessTransaction()?

schildbach · 2016-01-26T18:06:37Z

The formatting looks very good now. Thanks for making the effort.

You're right about witnesses not needed by the wallet (unless there will be future witnesses that are useful), so yes let's skip protobuf persistence for now.

Yes if you have further work for segwit feel free to put a commit on top. Ideally we'd work towards a usable feature (like those I pictured in my "rough roadmap" posted to the mailing list) and if one is finished, and the BIPs are finished/committed to by Bitcoin Core, we merge.

NicolasDorier · 2016-01-26T18:38:23Z

Can you advise also what you would prefer: Currently if you serialize then deserialize a transaction with no input BUT with output, it will fail, due to how segwit format has been designed.
In practice this is not a problem, a transaction without input does not really exist. But in some internal processing with NBitcoin I needed to clone the transaction, which I did by serializing/deserializing, and ran into problems. Also my tests broke because I often used input less transaction for expediency.

My fix to that was to put a flag in the version during serialization if those 3 conditions were true : Segwit supported, Input size is 0, Outputs size is above 0.
Do you want I do that here ?

schildbach · 2016-01-26T19:14:54Z

I prefer to do without that flag workaround. And you said this PR comes without – still it passes all the unit tests so I guess it's no problem for bitcoinj.

NicolasDorier · 2016-01-27T05:59:13Z

New hashSignature scheme implemented, I borrowed from @greenaddress greenaddress@891e349

NicolasDorier · 2016-01-28T09:27:47Z

@schildbach I think I finished all is needed for basic segwit support. To my mind there is still some nice to have method to make, like a way to go back and forth between Script and TransactionWitness class. (A TransactionWitness is basically a Script with only pushes encoded more efficiently)

Let me know if you would like something more or if you'd like any fix. (or if you have any question on segwit)

NicolasDorier · 2016-02-07T13:38:21Z

@schildbach it can be useful to merge that before it get merged to core so people can start testing on segnet. Let me know if you want I add something in this PR.

schildbach · 2016-02-08T11:00:58Z

I'd rather wait with merging to master until the spec is committed (will most likely not change any more).

I can merge it to a branch on our repo, but then again this is only useful if we have multiple developers working on segregated witness.

NicolasDorier · 2016-02-08T11:06:50Z

Ok no problem, I'll squash the commits as you said, and I will also add the segnet params. (I forgot about that)

NicolasDorier · 2016-02-08T12:02:38Z

Ok @schildbach segnet is now pushed and the commit you mentioned squashed. Ping @greenaddress I know you need that for your tests.

NicolasDorier · 2016-02-08T13:00:31Z

Added ref to the BIP, as well as the BIP's sample in tests

greenaddress · 2016-03-26T21:27:06Z

We tested on segnet3 this patch as our code doesn't use the rest.

what is the plan for this PR? I believe the spec is not changing anymore.

@NicolasDorier it probably needs rebase (and maybe squash) :)

schildbach · 2016-03-27T09:40:41Z

The plan is to let SegWit mature (as a whole, so also the work on Core). As soon as Core SegWit merged, I'd merge all parts of this PR that form a viable usecase.

Until then, if multiple people start contributing code to this PR I can merge it to a feature branch.

greenaddress · 2016-03-27T09:48:01Z

sounds good, thanks @schildbach!

NicolasDorier · 2016-03-27T13:33:33Z

rebased

coveralls · 2016-03-27T13:41:07Z

Changes Unknown when pulling 912d7e4 on NicolasDorier:can-parse-transaction-with-witness into * on bitcoinj:master*.

matsjj · 2016-05-18T12:26:13Z

Thanks a lot for starting to work on the heavy lifting required to utilise SegWit!

I started using this branch for (thunder)[https://github.com/blockchain/thunder/commit/e9132c9f30719f430385e88cc1c04d7611b5dfd0], and added a few utility methods in my code that probably rather belong over here (like constructing a TransactionWitness object out of a P2SH output and input).

I think a feature branch makes sense at this point

schildbach · 2016-05-18T13:14:32Z

Feature branch done: https://github.com/bitcoinj/bitcoinj/tree/segwit
I'm closing this PR as future work on segwit should go on new PRs against the segwit branch.

@matsjj Great to know you're working on thunder/lightning!

btcdrak · 2016-09-15T15:53:57Z

A note abotu segwit, the witness program size was extended to 40 bytes in BIP141 c.f. bitcoin/bips@d1b52cb

schildbach reviewed Jan 25, 2016
View reviewed changes

core/src/main/java/org/bitcoinj/core/Transaction.java

}

private void readWitness()

{

Copy link

Member

schildbach Jan 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'{' at end of line

NicolasDorier force-pushed the can-parse-transaction-with-witness branch 4 times, most recently from a32d924 to 4386a9d Compare January 25, 2016 23:17

schildbach reviewed Jan 26, 2016
View reviewed changes

NicolasDorier force-pushed the can-parse-transaction-with-witness branch 3 times, most recently from 9b2c6b1 to d12e856 Compare January 26, 2016 11:25

NicolasDorier force-pushed the can-parse-transaction-with-witness branch from d12e856 to 63217e6 Compare January 26, 2016 11:33

schildbach reviewed Jan 26, 2016
View reviewed changes

NicolasDorier force-pushed the can-parse-transaction-with-witness branch from 63217e6 to 7aa4ead Compare January 26, 2016 18:33

NicolasDorier force-pushed the can-parse-transaction-with-witness branch from dbf6b45 to 05fa6ce Compare January 27, 2016 05:57

NicolasDorier force-pushed the can-parse-transaction-with-witness branch 2 times, most recently from 5437939 to f708088 Compare January 28, 2016 09:21

NicolasDorier force-pushed the can-parse-transaction-with-witness branch from f708088 to 75169e2 Compare February 8, 2016 11:12

NicolasDorier force-pushed the can-parse-transaction-with-witness branch from 8ede6f7 to ea1596f Compare February 8, 2016 12:59

NicolasDorier added 3 commits March 27, 2016 13:27

Can serialize and deserialize transaction with witnesses

01773a9

Implement hashSignatureWitness

7637166

Add segnet params

912d7e4

NicolasDorier force-pushed the can-parse-transaction-with-witness branch from ea1596f to 912d7e4 Compare March 27, 2016 13:33

matsjj mentioned this pull request May 13, 2016

Use SegWit blockchain/thunder#10

Open

schildbach closed this May 18, 2016

btcdrak mentioned this pull request Oct 5, 2016

Outstanding issues for segwit integration #1301

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can serialize and deserialize transaction with witnesses #1171

Can serialize and deserialize transaction with witnesses #1171

NicolasDorier commented Jan 25, 2016

NicolasDorier commented Jan 25, 2016

schildbach commented Jan 25, 2016

schildbach Jan 25, 2016

NicolasDorier commented Jan 25, 2016

schildbach Jan 25, 2016

NicolasDorier commented Jan 25, 2016

NicolasDorier commented Jan 25, 2016

NicolasDorier commented Jan 26, 2016

schildbach commented Jan 26, 2016

schildbach Jan 26, 2016

NicolasDorier commented Jan 26, 2016

schildbach Jan 26, 2016

schildbach commented Jan 26, 2016

NicolasDorier commented Jan 26, 2016

schildbach commented Jan 26, 2016

NicolasDorier commented Jan 27, 2016

NicolasDorier commented Jan 28, 2016

NicolasDorier commented Feb 7, 2016

schildbach commented Feb 8, 2016

NicolasDorier commented Feb 8, 2016

NicolasDorier commented Feb 8, 2016

NicolasDorier commented Feb 8, 2016

greenaddress commented Mar 26, 2016

schildbach commented Mar 27, 2016

greenaddress commented Mar 27, 2016

NicolasDorier commented Mar 27, 2016

coveralls commented Mar 27, 2016

matsjj commented May 18, 2016

schildbach commented May 18, 2016

btcdrak commented Sep 15, 2016

Can serialize and deserialize transaction with witnesses #1171

Can serialize and deserialize transaction with witnesses #1171

Conversation

NicolasDorier commented Jan 25, 2016

NicolasDorier commented Jan 25, 2016

schildbach commented Jan 25, 2016

schildbach Jan 25, 2016

Choose a reason for hiding this comment

NicolasDorier commented Jan 25, 2016

schildbach Jan 25, 2016

Choose a reason for hiding this comment

NicolasDorier commented Jan 25, 2016

NicolasDorier commented Jan 25, 2016

NicolasDorier commented Jan 26, 2016

schildbach commented Jan 26, 2016

schildbach Jan 26, 2016

Choose a reason for hiding this comment

NicolasDorier commented Jan 26, 2016

schildbach Jan 26, 2016

Choose a reason for hiding this comment

schildbach commented Jan 26, 2016

NicolasDorier commented Jan 26, 2016

schildbach commented Jan 26, 2016

NicolasDorier commented Jan 27, 2016

NicolasDorier commented Jan 28, 2016

NicolasDorier commented Feb 7, 2016

schildbach commented Feb 8, 2016

NicolasDorier commented Feb 8, 2016

NicolasDorier commented Feb 8, 2016

NicolasDorier commented Feb 8, 2016

greenaddress commented Mar 26, 2016

schildbach commented Mar 27, 2016

greenaddress commented Mar 27, 2016

NicolasDorier commented Mar 27, 2016

coveralls commented Mar 27, 2016

matsjj commented May 18, 2016

schildbach commented May 18, 2016

btcdrak commented Sep 15, 2016