Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RawTransactions Serialized in Canonical Format #454

Closed
davidiw opened this issue Aug 6, 2019 · 1 comment

Comments

@davidiw
Copy link
Contributor

commented Aug 6, 2019

We are migrating RawTransactions from protobuf to CanonicalSerialization.

This is a breaking change for all clients!

background

Historically, Libra validators had to keep a copy of the client's originally serialized format of a RawTransaction and share those with other validators in order to validate the legitimacy of the transaction. The premise for this was that RawTransactions are serialized via protobufs in a non-deterministic fashion, i.e., it is entirely possible that two different implementations potentially programmed in two different languages or two different platforms would construct a different byte array (serialized version) for the exact same transaction.

Unfortunately, there are no popular deterministic / canonical serialization suites in mass use supported by multiple platforms. Most Blockchains implement their own form of specialized serialization for handling these type of serialization needs, for example, Ethereum uses RLP.

While not strictly required, the lack of deterministic serialization would severely cripple the blockchain as different validators may or may not be able to verify each other's signatures on the blackchain hashes. To address this for Testnet, we constructed the CanonicalSerializer so at least the validators would be able to communicate in a sensible fashion.

The Work

To address the final issue of RawTransactions, we have:

  • Migrated RawTransactions to our CanonicalSerializer, PR 454 - Merged
  • Published a spec on Canonical Serialization so the community can implement it in their clients, PR 622 - Pending review
  • Use CanonicallySerialized RawTransactions in place of their protobuf counterpart, PR 606 - Pending review

The Deployment Plan

Assuming there are no blocking issues with the above PRs. The current plan is to migrate testnet to this new format by next Wednesday (8/28).

Making a Client App Compatible

The specification for RawTransaction can be found at the bottom CanonicalSerialization. An implementation of the CanonicalSerializer can be found here. An example of the RawTransaction data being serialized can be found here.

@davidiw davidiw added the enhancement label Aug 6, 2019
@davidiw davidiw self-assigned this Aug 6, 2019
davidiw added a commit to davidiw/libra that referenced this issue Aug 6, 2019
Adding support for tuple serialization support for CanonicalSerializer
as a precursor to support for RawTransaction.

Unable to use macros due to lack of support within traits.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 6, 2019
Adding support for tuple serialization support for CanonicalSerializer
as a precursor to support for RawTransaction.

Unable to use macros due to lack of support within traits.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 6, 2019
Adding support for tuple serialization support for CanonicalSerializer
as a precursor to support for RawTransaction.

Unable to use macros due to lack of support within traits.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 6, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 6, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 7, 2019
Per the title, this is a precursor to support RawTransaction.

In this diff, we reuse the TransactionArgument_ArgType, this will likely
be deleted from the code when we know longer need protobuf support for
Transactions. We will still be able to maintain consistency by matching
the behavior with new code once that happens. For now, I did not want to
litter the code base with duplicate code.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 7, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 7, 2019
Per the title, this is a precursor to support RawTransaction.

In this diff, we reuse the TransactionArgument_ArgType, this will likely
be deleted from the code when we know longer need protobuf support for
Transactions. We will still be able to maintain consistency by matching
the behavior with new code once that happens. For now, I did not want to
litter the code base with duplicate code.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 7, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 7, 2019
Per the title, this is the last diff in the series that enables
canonical serialization. In the ensuing diffs, we will replace the
protobuf serialization with the new form as it is used throughout the
code base and across the wire.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 13, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 13, 2019
Per the title, this is the last diff in the series that enables
canonical serialization. In the ensuing diffs, we will replace the
protobuf serialization with the new form as it is used throughout the
code base and across the wire.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 15, 2019
Adding support for tuple serialization support for CanonicalSerializer
as a precursor to support for RawTransaction.

Unable to use macros due to lack of support within traits.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 15, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 15, 2019
Per the title, this is a precursor to support RawTransaction.

In this diff, we reuse the TransactionArgument_ArgType, this will likely
be deleted from the code when we know longer need protobuf support for
Transactions. We will still be able to maintain consistency by matching
the behavior with new code once that happens. For now, I did not want to
litter the code base with duplicate code.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 15, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 15, 2019
Per the title, this is the last diff in the series that enables
canonical serialization. In the ensuing diffs, we will replace the
protobuf serialization with the new form as it is used throughout the
code base and across the wire.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 16, 2019
Adding support for tuple serialization support for CanonicalSerializer
as a precursor to support for RawTransaction.

Unable to use macros due to lack of support within traits.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 16, 2019
Adding support for string serialization for CanonicalSerializer as a precusor for supporting RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 16, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 16, 2019
Per the title, this is a precursor to support RawTransaction.

In this diff, we reuse the TransactionArgument_ArgType, this will likely
be deleted from the code when we know longer need protobuf support for
Transactions. We will still be able to maintain consistency by matching
the behavior with new code once that happens. For now, I did not want to
litter the code base with duplicate code.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 16, 2019
Per the title, this is a precursor to support RawTransaction.

refs libra#454
davidiw added a commit to davidiw/libra that referenced this issue Aug 16, 2019
Per the title, this is the last diff in the series that enables
canonical serialization. In the ensuing diffs, we will replace the
protobuf serialization with the new form as it is used throughout the
code base and across the wire.

refs libra#454
davidiw added a commit that referenced this issue Aug 16, 2019
Adding support for tuple serialization support for CanonicalSerializer
as a precursor to support for RawTransaction.

Unable to use macros due to lack of support within traits.

refs #454
davidiw added a commit that referenced this issue Aug 16, 2019
Adding support for string serialization for CanonicalSerializer as a precusor for supporting RawTransaction.

refs #454
davidiw added a commit that referenced this issue Aug 16, 2019
Per the title, this is a precursor to support RawTransaction.

refs #454
davidiw added a commit that referenced this issue Aug 16, 2019
Per the title, this is a precursor to support RawTransaction.

In this diff, we reuse the TransactionArgument_ArgType, this will likely
be deleted from the code when we know longer need protobuf support for
Transactions. We will still be able to maintain consistency by matching
the behavior with new code once that happens. For now, I did not want to
litter the code base with duplicate code.

refs #454
davidiw added a commit that referenced this issue Aug 16, 2019
Per the title, this is a precursor to support RawTransaction.

refs #454
davidiw added a commit that referenced this issue Aug 16, 2019
Per the title, this is the last diff in the series that enables
canonical serialization. In the ensuing diffs, we will replace the
protobuf serialization with the new form as it is used throughout the
code base and across the wire.

refs #454
@davidiw davidiw changed the title [Feature Request] Leverage Canonical Serialization for All Transactions Leverage Canonical Serialization for All Transactions Aug 20, 2019
@davidiw davidiw changed the title Leverage Canonical Serialization for All Transactions Leverage Canonical Serialization for RawTransactions Aug 20, 2019
@davidiw davidiw changed the title Leverage Canonical Serialization for RawTransactions RawTransactions Serialized in Canonical Format Aug 20, 2019
@davidiw

This comment has been minimized.

Copy link
Contributor Author

commented Aug 30, 2019

Completed.

@davidiw davidiw closed this Aug 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.