Transaction.from_cbor is not faithful to the CBOR encoding

[nielstron]

Describe the bug
Decoding the following transaction from cbor will result in a transaction with a different CBOR. Crucially, the cbor encoding of the attached datum changes, modifying its datum hash and thus messing up the entire transaction when attempting to simulate it locally.

# repro.py
from pycardano import Transaction

tx_cbor_hex = "84a400d9010281825820cd072cd8be6f9f62ac4c09c28206e7e35594aa6b342f5d0a3a5e4842fab428f700018283581d709e4df79b06d035f7c9d1202bee9d25c6d832392af8c8a48c088442051a02faf08058202345b6d0e4660db69577dbc7cf7e673378a07ab3aef063244b6d57c44d48cbb982581d60e9f522e70a3504e66214147ddfedc49a55b73b55b812e7585739ed321a02f85537021a00029b490b5820edea6e5a0a31f0106762126c7f01a50413e1cdc9e7cbece91867e6f3e959d7b3a200d9010281825820fcd7805ba56b1646451e77a32dc13556da40f52a8959aa713e13f5ff37f5d55c5840a82e8c118742bad8167d2e6be968ec89c4ef8d5c6edb37af56272d10c2268b2632d9c78c70bb3d67f614460a1aa0eb7ede19f9809d2388a98a291e4a4f39c20304d9010281d8799f581c4e2bdc18071de73908448834d99b51959390031cbed8626f0d91ff871b0000016f5e2ff980fff5f6"
tx = Transaction.from_cbor(bytes.fromhex(tx_cbor_hex))
assert tx.to_cbor_hex() == tx_cbor_hex, "Got {}".format(tx_cbor_hex, tx.to_cbor_hex())

$ python3 repro.py
Traceback (most recent call last):
  File "/Users/niels/git/mockfrost/repro.py", line 5, in <module>
    assert tx.to_cbor_hex() == tx_cbor_hex, "Got {}".format(tx.to_cbor_hex())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Got 84a400d9010281825820cd072cd8be6f9f62ac4c09c28206e7e35594aa6b342f5d0a3a5e4842fab428f700018283581d709e4df79b06d035f7c9d1202bee9d25c6d832392af8c8a48c088442051a02faf08058202345b6d0e4660db69577dbc7cf7e673378a07ab3aef063244b6d57c44d48cbb982581d60e9f522e70a3504e66214147ddfedc49a55b73b55b812e7585739ed321a02f85537021a00029b490b5820edea6e5a0a31f0106762126c7f01a50413e1cdc9e7cbece91867e6f3e959d7b3a200d9010281825820fcd7805ba56b1646451e77a32dc13556da40f52a8959aa713e13f5ff37f5d55c5840a82e8c118742bad8167d2e6be968ec89c4ef8d5c6edb37af56272d10c2268b2632d9c78c70bb3d67f614460a1aa0eb7ede19f9809d2388a98a291e4a4f39c203049fd87982581c4e2bdc18071de73908448834d99b51959390031cbed8626f0d91ff871b0000016f5e2ff980fff5f6

zoom in on the datums

      04                                           #     unsigned(4)
      9f                                           #     array(*)
         d8 79                                     #       tag(121)
            82                                     #         array(2)
               58 1c                               #           bytes(28)
                  4e2bdc18071de73908448834d99b5195 #             "N+\xdc\x18\x07\x1d\xe79\x08D\x884\xd9\x9bQ\x95"
                  9390031cbed8626f0d91ff87         #             "\x93\x90\x03\x1c\xbe\xd8bo\r\x91\xff\x87"
               1b 0000016f5e2ff980                 #           unsigned(1,577,833,200,000)
         ff                                        #       break

vs

      04                                              #     unsigned(4)
      d9 0102                                         #     tag(258)
         81                                           #       array(1)
            d8 79                                     #         tag(121)
               9f                                     #           array(*)
                  58 1c                               #             bytes(28)
                     4e2bdc18071de73908448834d99b5195 #               "N+\xdc\x18\x07\x1d\xe79\x08D\x884\xd9\x9bQ\x95"
                     9390031cbed8626f0d91ff87         #               "\x93\x90\x03\x1c\xbe\xd8bo\r\x91\xff\x87"
                  1b 0000016f5e2ff980                 #             unsigned(1,577,833,200,000)

The underlying datum is a quite straightforward

@dataclass()
class VestingDatum(pycardano.PlutusData):
    CONSTR_ID = 0
    beneficiary: bytes
    deadline: int

To Reproduce
python3 repro.py

Expected behavior
The transaction cbor should be roundtrip correct.

Environment and software version (please complete the following information):

• OS: MacOS
• PyCardano Version: commit 95e5b02

Additional context
Notably, this cbor hex was generated in Mockfrost using pycardano.

[nielstron]

Seems like the datum fields are variably parsed as Infinitelist or Finite list. The issue originates already in cbor2.loads(cbor) which unrolls the cbor into an incorrect representation

[cffls]

Have you followed the instruction to disable C bindings here?

The cbor matches in my env without C bindings:

from pycardano import Transaction
tx_cbor_hex = "84a400d9010281825820cd072cd8be6f9f62ac4c09c28206e7e35594aa6b342f5d0a3a5e4842fab428f700018283581d709e4df79b06d035f7c9d1202bee9d25c6d832392af8c8a48c088442051a02faf08058202345b6d0e4660db69577dbc7cf7e673378a07ab3aef063244b6d57c44d48cbb982581d60e9f522e70a3504e66214147ddfedc49a55b73b55b812e7585739ed321a02f85537021a00029b490b5820edea6e5a0a31f0106762126c7f01a50413e1cdc9e7cbece91867e6f3e959d7b3a200d9010281825820fcd7805ba56b1646451e77a32dc13556da40f52a8959aa713e13f5ff37f5d55c5840a82e8c118742bad8167d2e6be968ec89c4ef8d5c6edb37af56272d10c2268b2632d9c78c70bb3d67f614460a1aa0eb7ede19f9809d2388a98a291e4a4f39c20304d9010281d8799f581c4e2bdc18071de73908448834d99b51959390031cbed8626f0d91ff871b0000016f5e2ff980fff5f6"
tx = Transaction.from_cbor(bytes.fromhex(tx_cbor_hex))
print(tx.to_cbor_hex() == tx_cbor_hex)
True

[nielstron]

Yep, that fixed it, thanks! Maybe we should add it to a common issues section / add a common issues section to the docs? Could also include "how to burn assets" etc

[cffls]

Yea makes sense to have a section like that. Will add it.