show us the encrypted data

[pro-wh]

alright, if you really want to look 🤷

no api

philosophy:

• collapse all callformat variants into a looser type with variable size nonce+data fields.
• same for public key, there's just a variable size slot for it.
• store the format alongside, in case anyone wants to make sense of the keys+nonces+data
• the "public key" in the tx is the one provided by the caller. I'm not aware of us capturing the runtime's public key.
• if anything goes wrong, log and leave it nil. the drawback is that NULL in the db could mean either plain or something failed during analysis (e.g. new callformat that we don't support)
• kinda oddly asymmetric, we don't store the data+result when it's unencrypted

[pro-wh]

🚂🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋🚋💨

[mitjat]

😆

I can think of a few ways around the runaway train:

1. Create a separate evm_transactions table, move the new fields (and probably also tx_eth_hash) in there. I'm a little worried about performance because we'll need probably at least tx_eth_hash and evm_encrypted_format (?) every time we're retrieving txs, even if en masse. Or we could denormalize slightly: we create an evm_encryption table, put just the new fields in it, and additionally store an is_encrypted flag in the main runtime_transactions table, because that's likely all we'll need for the non-detailed view. It's also nice because it generalizes to non-evm encryption (= Cipher).

2. Using a postgres composite type, i.e. a struct-typed column, to store the evm encryption info. You create them like so (first format), and use them from Go via pgtype.CompositeFields like so. The downside is that we're adding a little complexity to the DB interface/structure/usage.

3. Same as number 2, but with JSON instead of composite types. Pretty yuck and space-hungry. I prefer the train over this.

I started off favoring 2, but I'm now more in favor of the denormalized variant of 1.

Note on performance of 1 vs 2: Bulky table rows hurt performance; first in gradual ways (because of page size and disk caches), then at 8kB per row abruptly, because postgres stores rows over 8kB in a TOAST table, so every row lookup performs an implicit JOIN. This would imply 2 is faster, because we JOIN only explicitly, and only for single-tx results. However my understanding of TOAST is that only bulky columns are moved to the overflow (= TOAST) table, so with some luck, our composite type and the existing body type would be the ones to get moved, which should result in about the same performance as 1 if we only SELECT those columns for single-tx results.

[pro-wh]

too scary to do in this PR

[pro-wh]

#414

[mitjat]

Thank you for figuring out the deserialization incantations!

[mitjat]

😆

I can think of a few ways around the runaway train:

1. Create a separate evm_transactions table, move the new fields (and probably also tx_eth_hash) in there. I'm a little worried about performance because we'll need probably at least tx_eth_hash and evm_encrypted_format (?) every time we're retrieving txs, even if en masse. Or we could denormalize slightly: we create an evm_encryption table, put just the new fields in it, and additionally store an is_encrypted flag in the main runtime_transactions table, because that's likely all we'll need for the non-detailed view. It's also nice because it generalizes to non-evm encryption (= Cipher).

2. Using a postgres composite type, i.e. a struct-typed column, to store the evm encryption info. You create them like so (first format), and use them from Go via pgtype.CompositeFields like so. The downside is that we're adding a little complexity to the DB interface/structure/usage.

3. Same as number 2, but with JSON instead of composite types. Pretty yuck and space-hungry. I prefer the train over this.

I started off favoring 2, but I'm now more in favor of the denormalized variant of 1.

Note on performance of 1 vs 2: Bulky table rows hurt performance; first in gradual ways (because of page size and disk caches), then at 8kB per row abruptly, because postgres stores rows over 8kB in a TOAST table, so every row lookup performs an implicit JOIN. This would imply 2 is faster, because we JOIN only explicitly, and only for single-tx results. However my understanding of TOAST is that only bulky columns are moved to the overflow (= TOAST) table, so with some luck, our composite type and the existing body type would be the ones to get moved, which should result in about the same performance as 1 if we only SELECT those columns for single-tx results.

[Andrew7234]

are we able to decode the format/nonces/publickey into a more concrete type here? Or can it vary?

[pro-wh]

format is CallFormat from the sdk. it's an enum

in x25519 (callformat 1), the public key and nonce are specific length byte arrays, which I'm sure the cryptosystem is happy to keep opaque on the outside