Tighten type support and fix typing issues

[cburgdorf]

What was wrong?

Currently, any method that expects any of BaseHeaderAPI, BaseBlockAPI, UnsignedTransactionAPI, ReceiptAPI does not properly enforce type safe. In practice that means that a method such as:

def do_something_with_block(block: BlockAPI):`
   ...

Can be called as do_something_with_block(accidential_is_transaction) or do_something_with_block(accidential_is_header) and mypy does not complain about it.

This has allowed serious bugs to fly under the radar: ethereum/trinity#1830

The root cause for this is that anything derived from rlp.Serializable will be seen as Any by mypy. We currently do not enforce this to be invalid but if so, mypy would rightfully reject a series of types:

$ mypy -p eth --config-file mypy.ini 
eth/vm/interrupt.py:28: error: Class cannot subclass 'MissingTrieNode' (has type 'Any')
eth/vm/interrupt.py:54: error: Class cannot subclass 'MissingTrieNode' (has type 'Any')
eth/rlp/transactions.py:55: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/rlp/transactions.py:102: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/rlp/logs.py:20: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/rlp/headers.py:48: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/rlp/headers.py:66: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/rlp/accounts.py:20: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/rlp/receipts.py:23: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/rlp/blocks.py:19: error: Class cannot subclass 'Serializable' (has type 'Any')
eth/db/chain.py:77: error: Class cannot subclass 'Serializable' (has type 'Any')
Found 11 errors in 8 files (checked 228 source files)

How was it fixed?

I believe the best fix would be to do a serious overhaul of py-rlp to enforce strong type support. As I tried to add type support to py-rlp in a minimalistic fashion I noticed another way of approaching this which gets us halfway there without even touching py-rlp (and hence being faster to deliver).

Basically, what this does is rewrite all eth.abc types to not derive from rlp.Serializable and move the rlp.Serializable a level deeper. That means that things such as BaseBlock will still be seen as Any (just like before) but at least we have full type safety at the top level e.g. BlockAPI.

This does already a great job at exposing tons of bugs without opening a huge time sink such as refactoring py-rlp.

To-Do

• Clean up commit history

• Add entry to the release notes

Cute Animal Picture

[carver]

Why not just 'MiningHeaderAPI', so we can drop THeader?

[cburgdorf]

Because BlockHeaderAPI inherits this API and so we don't want to hardcode the return type to MiningHeaderAPI.

[cburgdorf]

But actually, we don't really need it to be on MiningHeaderAPI and can move it down to BlockHeaderAPI and then hardcode it. This makes the usage slightly nicer because otherwise mypy is asking us for a type annotation at every place where we use copy like so:

unused_header: BlockHeaderAPI = header.copy(gas_used=0)

If I move it down to BlockHeaderAPI we can simply do:

unused_header = header.copy(gas_used=0)

[carver]

Maybe add a quick comment on each of the methods that are copied over from rlp.Serializeable, saying that they can be deleted from this API after pyrlp properly exposes the types? (I think just these last three methods)

Edit: and the methods in the other classes, of course.

[cburgdorf]

Will do

[cburgdorf]

I added comments on all methods. Btw, I've also tried grouping all of them in a RLPSerializableAPI(Generic[T]) so that we don't have to manually add all these methods to all these different types and while this worked great for >= py37 it yielded an obscure meta class MRO error that I couldn't resolve.

[carver]

Ah, hooray for bugfixes. :) I guess worthy of a news fragment, and surprising and problematic that there's no test coverage on this. Maybe add a test that covers it (or at least an issue to do it as part of beta)? Looks like adding uncles was totally broken. (Is this overwritten in different forks, and that's why we didn't notice?)

[cburgdorf]

Good question. I have to investigate that once more tomorrow.

[cburgdorf]

Ok, the reason seems to be that this is essentially dead code. At least there's no code in Py-EVM or Trinity calling add_uncle. Since this is a public API there's a non-zero chance it is used at another place that I might not be aware of but it seems unlikely. Anyway, I'm not dropping it pre-merge in case I'm overseeing something but I'm happy to send another PR to remove it.

[carver]

Since this is a public API there's a non-zero chance it is used at another place that I might not be aware of but it seems unlikely.

Yeah, very unlikely, since every call to it would immediately fail 😆

I think we can safely remove an API that has been broken for 3 years:

3b0bcc219 eth/vm/forks/frontier/blocks.py   (2019-08-01 11:43:24 -0600 138)     def add_uncle(self, uncle: BlockHeaderAPI) -> "FrontierBlock":
26173f6ed evm/vm/flavors/frontier/blocks.py (2017-06-28 14:50:03 -0600 139)         self.uncles.append(uncle)
26173f6ed evm/vm/flavors/frontier/blocks.py (2017-06-28 14:50:03 -0600 140)         self.header.uncles_hash = keccak(rlp.encode(self.uncles))
26173f6ed evm/vm/flavors/frontier/blocks.py (2017-06-28 14:50:03 -0600 141)         return self

I'm happy to send another PR to remove it.

👍

[carver]

Wait, now I'm confused again. The init method adds uncles as []. So I guess maybe it was working before, and we have to think a little more carefully. It's a mess that it's created as a tuple in one place and a list in another.