Implement EIP1559 Fee Market + EIP3198 BaseFee #1148

jochem-brouwer · 2021-03-14T21:52:52Z

This PR implements EIP 1559.

codecov · 2021-03-14T21:54:39Z

Codecov Report

Merging #1148 (99cb89f) into master (4d3edaa) will decrease coverage by 1.95%.
The diff coverage is 83.20%.

Flag	Coverage Δ
block	`82.51% <75.36%> (-0.42%)`	⬇️
blockchain	`84.36% <93.10%> (+0.19%)`	⬆️
client	`84.70% <75.00%> (+0.11%)`	⬆️
common	`88.12% <100.00%> (+0.04%)`	⬆️
devp2p	`84.21% <66.66%> (-0.07%)`	⬇️
ethash	`82.08% <ø> (ø)`
tx	`88.74% <88.17%> (-0.09%)`	⬇️
vm	`80.87% <71.11%> (-5.90%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

holgerd77 · 2021-03-15T09:38:00Z

@jochem-brouwer cool, great that you gave this a start! 😄 👍 🎉

Yeah, it will very much make sense to further abstract here. Let's please give it a cut here, maybe you've got some time during the day and jump on a quick call (let me know when in chat)?

It actually get's hyper-tricky here. I guess the main problem is that the concept of inheritance just doesn't work well with this very abstract typed transaction EIP concept where a tx - theoretically - could contain anything. This might work for now - also e.g. regarding the question what is currently contained in BaseTransaction, but this is rather out of "randomness", since the feature set is just for this current set of typed transactions randomly fitting.

But one can already see where this leads to: in EIP-1559 there is e.g. no gasPrice and -whop - the whole BaseTransaction gets somewhat invalid from an API PoV. If there will be another tx type - e.g. with a new signature theme removing the v,r,s values - whop - base transaction respectively the whole inheritance chain will break again.

Not sure if this overloads us for now: but I at least theoretically wonder if e.g. some kind of module concept (gas module, signature module, access list module, eventually serialization module) might serve us better here and will be more future proof.

And a base transaction (or however it's called) might contain the information what kind of modules are activated, like so:

class BaseTransaction {
 
  modules: {
    gas: 'EIP1559',
    accessLists: true,
    signature: 'ECDSA', // future
    serialization: 'RLP',
  }

Something like this would also allow us to cleaner react in third-party code (VM, Block,...) to features available. The current approach will otherwise further down the road lead to a conditional hell like either always uncleanly just request for some thought-to-be-active properties (if (tx.gasPrice) { // do something... }) or (not better) adding to an ever growing chain of tx types (if (typeof tx in [AccessListTx, EIP1559Tx,... ]). All not really scaling.

Again, let's just jump on the call sometime. If we do this later the day eventually the others want to join as well.

holgerd77 · 2021-03-19T13:39:45Z

For reference: Base fee tests in Besu

holgerd77 · 2021-03-19T13:47:27Z

Nethermind EIP-1559 implementation: NethermindEth/nethermind#2341

jochem-brouwer · 2021-03-21T19:40:58Z

Rebased this one

jochem-brouwer · 2021-03-21T21:19:15Z

Yeah this design is getting rather complex already, I can fix this build error but the idea I got in mind might not be a really great design - will take a look at the other implementations first to get some inspiration here.

holgerd77 · 2021-03-22T11:46:03Z

@jochem-brouwer yeah, this is really tricky to answer and what level we want/need to refactor the whole library for this. In doubt just put this aside for now for a couple of days as some suggestion 😋 , sometimes this helps as well and we can pick this question up on our call and discuss together. If you would have alternatively some time for #1132 that would e.g. also be cool if we get this going a bit again. 😄

jochem-brouwer · 2021-03-27T14:14:44Z

Okay, added some very basic VM tests, but at this point I cannot continue until I have more test data. I will probably create some very simple EIP1559 tests to just see that it produces something (so I can sign a transaction for instance), but I cannot be sure if we are correct.

jochem-brouwer · 2021-04-25T13:31:50Z

Rebased upon master.

jochem-brouwer · 2021-04-25T14:07:50Z

Ok great, tests pass, now trying to connect to Aleut.

jochem-brouwer · 2021-04-25T17:28:38Z

We are only downloading 1 block at a time since for some reason the bootnode either sends invalid data, or we have a bug in devp2p. It tries to decode RLP message bodies where RLP states it should decode ~33k bytes, but the block is actually only 1023 bytes long. (The message body itself is 1024 bytes).

jochem-brouwer · 2021-04-26T20:52:55Z

Ok, this commit 8c112f0 fixes the consensus bug at block 55. The problem had nothing to do with EIP1559, it had to do with runBlock/runTx in VM: runBlock originally fed runTx an edited block than the block which is actually being ran; the gasUsed field is changed and used to keep track of the cumulative gas used. This commit is a hotfix and might be changed in the future (probably removing editing the block and just passing along the block which is actually being ran in the context).

We should add a test to ensure that clique blocks with txs pay the right miner, which is not tested in VM currently.

There also still is a problem with the Nethermind bootnode. If we request more than 1 block (might be ok to request 2 or 3) - lets say the original 50, then we get a RLP decode error on devp2p.

With these fixes we can sync up to block 3704:

ERROR [04-26|22:49:16] Error: invalid transactions: errors at tx 0: tx unable to pay base fee (block number=3704 hash=ee3a234b297c792ffbecc5f90042f053a14bb6760ec4058e61bca3f20d88c292)     at BlockHeader._error (/Volumes/Ethereum/ethereumjs-monorepo/packages/block/src/header.ts:857:15)
    at Block.validateData (/Volumes/Ethereum/ethereumjs-monorepo/packages/block/src/block.ts:272:25)
    at Block.validate (/Volumes/Ethereum/ethereumjs-monorepo/packages/block/src/block.ts:258:16)
    at /Volumes/Ethereum/ethereumjs-monorepo/packages/blockchain/src/index.ts:883:9
    at Blockchain.runWithLock (/Volumes/Ethereum/ethereumjs-monorepo/packages/blockchain/src/index.ts:410:21)
    at Blockchain._putBlockOrHeader (/Volumes/Ethereum/ethereumjs-monorepo/packages/blockchain/src/index.ts:854:5)
    at Blockchain.putBlock (/Volumes/Ethereum/ethereumjs-monorepo/packages/blockchain/src/index.ts:809:5)
    at Blockchain.putBlocks (/Volumes/Ethereum/ethereumjs-monorepo/packages/blockchain/src/index.ts:795:7)
    at Chain.putBlocks (/Volumes/Ethereum/ethereumjs-monorepo/packages/client/lib/blockchain/chain.ts:274:5)
    at BlockFetcher.store (/Volumes/Ethereum/ethereumjs-monorepo/packages/client/lib/sync/fetcher/blockfetcher.ts:84:5)

jochem-brouwer · 2021-04-26T21:12:59Z

Another note; we only connect to Nethermind, other 3 bootnodes are up, have to figure out what is the cause of this. http://eip1559-testnet.ops.pegasys.tech:3001/

packages/block/src/block.ts

packages/block/src/header.ts

holgerd77 · 2021-04-27T08:13:28Z

"Another node; we only connect to Nethermind"

You're too much in @jochem-brouwer. 😂 I've taken the freedom to correct. 😄

holgerd77 · 2021-04-27T08:46:52Z

packages/client/lib/blockchain/chain.ts

@@ -110,7 +110,7 @@ export class Chain extends EventEmitter {
        db: options.chainDB,
        common: this.config.chainCommon,
        validateBlocks: true,
-        validateConsensus: false,
+        validateConsensus: true,


This is putting too much load on the PoW sync, I guess what we can do is just to activate for PoA by including a conditional switch along the common instance setting from above.

holgerd77 · 2021-04-27T08:50:47Z

@ryanio hmm, still not sure if #1222 is fixing the problem, I did a rebase here locally and removed 42d83d0 (so mainly the constant bootstrap retriggering, setting maxPerRequest to 1 is not relevant yet in this context here) - also did another npm run bootstrap locally from within this branch to make sure - and yet again the Send BlockHeaders message is not to be seen. 😕

Phew, tricky thing.

ryanio · 2021-04-27T19:02:58Z

@holgerd77 this is tricky indeed, I rebased this branch on master (didn't touch the commit you specified) and was able to recreate the issue again.

I have a feeling it's still some kind of promise race condition we haven't handled properly. As a test I added a 5s timeout after starting the servers but before calling bootstrap (packages/client/lib/client.ts):

    ...
    await Promise.all(this.config.servers.map((s) => s.start()))

    // wait 5s
    function timeout(ms: number) {
      return new Promise(resolve => setTimeout(resolve, ms));
    }
    await timeout(5000)

    await Promise.all(this.config.servers.map((s) => s.bootstrap()))
    ...

and that worked for me! I got a Send GET_BLOCK_HEADERS line...so I will keep tracing this async code and see if there is something we are missing or not being awaited properly. I just tried again with 1s and that even worked. I also tried with setImmediate with no timeout period but that didn't work.

We have the @typescript-eslint/no-floating-promises rule turned off in packages/devp2p/.eslintrc.js so I have a hunch that it may be some floating promise causing the problem.

… call

…e-guarding with a new hardforkByHeadBlockNumber option (default: false)

…e test runner

… protocol is available and bound

…adBlockNumber option

holgerd77 · 2021-05-19T10:42:10Z

Guys, I will now merge this in here, tests are passing and I've given all code parts another final look. I have also written extensive tests for the tx library to lift the coverage up again to a decent level and have given all changes in the various libraries at least a rough and often somewhat thorough review.

We also do not have an imminently urgent release, at the same time more and more is worked "around" this one huge PR and all the other smaller work is blocked to a greater extend.

Everyone can still do reviews on parts where he feels the need, then please integrate your change requests along follow-up PRs.

Hope you can go along, but I have once again the feeling that we are at some point where things are not getting better by pushing this ahead. 😋 😀

Thanks @jochem-brouwer for this great work here! 🎉

holgerd77

LGTM

holgerd77 · 2021-05-19T10:44:48Z

Phew, 86 commits (and all very much decent ones, and not "fixed linting", "minor changes", "forgot to add x" 😛 ).

packages/tx/src/types.ts

packages/tx/src/eip1559Transaction.ts

jochem-brouwer · 2021-05-22T18:45:05Z

packages/tx/typedoc.js

@@ -4,7 +4,10 @@ module.exports = {
  plugin: 'typedoc-plugin-markdown',
  readme: 'none',
  gitRevision: 'master',
-  exclude: 'test/**/*.ts',
+  exclude: [
+    'src/util.ts',


Why do we exclude this?

The API of this feels too unfinished to me and I would like to keep this internal for now and give this some more thought and have some more discussions around it. I think no one every really reviewed these code parts (including me) and this would expose a huge new API surface.

Some things I would like to discuss (but which doesn't need to be discussed now, just for the mention):

Is this the best way to expose this functionality or might there be better - eventually more integrated - solutions (e.g. to re-bind the methods to the affected tx classes, just as some first idea)?

Relationship of this getAccessListData() (returning AccessListJSON as one of two items) and the getAccessListJSON() method

Should getDataFeeEIP2930() be exposed (I have the base feeling that methods which need a Common as an input often point to some suboptimality in design - we had these cases in the past - can't really point towards a concrete solution thought atm yet)?

I will actually open this as an issue, otherwise it will likely be too tempting to start the discussion here. 😄

packages/block/test/eip1559block.spec.ts

packages/client/lib/blockchain/chain.ts

jochem-brouwer added PR state: WIP package: block package: tx package: vm package: common labels Mar 14, 2021

holgerd77 mentioned this pull request Mar 19, 2021

EIP-1559: Changes specification to functional Python code ethereum/EIPs#2859

Merged

jochem-brouwer force-pushed the eip1559 branch from f071982 to ec78daf Compare March 21, 2021 19:40

holgerd77 force-pushed the eip1559 branch from be9b4f8 to 09b5658 Compare April 23, 2021 09:07

jochem-brouwer force-pushed the eip1559 branch 2 times, most recently from 6af1377 to 52cf035 Compare April 25, 2021 13:29

ryanio reviewed Apr 27, 2021

View reviewed changes

packages/block/src/block.ts Outdated Show resolved Hide resolved

ryanio reviewed Apr 27, 2021

View reviewed changes

packages/block/src/header.ts Outdated Show resolved Hide resolved

ryanio reviewed Apr 27, 2021

View reviewed changes

packages/block/src/header.ts Outdated Show resolved Hide resolved

holgerd77 reviewed Apr 27, 2021

View reviewed changes

client: added simple tx send mechanism for eth_sendRawTransaction RPC…

86b9303

… call

holgerd77 force-pushed the eip1559 branch from 5997f8f to 86b9303 Compare May 18, 2021 15:55

holgerd77 added 7 commits May 19, 2021 09:32

tx, vm, block: renamed maxInclusionFeePerGas -> maxPriorityFeePerGas

37c3d55

monorepo: updated package-lock.json

b03124d

monorepo: fixed Prettier linting by re-linting with Prettier v2.3.0

5de7a2f

vm, blockchain: avoid HF setting VM side effects in blockchain by saf…

e71bb60

…e-guarding with a new hardforkByHeadBlockNumber option (default: false)

client: renamed RPC sendRawTransaction test file to be included in th…

0eae159

…e test runner

client: only send tx in sendRawTransaction RPC method to peers if eth…

22ecae5

… protocol is available and bound

blockchain: deprecated main constructor, added tests for hardforkByHe…

99cb89f

…adBlockNumber option

holgerd77 marked this pull request as ready for review May 19, 2021 10:34

holgerd77 approved these changes May 19, 2021

View reviewed changes

holgerd77 merged commit a03be0f into master May 19, 2021

holgerd77 deleted the eip1559 branch May 19, 2021 10:43

holgerd77 added PR state: needs review and removed PR state: WIP labels May 19, 2021

ryanio mentioned this pull request May 20, 2021

tests: additional coverage #1261

Merged

holgerd77 added the package: devp2p label May 20, 2021

jochem-brouwer commented May 22, 2021

View reviewed changes

packages/tx/src/types.ts Show resolved Hide resolved

jochem-brouwer commented May 22, 2021

View reviewed changes

packages/tx/src/eip1559Transaction.ts Show resolved Hide resolved

jochem-brouwer commented May 22, 2021

View reviewed changes

packages/block/test/eip1559block.spec.ts Show resolved Hide resolved

jochem-brouwer commented May 22, 2021

View reviewed changes

packages/client/lib/blockchain/chain.ts Show resolved Hide resolved

ryanio mentioned this pull request May 23, 2021

Tiny Fixes Meta Issue (old) #905

Closed

18 tasks

holgerd77 mentioned this pull request May 25, 2021

Tx: Finalize AccessList API #1267

Open

ryanio mentioned this pull request May 29, 2021

1559: add more constraints to tx validation #1272

Merged

holgerd77 mentioned this pull request Jun 11, 2021

Tx: Improve EIP-2930 Tx Code Coverage #1224

Closed

holgerd77 mentioned this pull request Feb 16, 2022

v6 Breaking Release Planning #1717

Closed

51 tasks

holgerd77 mentioned this pull request May 17, 2022

Block: Directly throw in validation functions #1889

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement EIP1559 Fee Market + EIP3198 BaseFee #1148

Implement EIP1559 Fee Market + EIP3198 BaseFee #1148

jochem-brouwer commented Mar 14, 2021 •

edited

Loading

codecov bot commented Mar 14, 2021 •

edited

Loading

holgerd77 commented Mar 15, 2021

holgerd77 commented Mar 19, 2021

holgerd77 commented Mar 19, 2021

jochem-brouwer commented Mar 21, 2021

jochem-brouwer commented Mar 21, 2021

holgerd77 commented Mar 22, 2021

jochem-brouwer commented Mar 27, 2021

jochem-brouwer commented Apr 25, 2021

jochem-brouwer commented Apr 25, 2021

jochem-brouwer commented Apr 25, 2021

jochem-brouwer commented Apr 26, 2021 •

edited

Loading

jochem-brouwer commented Apr 26, 2021 •

edited by holgerd77

Loading

holgerd77 commented Apr 27, 2021

holgerd77 Apr 27, 2021

holgerd77 commented Apr 27, 2021

ryanio commented Apr 27, 2021 •

edited

Loading

holgerd77 commented May 19, 2021

holgerd77 left a comment

holgerd77 commented May 19, 2021

jochem-brouwer May 22, 2021

holgerd77 May 25, 2021

Implement EIP1559 Fee Market + EIP3198 BaseFee #1148

Implement EIP1559 Fee Market + EIP3198 BaseFee #1148

Conversation

jochem-brouwer commented Mar 14, 2021 • edited Loading

codecov bot commented Mar 14, 2021 • edited Loading

Codecov Report

holgerd77 commented Mar 15, 2021

holgerd77 commented Mar 19, 2021

holgerd77 commented Mar 19, 2021

jochem-brouwer commented Mar 21, 2021

jochem-brouwer commented Mar 21, 2021

holgerd77 commented Mar 22, 2021

jochem-brouwer commented Mar 27, 2021

jochem-brouwer commented Apr 25, 2021

jochem-brouwer commented Apr 25, 2021

jochem-brouwer commented Apr 25, 2021

jochem-brouwer commented Apr 26, 2021 • edited Loading

jochem-brouwer commented Apr 26, 2021 • edited by holgerd77 Loading

holgerd77 commented Apr 27, 2021

holgerd77 Apr 27, 2021

Choose a reason for hiding this comment

holgerd77 commented Apr 27, 2021

ryanio commented Apr 27, 2021 • edited Loading

holgerd77 commented May 19, 2021

holgerd77 left a comment

Choose a reason for hiding this comment

holgerd77 commented May 19, 2021

jochem-brouwer May 22, 2021

Choose a reason for hiding this comment

holgerd77 May 25, 2021

Choose a reason for hiding this comment

jochem-brouwer commented Mar 14, 2021 •

edited

Loading

codecov bot commented Mar 14, 2021 •

edited

Loading

jochem-brouwer commented Apr 26, 2021 •

edited

Loading

jochem-brouwer commented Apr 26, 2021 •

edited by holgerd77

Loading

ryanio commented Apr 27, 2021 •

edited

Loading