Tracking: Engine revamp

[mattsse]

Describe the feature

The current engine <-> tree design has several drawbacks

• sequential processing

• Tree Read lock that interferes with rpc (pending block access)

• very complex request processing that requires multiple tree access roundtrips

• This must be redesigned entirely

This should allow us to support other interfaces like ABCI,Arbitrum or others via the Node Builder

Tasks

Beta Give feedback

1. Add high level ChainOrchestrator abstraction #8744
   A-blockchain-tree C-enhancement S-stale +3
2. In Memory Engine Tree state #8745
   A-blockchain-tree C-debt +2
   
3. Database Migration Thread #8746
   3 of 3
   A-blockchain-tree C-debt +2
   
4. Blockchain Provider tree state integration #8747
   A-blockchain-tree C-debt +2
   
5. Node Builder integration of new engine implemention #9559
   C-enhancement +1
   
6. rename PipelineHandler #9008
   A-consensus C-debt +2
   
7. Merge EngineApiRequestHandler and EngineHandler #9009
   +0
   
8. Implement blockdownloader for p2p #9011
   A-consensus A-networking C-debt +3
   
9. Implement pipeline BackfillSync #9012
   A-consensus C-debt +2
   
10. feat: new engine API handler #8559
    A-consensus C-debt C-perf +3
11. Add ethereum/engine implementation #9151
    C-enhancement +1
    
12. Integrate tree and persistence setup in EthService #9576
    A-consensus C-enhancement +2
    
13. Support --debug.tip in EthService #9577
    C-enhancement +1
14. Convert Ethservice into stream #9582
    C-enhancement +1
    
15. Add new BlockchainProvider type #9607
    C-enhancement +1
    
16. Integrate memory overlay in canonical state #9614
    C-enhancement +1
    
17. Add more docs for tree engine #9615
    C-docs S-stale +2
18. Rename BlockSource::Database variant #9617
    D-good-first-issue S-blocked +2
19. Remove InMemoryState trait #9638
    C-debt +1
    
20. Integrate chainorchestrator events into node events #9644
    C-enhancement +1
21. Improve engine event variants #9671
    C-enhancement +1
    
22. Issue requests for missing blocks on forkchoice update #9672
    C-enhancement +1
    
23. Trigger pipeline sync from engine/tree handler #9673
    C-enhancement +1
    
24. Improve treeevent handling for downloaded blocks #9685
    C-debt C-enhancement +2
    
25. handle tree errors gracefully #9688
    C-debt +1
    
26. Recheck backfill state after we finished pipeline sync #9689
    +0
    
27. Update sync state status in the tree #9694
    C-enhancement +1
28. Implement make canonical #9696
    C-enhancement +1
29. add consistency check on startup #9700
    C-enhancement +1
    
30. Integrate payloadbuilder handler #9705
    C-enhancement +1
    
31. Integrate blockchainprovider2 in ethnodelauncher #9731
    C-enhancement +1
    
32. Better testing support for engine api #9749
    3 of 3
    +0
    
33. Add missing checks for canonical safe and finalized blocks #9750
    C-enhancement +1
34. Improve in memory block tracking #9752
    S-needs-design +1
    
35. Support sending messages to EngineApiImpl from itself #9753
    S-needs-design +1
    
36. Provide CanonicalInMemory state as input to EngineHandler #9760
    +0
    
37. replace generic stateprovider with boxed trait #9777
    C-enhancement +1
    
38. Move MemoryOverlayStateProvider to chain-state crate #9778
    C-debt +1
    
39. Make EngineHandler terminate if underlying channels close #9793
    C-debt +1
40. Improve engine observability #9794
    S-stale +1
41. Add metrics for inmemory canonical state #9806
    A-observability C-enhancement +2
    
42. Notify tracker about received engine messages #9807
    +0
    
43. Add support for merging execution outputs #9812
    +0
44. Improve backfill state tracking #9815
    C-enhancement D-good-first-issue +2
45. Make tree settings configurable #9821
    C-enhancement +1
    
46. reverse MemoryOverlayStateProvider block order #9825
    +0
    
47. Add BlockSource::Database variant #9828
    +0
48. Update pending block if new #9829
    +0
    
49. Remove StateProvider::pending_with_provider trait function #9849
    C-debt +1
    
50. ensure backfill only starts if persistence task finished #9893
    C-bug +1
    
51. Only persist the canonical chain #9897
    C-bug +1
    
52. check if head is already canonical #9899
    C-bug +1
    
53. Return fatal error if persistence task dropped #9909
    +0
    
54. Improve persistence action commits #9923
    C-debt +1
    
55. Support in memory lookups in BlockchainProvider2 #9924
    +0
    
56. Improve message tree task message model #9926
    C-debt +1
    
57. Handle inserting disconnected blocks gracefully #9935
    +0
    
58. Add metrics for in memory tree and canonical memor #9957
    A-observability +1
    
59. gracefully handle persistence errors #9958
    C-debt +1
    
60. dont truncate blocks to persist #9978
    C-perf +1
    
61. Propagate and handle errors in database and static file services #9535
    A-db A-static-files C-enhancement S-stale +4
62. Improve missing block downloading #10007
    C-bug +1
    
63. blocks to persist can be empty #10009
    A-observability +1
    
64. Live sync transition scenario tests #10015
    C-bug C-test +2
    
65. Improve connecting downloaded blocks #10017
    C-bug C-test +2
    
66. Improve download observability #10018
    +0
    
67. Improve in memory range lookups #10037
    C-enhancement D-good-first-issue +2
    
68. Use BtreeMap for block numbers #10038
    C-enhancement D-good-first-issue +2
69. Make payloadbuilder emit an executed block #10096
    +0
    
70. Introduce custom engine APi commands as request variants #10097
    +0
71. bubble up download event from engineHandler #10098
    +0
72. add devmode launcher implementation #10104
    C-enhancement D-complex +2
    
73. Ensure persistence receiver is dropped when value is received #10115
    C-bug +1
    
74. remove EngineApiTreeHandler trait #10123
    C-debt +1
    
75. Introduce EngineApiKind #10124
    C-enhancement S-stale +2
76. Make EthService reusable #10145
    +0
77. Convert pending block to a watch channel #10157
    C-enhancement +1
    
78. Integrate ExecutedBlock in BuiltPayload #10159
    +0
79. Integrate Engine2 for Optimism #10160
    C-enhancement +1
    
80. BlockNotfound in treestate warning #10179
    C-bug +1
81. Add threshold for how many blocks can be persisted #10187
    C-enhancement +1
    
82. Tracking: In-memory compliance for BlockchainProvider2 #10186
    9 of 11
    A-blockchain-tree A-db C-tracking-issue +3
83. track persistence duration #10220
    D-good-first-issue +1
    
84. extract persistence event handling into standalone functions #10221
    D-good-first-issue +1
    
85. Properly bubble up InsertBlockFatalError #10219
    A-blockchain-tree C-enhancement +2
    
86. Properly handle restart edge cases between ExEx and new engine #10489
    A-blockchain-tree A-exex M-prevent-stale S-needs-design +4
    
87. Provider returns incorrect or missing data when used externally #8751
    A-static-files C-bug M-prevent-stale +3
    
88. bug(engine): persisted executed block retrieval #11251
    C-bug +1
    

Additional context

No response

[emhane]

will solve these issues #8303, especially #8746 and #8745

[emhane]

#8745 and #8746 should solve #7322

[github-actions]

This issue is stale because it has been open for 21 days with no activity.