Release tracking issue: DEX

[zbuc]

This issue tracks implementation of Penumbra's DEX engine, the component that performs the four phases of DEX execution at the end of every block:

1. Adding all newly opened liquidity positions
2. Executing the batched swaps from the current block against that liquidity and recording the results
3. Arbing prices back to consistency
4. Closing all newly closed liquidity positions

To do this, we have a number of tasks, both on the in-consensus application side, as well as on the client side, and in the protocol tooling. These tasks break down as follows:

System / Mechanism design issues

This category is largely done, except for the details of trading function approximations and the reward claim mechanism. Since we don't currently have an LP reward system, reward claim design isn't blocking at the moment, and we'll do it after getting the rest of the system working.

• Derive trading function approximation #1908
• dex: design details of LP reward claim mechanism #2167

Crypto

• dex: swap claim should check diversified address integrity #3978

ActionHandler implementations

This category covers the consensus-side execution of the action handlers for position actions. Client-side logic like Planner support is tracked in #1991.

• PositionOpen implementation #2137
• PositionClose implementation #2138
• PositionWithdraw implementation #2139
• PositionRewardClaim implementation #2140 (Blocked on dex: design details of LP reward claim mechanism #2167, to be done after the rest of the system is working)
• Limit fee/gamma sizes to prevent division by zero #2203

DEX Engine

• Fill an order against a single position (done in component(dex): state-machine tests for execution #2145)
• Add a PositionRead method to stream position IDs by price #2211
• Fix position-by-price indexing to allow multiple positions at a given price #2212
• Update semantics of BatchSwapOutputData to support partial fills #2157
• Fill an order against multiple positions #2202
• Implement multi-hop routing #2283
• Maintain indices of available liquidity for open positions #2378
• Fill in SwapExecution data management #1946
• Store the SwapExecution with consensus state. #1947
• Implement arbitrage using multi-hop routing #1779
• Add complex state-machine tests for execution and order-book traversal #2237 (following Derive trading function approximation #1908)
• Add liquidity-based position eviction to the DEX #4077 (was dex: implement forced ranking for positions  #2928)
• dex: implement a value circuit breaker #2986
• dex: implement an execution circuit-breaker #3303
• Add a chain parameter for global slippage limits on swap execution #1943
• dex: record Positions, not just position IDs, in position indexes #2625
• dex(path_search): test liquidity heuristic with ibc assets #3039
• dex: define LP state machine #3851
• dex(perf): store indices in prefixed storage #3353
• RoutingParams::default() is needlessly inefficient #4000
• dex: add DexParameters, remove RoutingParams::default() #4066
• Implement an enabled switch for the DEX #4078
• dex: surface swap/position assets during candidate selection #4177

DEX engine bugs (non-exhaustive):

• dex: don't close Withdrawn positions #3850
• dex: move value circuit breaker accounting to dex engine boundaries #4025
• Investigate arb execution #3790
• Arb execution does not update value circuit breaker #4113
• dex: use big endian of complement in the base liquidity index  #4189
• dex: fix frontier cache incoherence #4283

Client support

We don't expect that in the long term, pcli will be used much for position management, but we'll use it as an MVP to exercise the chain logic at first. We need:

• Planner support for PositionClose #2231
• Planner support for PositionWithdraw #2232
• pcli tx position close POSITION_ID #2233
• pcli tx position withdraw POSITION_ID #2234
• Prepopulate LPNFT denoms in view server #2410
• Minimal smoke test support for position management #1909
• Provide a pcli interface to post xyk approximations #2319 (needs Derive trading function approximation #1908)
• Add an RPC method that queries for SwapExecutions and expose it in pcli q. #2386
• Tracking issue: DEX state legibility for API consumers #2267
• Brainstorm regarding which ABCI events should be emitted from the DEX component
• Fill in missing DEX RPC methods #2576
• Add RPC for trade simulation #2580
• pcli tx lp order sell gives a mispriced position when used with delegation tokens #2579
• Add support for storing and querying owned positions to the view service #2531
• dex_utils: make the RMM solver wasm friendly #2823
• dex_utils: the xyk solver sometimes returns negative solutions #2516
• dex: inventorize useful ABCI events to index #3645
• dex: make SimulateTrade pull its routing parameters from the engine #4060
• dex: add unfilled_input to SimulateTradeResponse #3842
• Add Rust view server support for Swap/SwapClaim linking #4083
• Remove SwapExecutions from consensus state #4084
• Auto-closing positions don't emit EventPositionClose #4123
• dex_utils: implement v3 approximations #4112

Node architecture integration work

• Update Component trait methods to allow state sharing during execution #2406
• Implement Component code calling out to new routing/filling logic #2398
• Remove existing StubCpmm and direct Swap executions at the DEX engine #1906

Testing wishlish (more to come..):

• Enshrine DEX position check_stateless and check_stateful expectations in tests #2421

Testnet Strategy

Some ideas:

• Create testnet allocation with multiple paper assets #2229
• Write a bot that maintains active liquidity by quoting prices with some spread #2204

Raw notes

Based on details in https://hackmd.io/rsgSwz3BQKuL0GrSF0LWXA# and https://hackmd.io/jtoiMn0pQnSTZlOi2ig5Wg