Autopilot Native Price API #3988

squadgazzz · 2025-12-17T11:14:19Z

Description

From the original issue #3831:

The orderbook(s) and the autopilot run as separate processes but both need access to native prices. Unfortunately they don't share the same underlying native price cache which can lead to inconsistencies - especially for niche tokens.
For example the orderbook could be able to fetch a native price and allow an order to be placed. Then the autopilot will pick up the order from the DB and try to fetch the native price as well. If a token is very niche there might only be 1 solver supporting it and it's possible that it serves the orderbook request correctly but rate limits the autopilot request.

To fix this we need to have 1 native price cache that all services can use. Ideally we'd move all the quoting logic into a separate service and make it use a Redis cache (in case we need to scale the service horizontally).
However that is a lot of work and we can get a pretty reasonably approximation with a lot less effort.

For that we just need to make the orderbook forward the native price requests to the autopilot. This will then cause the autopilot cache to be the 1 shared cache for all the processes.

Changes

Added axum server to autopilot to handle /native_price requests
The orderbook forwards requests to a configurable URL
- The new functionality is integrated into the current native price estimator config in the following way:
  - Driver native price estimator now has the Driver prefix to distinguish between legacy and new approach
  - The Forwarder prefix is added to use the Forwarder native price estimator.
- The orderbook also forwards the native price timeout. This query param is optional, and autopilot fallbacks to the configured one if not provided in the request.
Adjusted e2e tests
- Autopilot is now used as a native price estimator in all the tests.
- The dual_autopilot_only_leader_produces_auctions requires running 2 autopilots at the same time for some period of time. Since it doesn't make much sense to specify more than 2 autopilot URLs in the orderbook config, a reverse proxy is introduced to make this test pass. This proxy simulates k8s behavior: when 2 instances of the same service are running, requests are automatically routed between them. An alternative approach is to run 2 autopilot APIs on different ports and specify them in the orderbook config, but the reverse proxy simulates the production behavior.
The API server is initialized before the metrics for a reason. Currently, the metrics port is used as a readiness probe in the k8s cluster. To avoid introducing an additional readiness probe endpoint, I decided to continue using the metrics port. But if the metrics readiness probe becomes available before the API server, the new service is added to the k8s proxy pool and starts serving orderbook requests, but since the server is not initialized fully, there is a chance that some of the requests fail. Initializing the metrics only after the API server resolves this issue.
An openapi.yml file with the hope that this component will be decentralized in the future.

How to test

Updated e2e tests. This also requires running tests on staging manually.

Related Issues

Partially fixes #3831

Further implementation

Once we are confident the new approach works well, the old config with native price estimators can be dropped from the orderbook crate.

Copilot

Pull request overview

This PR introduces a native price API in the autopilot service and configures the orderbook to forward native price requests to it, addressing inconsistencies caused by separate caches in different processes. The implementation includes a fallback mechanism to the legacy price estimators if the autopilot is unavailable.

Key Changes:

Added an HTTP API server to autopilot exposing a /native_price/:token endpoint
Implemented a forwarding native price estimator in the orderbook that delegates to the autopilot API with fallback support
Updated e2e test infrastructure with a reverse proxy to support the dual-autopilot test scenario

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`crates/autopilot/src/infra/api.rs`	New HTTP API server for serving native price requests with error mapping
`crates/autopilot/src/infra/mod.rs`	Module declaration for the new API module
`crates/autopilot/src/run.rs`	Integration of the API server with graceful shutdown handling
`crates/autopilot/src/arguments.rs`	Added CLI argument for configuring the native price API bind address
`crates/autopilot/Cargo.toml`	Added axum, tower, and tower-http dependencies with tracing support
`crates/orderbook/src/native_price_forwarder.rs`	New forwarding estimator that proxies requests to autopilot with fallback logic
`crates/orderbook/src/lib.rs`	Module declaration for the native price forwarder
`crates/orderbook/src/run.rs`	Wraps the native price estimator with the forwarder when autopilot URL is configured
`crates/orderbook/src/arguments.rs`	Added CLI argument for the autopilot native price API URL
`crates/e2e/src/setup/proxy.rs`	New reverse proxy implementation for load balancing between two autopilot instances in tests
`crates/e2e/src/setup/mod.rs`	Module declaration for the proxy
`crates/e2e/src/setup/services.rs`	Updated default test configuration to use autopilot for native prices
`crates/e2e/tests/e2e/autopilot_leader.rs`	Updated dual-autopilot test to use the proxy for native price API failover
`Cargo.lock`	Dependency updates reflecting new autopilot dependencies

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/autopilot/src/infra/api.rs

github-actions · 2025-12-17T20:22:26Z

Reminder: Please consider backward compatibility when modifying the API specification.
If breaking changes are unavoidable, ensure:

You explicitly pointed out breaking changes.
You communicate the changes to affected teams (at least Frontend team and SAFE team).
You provide proper versioning and migration mechanisms.

Caused by:

crates/autopilot/openapi.yml

crates/e2e/src/setup/proxy.rs

# Conflicts: # crates/e2e/tests/e2e/autopilot_leader.rs

MartinquaXD

Looks alright overall. Main confusion for me at the moment are the changes to the CLI args in the tests.

crates/autopilot/src/infra/api.rs

MartinquaXD · 2025-12-22T09:29:30Z

crates/e2e/src/setup/proxy.rs

+    let path = parts
+        .uri
+        .path_and_query()
+        .map(|pq: &axum::http::uri::PathAndQuery| pq.as_str())
+        .unwrap_or("");


Looks like this would better live inside try_backend. Ideally handling of the body bytes would also go there but since warp::hyper::body::to_bytes has to consume the Body this doesn't work well unfortunately.

crates/e2e/src/setup/proxy.rs

MartinquaXD · 2025-12-22T09:42:41Z

crates/e2e/src/setup/services.rs

-    fn api_autopilot_arguments(&self) -> impl Iterator<Item = String> + use<> {
+    fn api_arguments(&self) -> impl Iterator<Item = String> + use<> {
        [
-            "--native-price-estimators=test_quoter|http://localhost:11088/test_solver".to_string(),


Not sure if this was already the case when I originally commented on this line but the --native-price-estimator argument in api_arguments now has to be overwritten in at least 1 setup (either api or autopilot) so I'm not sure if this should show up here at all. WDYT?

crates/e2e/src/setup/services.rs

crates/shared/src/price_estimation/native/forwarder.rs

MartinquaXD · 2025-12-22T09:56:33Z

crates/e2e/src/setup/services.rs

-    fn api_autopilot_arguments(&self) -> impl Iterator<Item = String> + use<> {
+    fn api_arguments(&self) -> impl Iterator<Item = String> + use<> {
        [
-            "--native-price-estimators=test_quoter|http://localhost:11088/test_solver".to_string(),


Also originally I thought the new proxy is supposed to be used for every test - not just for the autopilot leader test. If it's only supposed to be used for very few tests making Forwarder the default here doesn't seem right. Sorry for the confusion.

And if the new proxy is only intended to be used for the autopilot leader tests the only tests that should require changes to the CLI args would be those, no?

crates/e2e/src/setup/proxy.rs

rdin777 · 2025-12-22T10:49:34Z

jmg-duarte

Just a nit

crates/e2e/src/setup/proxy.rs

MartinquaXD

LGTM. Just nits remaining.

crates/e2e/tests/e2e/limit_orders.rs

crates/e2e/src/setup/proxy.rs

squadgazzz added 2 commits December 17, 2025 11:14

Autopilot Native Price API

d914f6b

Fix port

4699fb5

squadgazzz force-pushed the autopilot/native-price-api branch from f37f048 to 4699fb5 Compare December 17, 2025 11:17

squadgazzz added 6 commits December 17, 2025 14:02

Better errors

e106870

Fix

9a7c718

Ignore dual autopilot test

285a3e5

Fixed test with a proxy

d6cfb57

fix

273faae

Init API server before metrics

5db5fc0

squadgazzz force-pushed the autopilot/native-price-api branch from 31968f3 to 5db5fc0 Compare December 17, 2025 19:42

squadgazzz marked this pull request as ready for review December 17, 2025 20:09

squadgazzz requested a review from a team as a code owner December 17, 2025 20:09

Copilot AI review requested due to automatic review settings December 17, 2025 20:09

Copilot started reviewing on behalf of squadgazzz December 17, 2025 20:10 View session

Copilot AI reviewed Dec 17, 2025

View reviewed changes

crates/autopilot/src/infra/api.rs Outdated Show resolved Hide resolved

OpenAPI

25a0a24

squadgazzz added 2 commits December 17, 2025 20:26

Include error reason

785582e

Forward the timeout

45f99ab

fafk reviewed Dec 18, 2025

View reviewed changes

crates/e2e/src/setup/proxy.rs Show resolved Hide resolved

squadgazzz marked this pull request as draft December 18, 2025 10:14

squadgazzz added 9 commits December 18, 2025 12:44

Migrate the config

b6089cd

Merge branch 'main' into autopilot/native-price-api

c0ce7b3

# Conflicts: # crates/e2e/tests/e2e/autopilot_leader.rs

Formatting

5ac6341

Fix and docs

065a5aa

Fix

ab91606

fix

9b571d3

Formatting

c2fc05b

Fix

ebfeafb

Fix

ef3a1a2

squadgazzz added 8 commits December 19, 2025 12:32

Review comments

34202be

Review comments

3b4a1f9

Logs

0477a38

Better parser

87f29ad

Proxy rework

fc11acd

e2e config rework

bd79595

Update proxy

556cc2b

Doc

449abcb

squadgazzz requested review from MartinquaXD and jmg-duarte December 19, 2025 13:47

fafk approved these changes Dec 19, 2025

View reviewed changes

Refactor

48414f2

MartinquaXD reviewed Dec 22, 2025

View reviewed changes

squadgazzz added 3 commits December 22, 2025 10:41

Allowed timeout range

9721a90

Timer

114c398

Refactor

687fb30

squadgazzz requested a review from MartinquaXD December 22, 2025 10:51

Doc

b7914fc

jmg-duarte approved these changes Dec 22, 2025

View reviewed changes

crates/e2e/src/setup/proxy.rs Outdated Show resolved Hide resolved

squadgazzz added 2 commits December 22, 2025 18:37

Body ownership

487f081

Merge branch 'main' into autopilot/native-price-api

2a0e063

MartinquaXD approved these changes Dec 23, 2025

View reviewed changes

crates/e2e/tests/e2e/limit_orders.rs Show resolved Hide resolved

crates/e2e/tests/e2e/limit_orders.rs Show resolved Hide resolved

crates/e2e/src/setup/proxy.rs Outdated Show resolved Hide resolved

squadgazzz added 2 commits December 23, 2025 16:58

Small refactor

2243fdf

Nit

3ed714c

squadgazzz added this pull request to the merge queue Dec 23, 2025

Merged via the queue into main with commit 9c3ec82 Dec 23, 2025
19 checks passed

squadgazzz deleted the autopilot/native-price-api branch December 23, 2025 18:28

github-actions bot locked and limited conversation to collaborators Dec 23, 2025

Autopilot Native Price API #3988

Autopilot Native Price API #3988

Uh oh!

Conversation

squadgazzz commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

How to test

Related Issues

Further implementation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

Uh oh!

MartinquaXD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MartinquaXD Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MartinquaXD Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MartinquaXD Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rdin777 commented Dec 22, 2025

Uh oh!

jmg-duarte left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MartinquaXD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

squadgazzz commented Dec 17, 2025 •

edited

Loading