XDP by default by alexpyattaev · Pull Request #12119 · anza-xyz/agave

alexpyattaev · 2026-04-22T11:14:34Z

Problem

XDP should not be experimental anymore https://discord.com/channels/428295358100013066/478692221441409024/1496253676891930865

Summary of Changes

Enable by default for agave-validator
Still disabled for multinode and test-validator (since those are expected to run on localhost)

Testing

This is currently running on an a staked testnet node DZ6m64cghUKgVcvsFgAb3NDaeEyRpuMSK8hwqETndNTS

codecov-commenter · 2026-04-25T20:01:14Z

Codecov Report

❌ Patch coverage is 78.04878% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.2%. Comparing base (10bb14c) to head (e9c098a).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #12119       +/-   ##
===========================================
+ Coverage    72.1%    83.2%    +11.0%     
===========================================
  Files         748      855      +107     
  Lines      227236   341670   +114434     
===========================================
+ Hits       164042   284518   +120476     
+ Misses      63194    57152     -6042

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

gregcusack · 2026-06-14T12:38:43Z

made a replacement PR here: #13192
^ did this since:

we want copy mode default
we want to use --no-xdp
alex focusing on votor work.

although, I happy to build on top of this PR is and get rid of mine. but if we go that way, we need to add the things mentioned above.

alexpyattaev · 2026-06-14T17:32:22Z

made a replacement PR here: #13192 ^ did this since:
1. we want copy mode default

This is not changing the mode used by default, and agave default is copy mode.

2. we want to use `--no-xdp`

This was rejected by @bw-solana in #12119 (comment)

gregcusack

ignore my comment above! sorry, i completely misread that this had xdp copy mode on by default. closed my other PR. this looks good! few thoughts!

also should probably promote experimental poh arg before we merge this one imo. see: #13191

gregcusack · 2026-06-14T20:09:04Z

  --no-wait-for-vote-to-start-leader
  --full-rpc-api
  --allow-private-addr
+  --disable-xdp


per brooks' comment and brennan's thumbs up in slack. let's change this back to --no-xdp

yes, apologies for causing the thrash. I'm redacted again

gregcusack · 2026-06-14T20:12:00Z

    if bind_addresses.len() > 1 && public_tvu_addr.is_some() {
-        Err(String::from(
-            "--public-tvu-address can not be used in a multihoming context",
-        ))?;
+        Err(String::from("XDP can not be used in a multihoming context"))?;
    }


we can probably remove this? doesn't seem right to base this on public_tvu_addr. in fact if someone passes in two bind addresses, we should force them to also pass in --no-xdp.

gregcusack · 2026-06-14T20:13:05Z

+* XDP transmit is now enabled by default on Linux. The validator requires `CAP_NET_ADMIN` and
+  `CAP_NET_RAW` capabilities (plus `CAP_BPF` and `CAP_PERFMON` for `--xdp-zero-copy`). Pass
+  `--disable-xdp` to disable XDP and fall back to UDP sockets.


i would just mention that xdp in skb mode (copy mode) is enabled to be clear.

gregcusack · 2026-06-14T20:14:19Z

+* The `--experimental-retransmit-xdp-interface`, `--experimental-retransmit-xdp-cpu-cores`, and
+  `--experimental-retransmit-xdp-zero-copy` flags have been deprecated. Use `--xdp-interface`,


haven't these flags already been deprecated? so this should be a separate PR. they should probably be removed in 4.2, right? or we need to wait for major version bump?

this last comment should stay though:

XDP is now enabled by default; use `--disable-xdp` to opt out.

Yep just needed to rebase.

gregcusack · 2026-06-14T20:15:05Z

+  `--experimental-retransmit-xdp-zero-copy` flags have been deprecated. Use `--xdp-interface`,
+  `--xdp-cpu-cores`, and `--xdp-zero-copy` instead. XDP is now enabled by default; use `--disable-xdp` to opt out.
 #### Changes
+* Added `--disable-xdp` flag to disable XDP transmit and fall back to UDP sockets.


don't need to mentioned --disable-xdp twice. either leave here or leave above. i'd probably recommend leaving here and getting rid above

gregcusack · 2026-06-14T20:15:20Z

+* Added `--xdp-interface` to specify the network interface for XDP transmit (auto-detected from
+  default route if not specified).
+* Added `--xdp-cpu-cores` to specify CPU cores for XDP transmit (defaults to 1 auto-selected core
+  that avoids the PoH core and its hyperthread sibling).
+* Added `--xdp-zero-copy` to enable XDP zero-copy mode (requires hardware/driver support).


this should be separate PR since these were already added in a previous PR.

yes this somehow bypassed rebase.

gregcusack · 2026-06-14T20:16:36Z

+To run without XDP, pass `--disable-xdp` instead (no capabilities required in this case).
+


tbh i wouldn't advertise that you can run without xdp. so i would delete this. make them look for it if they really want to turn it off lol

gregcusack · 2026-06-14T20:17:39Z

 #### Breaking
 * `--block-production-method central-scheduler` is no longer supported. If passed, a warning is emitted and behavior
  will default to the greedy-scheduler implementation.
+* XDP transmit is now enabled by default on Linux. The validator requires `CAP_NET_ADMIN` and


i would also add in here that we now enforce pinned cpu core does not overlap with xdp core.

gregcusack · 2026-06-14T20:19:11Z

+        let poh_pinned_cpu_core =
+            value_of(matches, "poh_pinned_cpu_core").or(poh_service::DEFAULT_PINNED_CPU_CORE);
+        let xdp_cpu_cores = matches
+            .value_of("xdp_cpu_cores")
+            .or_else(|| matches.value_of("experimental_retransmit_xdp_cpu_cores"));
+        let cpus = if let Some(cpu_str) = xdp_cpu_cores {
+            if bind_addresses.len() > 1 {
+                Err(String::from(
+                    "--xdp-cpu-cores cannot be used in a multihoming context",
+                ))?;
+            }
+            Some(parse_cpu_ranges(cpu_str).expect("clap validator already accepted this CPU list"))
+        } else {
+            // Auto-select a single core for XDP.
+            match cpu_affinity(None) {
+                Ok(allowed) => {
+                    match allowed
+                        .iter()
+                        .rev()
+                        .map(|cpu| **cpu)
+                        .find(|cpu| Some(*cpu) != poh_pinned_cpu_core)
+                    {
+                        Some(cpu) => Some(vec![cpu]),
+                        None => {
+                            return Err(format!(
+                                "XDP requires a dedicated CPU core separate from PoH (core \
+                                 {poh_pinned_cpu_core:?}), but none is available. Pass \
+                                 --disable-xdp to disable XDP."
+                            ))?;
+                        }
+                    }
+                }
+                Err(e) => {
+                    return Err(format!(
+                        "failed to query CPU affinity: {e}. Pass --disable-xdp to disable XDP, or \
+                         provide --xdp-cpu-cores explicitly."
+                    ))?;
+                }
+            }
+        };
+        cpus.map(|cpus| {
+            info!("XDP enabled on CPU cores: {cpus:?}");
+            XdpConfig::new(xdp_interface, cpus, xdp_zero_copy)
+        })
    };


can we pull this out into a function so we can write a test or two to make sure we haven't screwed up slash don't screw up cli args

gregcusack · 2026-06-14T20:27:43Z

+        let cpus = if let Some(cpu_str) = xdp_cpu_cores {
+            if bind_addresses.len() > 1 {
+                Err(String::from(
+                    "--xdp-cpu-cores cannot be used in a multihoming context",
+                ))?;
+            }
+            Some(parse_cpu_ranges(cpu_str).expect("clap validator already accepted this CPU list"))


if you explicitly pass in the same xdp core as poh core, this will not fail. so probably want to ensure these don't conflict here.

gregcusack

looks better! thank you! will you also update docs/src/operations/running-with-af-xdp.md with the changes to that md file seen here please: https://github.com/anza-xyz/agave/pull/13192/changes

gregcusack · 2026-06-15T20:08:28Z

+                 been permitted to the current process: {missing_caps:?}. If XDP is not needed, \
+                 pass --no-xdp to disable it",


nit: i would remove this part lol:

If XDP is not needed, pass --no-xdp to disable it

gregcusack · 2026-06-15T20:14:53Z

-    if bind_addresses.len() > 1 && public_tvu_addr.is_some() {
-        Err(String::from(
-            "--public-tvu-address can not be used in a multihoming context",
-        ))?;
-    }
-


this deletion seems unrelated?

spooky action at a distance. reverted.

gregcusack · 2026-06-15T20:19:39Z

+$ sudo setcap 'cap_net_admin,cap_net_raw+eip' target/release/agave-validator
+```
+
+For XDP zero-copy mode (`--xdp-zero-copy`), additional capabilities are needed:
+
+```bash
+$ sudo setcap 'cap_net_admin,cap_net_raw,cap_bpf,cap_perfmon+eip' target/release/agave-validator
+```


above it says run ./cargo build but then we say to use setcap on target/release/agave-validator but ./cargo build won't put the agave-validator binary into release/

gregcusack · 2026-06-15T20:22:51Z

+    fn test_explicit_xdp_core_conflicts_with_poh_core_is_error() {
+        let default_args = DefaultArgs::default();
+        let app = add_args(clap::App::new("agave-validator"), &default_args);
+        // poh_pinned_cpu_core defaults to DEFAULT_PINNED_CPU_CORE (1); set XDP to the same.


this defaults to 0, not 1

I've just removed the comment.

alexpyattaev · 2026-06-16T13:55:37Z

will you also update docs/src/operations/running-with-af-xdp.md with the changes to that md file seen here please: https://github.com/anza-xyz/agave/pull/13192/changes

Docs would have to be another PR to another repo - docs were moved.

gregcusack · 2026-06-16T14:02:59Z

will you also update docs/src/operations/running-with-af-xdp.md with the changes to that md file seen here please: https://github.com/anza-xyz/agave/pull/13192/changes

Docs would have to be another PR to another repo - docs were moved.

they removed docs?!?! sad. ok now it's harder to match code updates to doc updates. but i get why they did it.

gregcusack

last few thoughts!

gregcusack · 2026-06-16T14:12:39Z

+        }
+        Some(parsed)
+    } else {
+        // Auto-select a single core, avoiding the PoH core and its hyperthread sibling.


nit: "and its hyperthread sibling." i am not sure this is actually true...

yes this used to be true with previous version that did not use cpu-utils. stale.

gregcusack · 2026-06-16T14:16:14Z

+$ sudo setcap 'cap_net_admin,cap_net_raw+eip' target/debug/agave-validator
+```
+
+For XDP zero-copy mode (`--xdp-zero-copy`), additional capabilities are needed:
+
+```bash
+$ sudo setcap 'cap_net_admin,cap_net_raw,cap_bpf,cap_perfmon+eip' target/debug/agave-validator


nit: sorry, maybe we should just say

sudo setcap 'cap_net_admin,cap_net_raw+eip' <path-to-agave-validator-binary>

because if we're running in debug mode, why are we using xdp. but then again above it says:

> [!NOTE] > Note that this builds a debug version that is **not suitable for running a testnet or mainnet validator**. Please read [`docs/src/cli/install.md`](docs/src/cli/install.md#build-from-source) for instructions to build a release version for test and production uses.

^ so maybe that's clear enough that if you are actually going to run a validator you don't want to run in debug mode.

gregcusack · 2026-06-16T14:27:48Z

+    let poh_pinned_cpu_core =
+        value_of(matches, "poh_pinned_cpu_core").or(poh_service::DEFAULT_PINNED_CPU_CORE);


ok so we also need to check for the experimental version of this pinned poh flag as well. or else we won't catch that the xdp and poh cores overlap

Gotta love ancient version of clap..

mergify · 2026-06-20T14:59:24Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

gregcusack

lgtm!

alexpyattaev changed the title ~~Xdp by default~~ XDP by default Apr 22, 2026

alexpyattaev force-pushed the xdp_by_default branch from 4d113c2 to c898197 Compare April 23, 2026 06:56

bw-solana reviewed Apr 23, 2026

View reviewed changes

Comment thread validator/src/commands/run/args.rs Outdated

Comment thread validator/src/commands/run/execute.rs Outdated

Comment thread validator/src/commands/run/execute.rs Outdated

Comment thread xdp/src/cpu_topology.rs Outdated

Comment thread validator/src/commands/run/args.rs Outdated

alexpyattaev added noCI Suppress CI on this Pull Request CI Pull Request is ready to enter CI and removed noCI Suppress CI on this Pull Request labels Apr 23, 2026

anza-team removed the CI Pull Request is ready to enter CI label Apr 24, 2026

alexpyattaev force-pushed the xdp_by_default branch from 07d0d3c to af9c08f Compare April 24, 2026 14:50

bw-solana mentioned this pull request Apr 24, 2026

add cpu-utils #8944

Merged

alexpyattaev force-pushed the xdp_by_default branch from af9c08f to da27354 Compare April 25, 2026 19:31

alexpyattaev mentioned this pull request May 2, 2026

XDP: graduate from experimental status #12266

Merged

alexpyattaev force-pushed the xdp_by_default branch 2 times, most recently from 1e17c53 to e39da1a Compare June 12, 2026 09:58

alexpyattaev marked this pull request as ready for review June 12, 2026 10:53

alexpyattaev requested a review from a team June 12, 2026 10:53

gregcusack mentioned this pull request Jun 14, 2026

Enable XDP copy mode by default #13192

Closed

gregcusack reviewed Jun 14, 2026

View reviewed changes

alexpyattaev force-pushed the xdp_by_default branch from e39da1a to 8195995 Compare June 15, 2026 19:45

alexpyattaev requested a review from gregcusack June 15, 2026 20:17

gregcusack reviewed Jun 15, 2026

View reviewed changes

alexpyattaev force-pushed the xdp_by_default branch from 8195995 to 9865524 Compare June 16, 2026 13:54

alexpyattaev requested a review from gregcusack June 16, 2026 13:59

gregcusack reviewed Jun 16, 2026

View reviewed changes

XDP: enable by default

e9c098a

alexpyattaev force-pushed the xdp_by_default branch from 9865524 to e9c098a Compare June 16, 2026 19:56

alexpyattaev requested review from bw-solana and gregcusack June 17, 2026 07:48

bw-solana approved these changes Jun 20, 2026

View reviewed changes

gregcusack approved these changes Jun 22, 2026

View reviewed changes

gregcusack added this pull request to the merge queue Jun 22, 2026

Merged via the queue into anza-xyz:master with commit 1d0d634 Jun 22, 2026
43 checks passed

gregcusack deleted the xdp_by_default branch June 22, 2026 16:41

mjain-jump mentioned this pull request Jul 1, 2026

mjain/make program cache infallible firedancer-io/agave#32

Closed

		* The `--experimental-retransmit-xdp-interface`, `--experimental-retransmit-xdp-cpu-cores`, and
		`--experimental-retransmit-xdp-zero-copy` flags have been deprecated. Use `--xdp-interface`,

		To run without XDP, pass `--disable-xdp` instead (no capabilities required in this case).

		been permitted to the current process: {missing_caps:?}. If XDP is not needed, \
		pass --no-xdp to disable it",

		let poh_pinned_cpu_core =
		value_of(matches, "poh_pinned_cpu_core").or(poh_service::DEFAULT_PINNED_CPU_CORE);

Uh oh!

Conversation

alexpyattaev commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Summary of Changes

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Apr 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gregcusack commented Jun 14, 2026

Uh oh!

alexpyattaev commented Jun 14, 2026

Uh oh!

gregcusack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bw-solana Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gregcusack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexpyattaev commented Jun 16, 2026

Uh oh!

gregcusack commented Jun 16, 2026

Uh oh!

gregcusack left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexpyattaev commented Apr 22, 2026 •

edited

Loading

codecov-commenter commented Apr 25, 2026 •

edited

Loading

bw-solana Jun 20, 2026 •

edited

Loading