Skip to content

fix(setup-dpdk): ip link down each NIC + mount hugetlbfs at /mnt/huge1g#85

Merged
skullcrushercmd merged 1 commit intomainfrom
fix/setup-dpdk-bind-link-down
Apr 28, 2026
Merged

fix(setup-dpdk): ip link down each NIC + mount hugetlbfs at /mnt/huge1g#85
skullcrushercmd merged 1 commit intomainfrom
fix/setup-dpdk-bind-link-down

Conversation

@skullcrushercmd
Copy link
Copy Markdown
Contributor

Summary

Two operator-side speedbumps surfaced on c6in.metal during anygpt-52 (PR #65 issuecomment-4339242358):

1. dpdk-devbind refuses to bind active interfaces:

Warning: routing table indicates that interface is active. Not modifying.

Operators had to ip link set <ifc> down on every NIC by hand before the bind succeeded.

2. Reserving 1 GiB hugepages was not enough for rte_eal_init:

EAL: No available 1048576 kB hugepages reported on node 0/1

Operators had to mount -t hugetlbfs -o pagesize=1G nodev /mnt/huge1g manually before the scanner could find the pages.

Fix

  • bdf_to_iface() walks /sys/bus/pci/devices/<bdf>/net/ to map BDF → iface name. cmd_bind invokes ip link set <ifc> down on each target before invoking dpdk-devbind. Best-effort: missing ip, missing iface (already vfio-pci), and already-down ifaces all proceed to the bind.
  • ensure_hugetlbfs_mount() mounts a hugetlbfs of the matching pagesize at the configured path after a successful nr_hugepages reservation. Default targets are /mnt/huge1g (1 GiB) and /mnt/huge2m (2 MiB); override via ANYSCAN_DPDK_HUGEPAGES_1G_MOUNT / _2M_MOUNT, or set to empty to opt out (operators provisioning hugetlbfs via fstab). Idempotent.
  • ANYSCAN_DPDK_LOAD_ONLY=1 hook lets the test source the script without triggering its argv dispatch.

Test plan

  • bash tools/test-setup-dpdk.sh — 9/9 pass
  • cmd_bind invokes ip link set <ifc> down BEFORE dpdk-devbind --bind=vfio-pci (order verified by line numbers in a single cmd log)
  • ensure_hugetlbfs_mount calls mount -t hugetlbfs -o pagesize=1G nodev <path>
  • ensure_hugetlbfs_mount is no-op when target is already hugetlbfs with the matching pagesize
  • ensure_hugetlbfs_mount is no-op with empty mount path
  • bdf_to_iface resolution against a fake /sys tree (positive + negative)

🤖 Generated with Claude Code

Two operator-side speedbumps surfaced on c6in.metal during anygpt-52
(PR #65 issuecomment-4339242358):

1. dpdk-devbind refuses to bind active interfaces:
     Warning: routing table indicates that interface is active.
     Not modifying.
   Operators had to `ip link set <ifc> down` on every NIC by hand
   before the bind step succeeded.

2. Reserving 1 GiB hugepages was not enough for rte_eal_init:
     EAL: No available 1048576 kB hugepages reported on node 0/1
   Operators had to `mount -t hugetlbfs -o pagesize=1G nodev /mnt/
   huge1g` themselves before the scanner could find the pages.

Fix:

- bdf_to_iface() walks /sys/bus/pci/devices/<bdf>/net/ to map BDF →
  kernel iface name. cmd_bind invokes `ip link set <ifc> down` on
  each target before invoking dpdk-devbind, with best-effort failure
  semantics (missing ip command, missing iface, already-down iface
  all proceed to the bind).

- ensure_hugetlbfs_mount() mounts a hugetlbfs of the matching pagesize
  at the configured path after a successful nr_hugepages reservation.
  Default targets are /mnt/huge1g (1 GiB) and /mnt/huge2m (2 MiB);
  ANYSCAN_DPDK_HUGEPAGES_1G_MOUNT / _2M_MOUNT override or set them
  to "" to opt out (operators provisioning hugetlbfs via fstab).
  Idempotent: detects existing hugetlbfs of the right pagesize via
  findmnt / /proc/mounts and skips remount.

- ANYSCAN_DPDK_LOAD_ONLY=1 hook lets test-setup-dpdk.sh source the
  script for hermetic helper testing without triggering the argv
  dispatch.

Tests (new tools/test-setup-dpdk.sh):

- cmd_bind invokes `ip link set <ifc> down` BEFORE dpdk-devbind
  --bind=vfio-pci (order verified by line numbers in a single cmd
  log).
- ensure_hugetlbfs_mount calls `mount -t hugetlbfs -o pagesize=1G
  nodev <path>`.
- ensure_hugetlbfs_mount is a no-op when target is already hugetlbfs
  with the matching pagesize.
- ensure_hugetlbfs_mount is a no-op with an empty mount path.
- bdf_to_iface returns iface for a populated fake /sys tree.
- bdf_to_iface returns empty when net/ dir is missing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@skullcrushercmd skullcrushercmd merged commit 2121e1c into main Apr 28, 2026
@skullcrushercmd skullcrushercmd deleted the fix/setup-dpdk-bind-link-down branch April 28, 2026 22:25
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 88f3bdcc4d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/setup-dpdk.sh
Comment on lines +79 to +80
ANYSCAN_DPDK_HUGEPAGES_1G_MOUNT="${ANYSCAN_DPDK_HUGEPAGES_1G_MOUNT:-/mnt/huge1g}"
ANYSCAN_DPDK_HUGEPAGES_2M_MOUNT="${ANYSCAN_DPDK_HUGEPAGES_2M_MOUNT:-/mnt/huge2m}"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve empty mount override when reading env defaults

The script documents that operators can disable auto-mounting by setting ANYSCAN_DPDK_HUGEPAGES_1G_MOUNT / _2M_MOUNT to an empty string, but these assignments use ${VAR:-default} so an empty value is treated as unset and replaced with /mnt/huge1g or /mnt/huge2m. In practice, the opt-out path is unreachable and ensure_hugetlbfs_mount will still try to create/mount those paths, which can cause unwanted mount attempts on hosts where hugetlbfs is managed externally (e.g., via fstab).

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant