Skip to content

task 6 updated

Jump to bottom
Aditya-Sarna edited this page Oct 1, 2026 · 2 revisions

6. Document how to install a BF-Bundle on BlueField-3

I did not flash a bundle on dh5. I read NVIDIA’s current install docs, compared them with the BlueField-2 recipe in this lab repository, and wrote a repeatable BF3 procedure from those sources. Mircea installed DOCA 3.5 on the host and the DPU. Recaptured 2026-10-01 on root@dh5 and ubuntu@172.22.222.3. The BF3 ubuntu password is in 1Password.

Check What I did Evidence
NVIDIA recommended install Read current DOCA pages Quotes and URLs below
Compare with lab BF2 Read hardware/dh2/README.md and ansible/firmware.yml Commands copied from those files
BF3-specific differences Same docs + live PCIe/rshim from tasks 4/6 Table below
Run bfb-install myself Not run No installer log from me
Host DOCA after 3.5 Recaptured 2026-10-01 on root@dh5 doca_caps DOCA 3.5.0098; ofed_info -s OFED-internal-26.07-0.7.7. Ubuntu 24.04.5, kernel 6.8.0-142-generic
Host still sees the DPU Recaptured 2026-10-01 lspci BF3 at 0d:00.0/.1/.2; rshim DEV_INFO BlueField-3(Rev 1), BF_MODE DPU mode; rshim ping 2/2; rshim SSH → localhost.localdomain
DPU OS / mlnx-release / bfver Recaptured 2026-10-01 on ubuntu@172.22.222.3 (sudo bfver) Ubuntu 24.04.5, 6.8.0-1030-bluefield-64k. /etc/mlnx-release bf-bundle-3.5.0-89_26.07_ubuntu-24.04_64k_prod. ATF 4.16.0-8, UEFI 4.16.0-48, BSP 4.16.0.14075

OOB SSH host key for 172.22.222.3 changed (ED25519 SHA256:XTzDfcnjS/3Md+81CuFC5rRJdFKimaWwp8xJWBklooE). After ssh-keygen -R 172.22.222.3 the login is the same bundle string as above. Earlier “SSH refused / no password / DPU blocked” is stale.

NVIDIA’s current recommended procedure

I used BF-Bundle Installation and Upgrade (DOCA 3.5 archive of that page) and DOCA Installation Guide for Linux. Files come from DOCA Downloads.

NVIDIA’s order:

To deploy the BF-Bundle software on the BlueField networking platform (DPU or SuperNIC), you must:

  1. Install or upgrade DOCA on the host side.
  2. Install or upgrade BF-Bundle on the BlueField device.

Make sure to update DOCA on the host side before installing the BF-bundle on the BlueField.

The install command NVIDIA documents:

host# sudo bfb-install --rshim rshim<N> --bfb <image_path.bfb>

If there is one card, they say use rshim0. List devices with ls -la /dev/ | grep rshim.

To confirm the card finished booting, NVIDIA says wait about 90 seconds after the installer exits, then:

host# sudo cat /dev/rshim<N>/misc

and look for Linux up and DPU is ready.

Optional bf.cfg with ubuntu_PASSWORD=… sets the ubuntu password. NVIDIA: if --config is not used, the first login prompts to change the default ubuntu password.

BF3-only warning on the same page:

The ATF will not boot 150W BlueField-3 platforms if the ATX +12V power supply is not connected.

NVIDIA also lists apt/yum on the DPU (upgrade without overwriting the OS) and ISO/PXE. A full rebuild is the BF-Bundle path above. Customer-affecting notes say DOCA 3.5 is the last release with BFB-Install; 3.6 switches to doca-installer.

What the lab repository already does for BlueField-2

I read hardware/dh2/README.md. The documented flash is:

wget https://content.mellanox.com/BlueField/BFBs/Ubuntu22.04/bf-bundle-2.7.0-33_24.04_ubuntu-22.04_prod.bfb
cat bf-bundle-2.7.0-33_24.04_ubuntu-22.04_prod.bfb > /dev/rshim0/boot

Rshim login in the same file:

ifconfig tmfifo_net0 192.168.100.1/30 up
ping 192.168.100.2
ssh ubuntu@192.168.100.2

/dev/rshim0/misc in that README says DEV_INFO BlueField-2(Rev 1) and DEV_NAME pcie-0000:0f:00.2.

I read ansible/firmware.yml (play Nvidia | Upgrade Nvidia BF2 FWs, hosts: bf2; the flash block Nvidia | Update BlueField using BFB image from the Host is delegate_to: dh2). It uses the same 2.7 filename and still runs cat {{ bfb_local_file }} > /dev/rshim0/boot. A TODO in that file mentions bfb-install and was not applied.

hardware/dh5/README.md has no BFB install steps (stub: “Nvidia Blue Filed 3”).

BlueField-3-specific differences (with source)

Difference Evidence
Do not use the dh2 cat … > /dev/rshim0/boot 2.7 file on this card dh2 README uses bf-bundle-2.7.0-33_…. NVIDIA install for current DOCA is bfb-install. That 2.7 image is the BF2 lab pin, not the 3.5 pair
Use bfb-install, not cat NVIDIA quote above. ansible TODO agrees, but BF2 playbook still cats
Card / PCIe Live lspci on dh5: BF3 ConnectX-7 at 0d:00.0/.1, SoC mgmt 0d:00.2. dh2 README: BF2 at 0f:00
rshim is a userspace service on dh5 modprobe rshim failed; systemctl start rshim worked. Package now rshim 2.8.5-1
150 W BF3 needs +12 V ATX NVIDIA BF-Bundle page, quoted above
Host OS Live dh5 is Ubuntu 24.04.5 (6.8.0-142-generic). dh2 BFB URL is Ubuntu 22.04 Arm DOCA 2.7. Host DOCA repo must match the Dell OS
After install, OOB may exist as well as rshim Live OOB ubuntu@172.22.222.3 (oob_net0 MAC 9c:63:c0:e4:b7:f4). BMC root@172.22.222.2 (dpu-bmc)

How I would install (repeatable; I have not run this on dh5)

On root@dh5 (172.22.1.5).

  1. Host DOCA first (NVIDIA). Confirm doca_caps --version matches the BFB release (live host is already 3.5.0098).
  2. systemctl enable --now rshim; cat /dev/rshim0/misc must say BlueField-3.
  3. Download BF-Bundle 3.5 for this card from DOCA Downloads. Record filename and SHA256. Do not use bf-bundle-2.7.0-33_….
  4. Optional: openssl passwd -1 and ubuntu_PASSWORD=… in bf.cfg so the DPU password is known (NVIDIA Option 2). Store the password in 1Password, not git.
  5. bfb-install --rshim rshim0 --bfb /root/<file>.bfb --config /root/bf.cfg
  6. cat /dev/rshim0/misc until DPU is ready.
  7. ip link set tmfifo_net0 up; ip addr add 192.168.100.1/30 dev tmfifo_net0; ping 192.168.100.2; ssh ubuntu@192.168.100.2. Also check OOB ssh ubuntu@172.22.222.3.

Verify OS, DOCA, firmware

Host (recaptured root@dh5, 2026-10-01):

doca_caps --version   → DOCA Version: 3.5.0098
ofed_info -s          → OFED-internal-26.07-0.7.7
uname -r              → 6.8.0-142-generic
ethtool -i enp13s0f0np0
                      → mlx5_core 2607.0.100
                        firmware-version: 32.50.1002 (MT_0000000884)
                        bus-info: 0000:0d:00.0

ethtool is the host ConnectX-7 NIC view, not DPU bfver. Earlier 32.40.1000 / kernel 6.8.0-139 are stale.

DPU (recaptured ubuntu@172.22.222.3, 2026-10-01):

hostname              → localhost.localdomain
cat /etc/os-release   → Ubuntu 24.04.5 LTS (noble)
uname -a              → 6.8.0-1030-bluefield-64k #35-Ubuntu … aarch64
cat /etc/mlnx-release → bf-bundle-3.5.0-89_26.07_ubuntu-24.04_64k_prod
sudo bfver            → /dev/mmcblk0boot0
                        BlueField ATF version: v2.2(release):4.16.0-8-gda930bca9
                        BlueField UEFI version: 4.16.0-48-g3e1cb60aa0
                        BlueField BSP version: 4.16.0.14075
                        OS Release Version: bf-bundle-3.5.0-89_26.07_ubuntu-24.04_64k_prod

That bundle is DOCA 3.5.0 / OFED 26.07 on Ubuntu 24.04 Arm 64k. It matches the host generation. The pre-3.5 rshim string DOCA_2.6.0_BSP_4.6.0_Ubuntu_22.04-5.24-01.prod and the mentor 24.04.4 screenshot are stale. bfver as ubuntu (no sudo) fails.

Verify the host still detects and talks to the DPU

On root@dh5, 2026-10-01:

lspci | grep -i bluefield
  0d:00.0 / 0d:00.1  BlueField-3 integrated ConnectX-7
  0d:00.2            BlueField-3 SoC Management Interface
cat /dev/rshim0/misc
  DEV_INFO  BlueField-3(Rev 1)
  BF_MODE   DPU mode
  DEV_NAME  pcie-0000:0d:00.2
  INFO[MISC]: Linux up
  INFO[MISC]: DPU is ready
ping -c 2 192.168.100.2   → 2/2
ping -c 2 172.22.222.3    → 2/2
ssh ubuntu@192.168.100.2 hostname → localhost.localdomain

OOB: ssh ubuntu@172.22.222.3. BMC: ssh root@172.22.222.2 → root@dpu-bmc. Planned 172.22.3.5 / 172.22.4.5 are unused.

Status

I have NVIDIA’s procedure, the lab BF2 comparison, and BF3 differences, all with sources. I did not run bfb-install. Host and DPU are a 3.5 / 26.07 pair: host DOCA 3.5.0098 / OFED-internal-26.07-0.7.7, DPU bf-bundle-3.5.0-89_26.07_ubuntu-24.04_64k_prod, BSP 4.16.0.14075. OOB and rshim login both work. Password is in 1Password.

Clone this wiki locally