Skip to content

task7_updated

Jump to bottom
Aditya-Sarna edited this page Oct 1, 2026 · 1 revision

7. Document how to reinstall "SERVER DPU HOST ?" / "DPU Host 5"

Goal: return dh5 to a known-good state after testing. A host reinstall plus the BF3 BF-Bundle (task 6) rebuilds both sides. I have not reinstalled dh5. This is the lab procedure plus live details from tasks 1 and 4 (recaptured 2026-10-01).

Host wipe does not flash the DPU. The DPU OS and BMC are not reinstalled. They may bounce when iDRAC resets the chassis, then come back (today: OOB ubuntu@172.22.222.3, BMC root@172.22.222.2). rshim on the host is gone until DOCA-Host is installed again.

Review of existing setup documentation

Source What it gives
Documentation/server-setup.md Manual Ubuntu install, prereqs, sample netplan (not dh5), DHCP
Documentation/idrac_os_installation_steps_virtual_media.md iDRAC reinstall from an ISO on your laptop (Virtual Media)
Documentation/idrac_os_installation_steps_network.md iDRAC reinstall from an ISO on the lab web server (Remote File Share)
Documentation/auto_install_ubuntu_iso.md Build the unattended ubuntu-22.04-autoinstall.iso
Documentation/hosting_files_on_webserver.md Serve files from http://172.22.0.1/...
ansible/install_os.yml Dell iDRAC OS-deploy playbook (dh1bmc, dh4bmc only)
ansible/bios.yml Redfish BIOS attributes + reboot job (hosts: hostbmcs)

The lab reinstalls Dell hosts through iDRAC. dh5 is a PowerEdge R760xa (service tag 7GDGN34) with iDRAC at https://172.22.2.5/, so the Dell/iDRAC path applies. hardware/dh5/README.md still says “R760ax”.

How the reinstall is done in this lab

Three documented ways, all driven from iDRAC (F5 VPN first):

  1. iDRAC Virtual Media — mount the ISO from your laptop, boot it. Simple, slow (~1 h to read the ISO). Manual doc: idrac_os_installation_steps_virtual_media.md.
  2. iDRAC Remote File Share — point iDRAC at an ISO already hosted on the lab web server. The file the network doc names is http://172.22.0.1/ubuntu-22.04.5-live-server-amd64.iso. The autoinstall ISO is not listed as hosted. Host it first if you want that file over RFS or Ansible. Doc: idrac_os_installation_steps_network.md.
  3. Ansible install_os.yml — automated Dell OS deployment over Redfish. Caveat below.

install_os.yml caveat: hosts: dh1bmc, dh4bmc. There is no dh5bmc in ansible/inventory [hostbmcs], so the play never sees this iDRAC. Role vars already set is_custom_iso: true and iso_name: ubuntu-22.04-autoinstall.iso (fetched from mgmt = http://172.22.0.1/ubuntu-22.04-autoinstall.iso). os_name: RHEL / os_version: 9 is the Dell-module workaround (playbook comment + issue 780; server-setup.md also cites PR 784). Those ISO vars are already the custom Ubuntu ISO. To automate dh5: add dh5bmc (172.22.2.5, ansible_user=root) to inventory and to the play hosts: line, and put ubuntu-22.04-autoinstall.iso on the mgmt web server. BIOS attrs belong in the inventory for bios.yml, not this play. Until then the reliable path is manual iDRAC.

I do not assume PXE: repo dhcpd.conf has no host dh5 and no dh5 iDRAC reservation (only the unknown pool 172.22.222.1–254). iDRAC is the reachable management path.

OS image and version

  • Lab docs pin Ubuntu 22.04 LTS server, x86-64. Hosted stock ISO: ubuntu-22.04.5-live-server-amd64.iso (http://172.22.0.1/...). server-setup.md still names the older ubuntu-22.04.1-live-server-amd64.iso; the network/hosting docs are the 22.04.5 file.
  • Unattended build: ubuntu-22.04-autoinstall.iso from auto_install_ubuntu_iso.md (7z + xorriso; hash the password with openssl passwd -6). Sample ansible/iso_files/server/user-data is hostname/user opi, storage name: lvm, and password: "hashedpassword" (placeholder — replace before building).
  • Note vs live: dh5 currently runs Ubuntu 24.04.5 (6.8.0-142-generic, hostname dh5). Reinstall to 22.04 to match the docs, or agree to bump the docs to 24.04. Pick one before flashing. DOCA-Host must match the Dell OS you just installed, not the DPU Arm OS.

Boot / storage / BIOS / network / remote-management config

  • Remote management: iDRAC https://172.22.2.5/ (or ssh root@172.22.2.5). Dell root password (1Password). Virtual Console for the screen.
  • Boot: Virtual Console → Boot → virtual CD/DVD first, then Power → Reset System (warm reboot).
  • BIOS: hardware/dh5/README.md says power always on. bios.yml applies inventory bios_attributes to [hostbmcs] (Dell pattern on dh4bmc: AcPwrRcvry: On, ProcVirtualization: Enabled). dh5bmc is missing, so that play has not run on this chassis. Confirm always-on and virtualization in iDRAC if you rebuild.
  • Storage: iDRAC install pages say disable LVM. server-setup.md marks that optional. Repo autoinstall user-data sets name: lvm. Those disagree — pick one; do not assume the ISO follows the manual page.
  • Network: installer uses DHCP. Live host is eno12399np0 172.22.1.5/16. There is no repo reservation; after a wipe it may come back as 172.22.222.x (it was 172.22.222.14 once). Do not assume 1.5. Enable OpenSSH server (autoinstall already has install-server: true).
  • Profile: docs / autoinstall use opi / opi. Live login today is root@dh5. server-setup.md optionally sets PermitRootLogin yes. Do not apply the sample netplan in that file (10.36.118.210 / ens160); that is not this host. iDRAC docs: do not install extra packages (Docker, etc.) during the installer.
  • After install, disconnect the virtual media / remote share and reboot.

NVIDIA / DOCA host components to install afterwards

Match the host to the BF3 BF-Bundle generation (tasks 4/5/6). Live pair today is DOCA 3.5 / OFED 26.07. Host OS and Arm OS do not have to be the same number.

On the new host, from DOCA Downloads, same release row as the BFB:

sudo dpkg -i doca-host_*_ubuntu<ver>_amd64.deb    # <ver> = 2204 or 2404 for the OS you installed
sudo apt-get update
sudo apt-get -y install doca-all

doca-all is what is on the live host (Level 2 vs BF-Bundle). Then reboot and start rshim as a userspace service (modprobe rshim fails on this host):

systemctl enable --now rshim

Target versions (live install, 2026-10-01 — use these as the known-good pin):

Component Live known-good
doca_caps --version DOCA 3.5.0098
ofed_info -s OFED-internal-26.07-0.7.7
doca-all / doca-ofed 3.5.0-082000
rshim package 2.8.5-1, service active; /dev/rshim0/misc = BlueField-3, DPU mode
MFT 4.37.0-154
NIC (ethtool -i enp13s0f0np0) mlx5_core 2607.0.100, firmware 32.50.1002 (0000:0d:00.0)

Install DOCA on the host first, then the BF-Bundle on the DPU (task 6). Host-only rebuild: skip the BFB; DPU /etc/mlnx-release stays bf-bundle-3.5.0-89_26.07_ubuntu-24.04_64k_prod. A BFB flash can change the OOB address and the ubuntu password (1Password).

Verify the fresh host detects the BlueField-3

On the new host after reinstall + DOCA (live today: root@172.22.1.5, hostname dh5; autoinstall would be opi@ whatever DHCP gave you):

lspci | grep -i bluefield
# expect: 0d:00.0 / 0d:00.1 ConnectX-7 ; 0d:00.2 SoC Management
mst start && mst status -v
# expect: BlueField3 (rev 1) at 0d:00.0/.1  → enp13s0f0np0 / enp13s0f1np1
systemctl start rshim
cat /dev/rshim0/misc
# expect: DEV_INFO BlueField-3(Rev 1), BF_MODE DPU mode, DEV_NAME pcie-0000:0d:00.2

Verify DOCA host-to-DPU communication again

ip link set tmfifo_net0 up

Clone this wiki locally