Hardware Verification

Hardware Verification

A worklist of Warm-TDM changes that are code-complete but still need a pass on real hardware before we can call them done, each with a step-by-step bench procedure. This page exists so a new team member can pick an item, follow the recipe, and record a clear pass/fail.

How to use this page

1. Read Common bench setup once — every procedure assumes it.
2. Pick an item from the table below and open its subpage.
3. Run the steps, fill in the Pass criteria and Record block, and post the result on the item's GitHub issue (and flip its board status off Needs HW Test).

Live status is on the Warm-TDM Roadmap board — the items here are the ones in board status Needs HW Test, plus the two open PRs whose only remaining gate is analog-bench validation. When an item passes, update the board, not this page.

Terminology: items are referenced by GitHub type + number + title (an Issue is the durable what/why; a PR is one how). See AGENTS.md.

Helper scripts: the software-observable parts of some procedures are automated in software/scripts/hwtest/ (on wtj-refactor) — they run the steps a host can judge on its own and print a PASS/FAIL checklist. Where one exists it is called out at the top of the subpage. Anything needing an instrument (a DMM on a load board, a scope) stays a manual step.

Worklist

Item	Title	Subpage	Kind
Issue #24	RS/SQ1FB/SQ1B/SSAFB power on at full scale; RS polarity	Power-on DAC full-scale	Firmware behavior on the bench
Issue #86	FastDacDriver drops force/override writes outside IDLE (also Issue #32)	FastDac override race	Confirm committed SW fix
Issue #55 / PR #79	Software TES bias waveform generator	TES bias waveform generator	New feature, measure rate
Issue #60	Dead-list masking drops the wrong channels	Dead-list masking	Bug fix / feature check
Issue #50	How to check health of the fiber link	Fiber link health	New diagnostic
Issue #36	SSA-curve discontinuities from front-end gain model	SSA gain calibration	Calibration check
Issue #68 / PR #78	Operations refactor — analog-bench tune	Operations refactor tune	Full-stack regression
Issue #69	Adopt surf Ad9681 DDR ADC readout	Ad9681 DDR readout	New readout path (build gated on surf)

Tracked but not yet ready to verify (firmware not built/merged — no procedure here yet): Issue #70 (Floating-point PID firmware — own track, last) and Issue #73 (Wire maxRows to tree sizing — needs the ROW_ADDR_BITS_G register + tree wiring first). Add subpages when those land and reach Needs HW Test.

---

Common bench setup

Every procedure below starts from a running server and a client. Do this once per session.

1. Environment

Firmware builds and the software both live in the warm-tdm repo. For the software side (server, GUI, client, notebooks) use the conda environment with a recent rogue:

cd <warm-tdm>
conda activate warm-tdm-r615      # rogue 6.15.x; the base warm-tdm-env is too old

For firmware builds (only the items that need a rebuild — #69, #70) you must use Vivado 2024.1 (later versions cause hold-time errors):

source /sdf/group/faders/tools/xilinx/2024.1/Vivado/2024.1/settings64.sh

2. Start the server

The server entry points are in software/scripts/. Match the flags to the crate on the bench (board counts, coordinator IP, and — importantly — the row address width the deployed bitfile was built with):

cd software/scripts
# Real hardware, headless (attach a GUI later):
python warmTdmServer.py --ip 192.168.3.11 --columnBoards 1 --rowBoards 1 \
                        --rowAddrBits 8 --maxRows 256
# Real hardware, with GUI:
python warmTdmGui.py    --ip 192.168.3.11 --columnBoards 1 --rowBoards 1
# No hardware (sanity-check the software only — will NOT exercise analog):
python warmTdmGui.py    --emulate

Key warmTdmServer.py / warmTdmGui.py args (from warm_tdm_api/_ArgParser.py): --ip (coordinator board IP, default 192.168.3.11), --columnBoards, --rowBoards, --rowAddrBits (must equal the RTL ROW_ADDR_BITS_G the bitfile was built with), --maxRows, --gui, --emulate, --sim, --pollEn. There is no config-file flag — configs are loaded at runtime (step 4).

Confirm the board is reachable first (ping 192.168.3.11). The server prints the ZMQ port it serves on (default 9099).

3. Connect a client

For scripted / notebook work, use the operations API. In a Jupyter kernel or python:

import warm_tdm_api.operations as ops
sess = ops.connect(host="localhost", port=9099)   # returns the default Session
ops.status()                                       # one-shot state summary
ops.print_hardware()                               # firmware build stamp per board
r     = sess.root                                  # raw pyrogue tree (client mirror)
group = sess.group                                 # the Group device

ops.print_hardware() prints each board's BuildStamp / git hash — record these in every result so a pass is pinned to a specific firmware+software version. A worked end-to-end example lives at software/jupyter/operations_template.py.

4. Choose the enabled set (the anchor)

Almost every analog procedure needs an enabled set chosen first (see docs/design/muxed-run-bringup.md): which columns and rows participate.

group.ColTuneEnable.set([True] * 8)        # which columns participate
group.RowMap1x32()                         # a row map appropriate to your array
group.RowIndexOrderList.set([0, 1, 2, 3])  # logical rows to read out

5. Safe state when you are done

ops.stop_and_zero()   # best-effort: ends the run, drops to manual timing, zeros
                      # column setpoints. NOT a hardware interlock — see Issue #86.

---

Recording a result

For each item, capture:

• Firmware build stamps + git hashes (ops.print_hardware()), software branch/commit (git -C <warm-tdm> describe --long --tags), and the conda env.
• Which board / assembly is on the bench (e.g. "UIUC column module, assembly -01"), and the cryo vs. load-board vs. off-cryo condition.
• The exact config used (enabled columns/rows, timing settings).
• Pass/fail against the criteria on the subpage, with the plots or numbers.

Post it as a comment on the item's issue and update the board status.