Skip to content

HW Verify Issue 86 FastDac Override Race

Jump to bottom
Benjamin Reese edited this page Aug 14, 2026 · 3 revisions

HW Verify — Issue #86: FastDacDriver override-write race (confirm the stop_and_zero fix)

Item: Issue #86 (FastDacDriver drops force/override DAC writes issued outside IDLE — one-shot race). Also closes the loop on Issue #32 (SW zero'ing function). Board status: the software fix is committed on wtj-refactor; this bench pass is its remaining gate. Track: Firmware (with a committed SW mitigation).

← back to Hardware Verification

Helper script: software/scripts/hwtest/verify_stop_and_zero.py automates the readback half — set nonzero → run → stop_and_zero → assert DacCurrentNow ≈ 0 across N cycles.

python verify_stop_and_zero.py --host localhost --port 9099 --cycles 5

A readback of 0 is necessary but not sufficient — the load-board DMM confirmation (Part 2) is what closes Issue #32 and stays manual.

What we are verifying

Writing a force/override value to a fast DAC (Sq1FbForceCurrent, SaFbForceCurrent, Sq1BiasForceCurrent) can silently fail to reach the DAC if the write lands while a run is active or in the window just after EndRun. Root cause (see docs/design/fastdac-override-race.md): overrideWrValid is a one-cycle pulse that FastDacDriver only samples in its IDLE_S state, so a write that arrives mid-sequence is dropped. This is the firmware defect behind the Issue #32 report that "biases don't zero after MUX."

A software mitigation is already committed: operations.stop_and_zero was reordered to end the run, poll TimingTx.Running until it drops, and only then write the zeros — so each override write finds the FSM parked in IDLE_S. It was verified in emulate, but emulate does not clock the DAC FSM against live timing, so the fix is unproven on hardware.

This task confirms the committed fix works on the bench (reproduces swh76's Issue #32 scenario). It does not require the proposed firmware pending-latch — that is a separate, not-yet-implemented hardening.

The two things to show

  1. Reproduce the original failure (optional but valuable): a force-zero issued the old way — during/right after a run — does not move the DAC.
  2. Confirm the fix: stop_and_zero (stop → wait Running==0 → zero) does drive the column force/bias DACs to zero after a real muxed run.

You will need

  • A column module on the bench with its fast-DAC outputs measurable — either a load board + DMM/scope (cleanest, off-cryostat) or the DAC readback registers (below). A load-board measurement is the definitive check.
  • The wtj-refactor software (which carries the reorder).

Procedure

Setup

  1. Start the server on wtj-refactor, connect, and choose an enabled set (see Common bench setup).

  2. Establish a real muxed run so the DAC FSM is actively cycling:

    ops.setup_mux(num_pts=512, sample_end_offset=100, sample_num=250, enable_pid=True)
# confirm it is running:
sess.group.ColumnBoard[0].WarmTdmCore.Timing.TimingTx.Running.get()   # -> True

  3. Set the SQ1 feedback force current to a clearly nonzero value and confirm the DAC actually moved (measure the load-board drop, or read the live readback):

    drv = sess.group.ColumnBoard[0].SQ1Fb           # FastDacDriver device
sess.group.Sq1FbForceCurrent.set([50.0] * len(sess.group.ColTuneEnable.get()))  # µA
drv.DacCurrentNow.get()   # live per-channel readback (µA), pollInterval=1
drv.DacRawNow.get()       # raw DAC codes

    Record the measured output — this is your "DAC is here" baseline.

Part 1 — reproduce the drop (old ordering)

  1. While still running, command a force-zero directly (this is the racy path):

    sess.group.Sq1FbForceCurrent.set([0.0] * len(sess.group.ColTuneEnable.get()))
drv.DacCurrentNow.get()   # did it actually go to ~0?

    Then EndRun and immediately re-issue the zero the old way (no wait):

    sess.group.ColumnBoard[0].WarmTdmCore.Timing.TimingTx.EndRun()
sess.group.Sq1FbForceCurrent.set([0.0] * len(sess.group.ColTuneEnable.get()))
drv.DacCurrentNow.get()   # measure on the load board too

    Expected (bug present): at least sometimes the DAC holds its previous (~50 µA) value instead of zeroing. Note how reliably it reproduces — it is a race, so it may not fail every time.

Part 2 — confirm the fix (stop_and_zero)

  1. Re-establish the nonzero force current and a running mux (repeat steps 2–3).

  2. Now use the fixed helper, which stops, waits for Running to drop, then zeros:

    ops.stop_and_zero()       # stop MUX → poll TimingTx.Running==0 → write zeros

  3. Verify the outputs are actually at zero — on the load board (0 mV across the 100 Ω / differential mid-scale) and via readback:

    sess.group.ColumnBoard[0].SQ1Fb.DacCurrentNow.get()    # ~0 µA
sess.group.ColumnBoard[0].SAFb.DacCurrentNow.get()
sess.group.ColumnBoard[0].SQ1Bias.DacCurrentNow.get()

  4. Repeat step 5–7 several times (≥5). The fix is timing-by-construction, so the value of the test is showing it holds every time across repeated runs.

Pass criteria

  • Baseline: a force write to a running DAC moves it (readback + load board agree).
  • stop_and_zero drives all three column fast DACs (SQ1Fb, SAFb, SQ1Bias) to differential zero after a real muxed run, confirmed on the load board.
  • Zeroing is reliable across ≥5 stop/zero cycles (no residual hold).
  • (Optional) Old-ordering repro of the drop documented for the record.

If stop_and_zero reliably zeros → the committed SW fix is confirmed; note it on Issue #86 and Issue #32, and the firmware pending-latch can stay proposed (not required). If it still drops writes → the firmware pending-latch (option 1 in the design doc) is needed; record the failing sequence.

Record

  • Firmware build stamp + git hash, software commit (wtj-refactor), conda env:
  • Column module / measurement method (load board vs. readback):
  • Part 1 repro result (how often it dropped):
  • Part 2 result (zeroed reliably? cycles tested):

References

  • Issue #86 (firmware root cause + tasks); Issue #32 (original symptom, closed)
  • docs/design/fastdac-override-race.md — full analysis + proposed FW hardening
  • firmware/common/warm_tdm/rtl/FastDacDriver.vhd — the FSM / overrideWrValid
  • firmware/python/warm_tdm/_FastDacDriver.py — OverrideRaw, OverrideCurrent, DacCurrentNow, DacRawNow
  • operations.stop_and_zero — software/python/warm_tdm_api/operations/session.py

Clone this wiki locally