HW Verify Issue 60 Dead List Masking

HW Verify — Issue #60: Dead-list masking drops the wrong channels

Item: Issue #60 (Dead list bug). Board status: Needs HW Test · Track: Software · Priority: P1.

← back to Hardware Verification

Helper script: software/scripts/hwtest/verify_dead_masks.py automates this whole procedure (baseline → apply masks → masked → compare) and prints a PASS/FAIL. It is fully software-verifiable — no instrument needed.

python verify_dead_masks.py --host localhost --port 9099 --cols c4r3,c4r19,c5r58

The manual steps below explain what it does and how to isolate a failure.

What we are verifying

Dead-channel masking is supposed to remove only the channels you name from the readout. The bug report: after masking c4r3, c4r19, c5r58, c6r0, c6r58 (per-column row-enable bitmasks), the data file was missing a different set of channels (c4r10, c4r58, c5r0, c6r0, c6r1) than the ones requested — i.e. the mask bit→(col,row) mapping appears off.

The operations refactor added a clean path for this — the make_dead_masks helper (parses c<col>r<row> strings into {col: mask}) and Session.apply_dead_masks, which writes each mask to AdcDsp[chan].RowEnableMask. This task confirms, on hardware, that naming a channel dead removes exactly that channel from the stream — no more, no less.

The likely culprit is an indexing/bit-order mismatch somewhere in mask-string → bitmask → RowEnableMask bit → the row field written into the data file. The test is designed to expose that mapping directly.

You will need

• A working muxed readout on the bench (real tune not required — you just need channels producing data so you can see which drop out). A cryostat with live SQUIDs is ideal but any configuration that yields per-(col,row) data works.
• Enough columns/rows enabled that the masked and unmasked channels are distinguishable in the file.

Procedure

Setup

1. Start the server, connect, choose an enabled set with several columns and rows (see Common bench setup). Use a row map/order broad enough to include the channels you will mask.

2. Get a muxed run going and take a baseline file with no masks:

   ops.setup_mux(num_pts=512, sample_end_offset=100, sample_num=250, enable_pid=True)
   baseline = ops.take_data(acq_time_sec=10.0)

3. List the (col,row) channels present in the baseline file — this is your "everything on" reference set:

   ops.plot_stream_data("c*r*", stream_data_id=baseline)   # shows which channels have data

Part 1 — mask a known set

4. Build masks for a small, specific set and apply them:

   from warm_tdm_api.operations import make_dead_masks
   dead = ['c4r3', 'c4r19', 'c5r58', 'c6r0', 'c6r58']   # the Issue #60 set
   masks = make_dead_masks(dead, ncol=8, nrow=256)       # -> {col: bitmask}
   sess.apply_dead_masks(masks)                          # writes AdcDsp[chan].RowEnableMask

5. Read back the masks actually on hardware and confirm they match masks:

   for col in {int(c.split('r')[0][1:]) for c in dead}:
       cb, chan = sess.group.ColumnBoard[0], col   # single-column-board case; else map col->board/chan
       print(col, hex(cb.DataPath.AdcDsp[chan].RowEnableMask.get()))

   (For a multi-column-board crate, use col_to_board_chan(col) to find the right ColumnBoard/AdcDsp index.)

Part 2 — take masked data and compare

6. Take a masked file and list its channels:

   masked = ops.take_data(acq_time_sec=10.0)
   ops.plot_stream_data("c*r*", stream_data_id=masked)

7. Compare the two channel sets. Compute exactly which (col,row) disappeared:

   dropped = baseline_channels − masked_channels
   

   The test passes iff dropped == the set you masked (dead), with every other channel still present.

Part 3 — isolate the mapping (if it fails)

8. If dropped != dead, mask a single channel (e.g. just c4r3) and repeat. The pair (requested channel, actually-dropped channel) pins the offset — e.g. a fixed row shift, an inverted bit order, or a col/row swap. Record several single-channel cases so the mapping bug is fully characterized.

   Relevant mapping points to inspect: make_dead_masks bit assignment (operations/channels.py), the RowEnableMask bit → servo-row correspondence in AdcDsp, and the row/col fields written per sample in _DataFormats.py (DataSample.row = arr[4], col = arr[5]).

Pass criteria

• Baseline channel set captured with no masks.
• apply_dead_masks readback matches the intended {col: mask}.
• The set of channels that disappear equals exactly the masked set — nothing extra dropped, nothing masked-but-still-present.
• (If failing) single-channel cases recorded that pin the mapping offset.

Record

• Firmware build stamp + git hash, software commit, conda env:
• Enabled columns/rows and row map used:
• Baseline vs. masked channel lists; computed dropped set:
• Pass/fail; if fail, the requested→dropped mapping table:

References

• Issue #60 (bug report with masks + missing-channel evidence)
• software/python/warm_tdm_api/operations/channels.py — make_dead_masks, get_row_col, col_to_board_chan
• software/python/warm_tdm_api/operations/session.py — apply_dead_masks
• firmware/python/warm_tdm/_AdcDsp.py — RowEnableMask (256-bit, offset 0x60)
• firmware/python/warm_tdm/_DataFormats.py — per-sample row/col fields