Fault injection recipes

Real instruments fail in a handful of typical ways. Each recipe below shows the failure, the scpi-sim fault that reproduces it, and a pytest test for the code that has to cope. All examples use the scpi_bench fixture with the bench of the Browser demo:

pytest --scpi-bench examples/profiles/bench.yaml

import time

import pyvisa
import pytest

OPTS = dict(read_termination="\n", write_termination="\n", timeout=500)


@pytest.fixture
def rm():
    manager = pyvisa.ResourceManager("@py")
    yield manager
    manager.close()

The fault reference (parameters, lifetime) is in the README.

The instrument does not answer

Real life: a meter busy with a long measurement, a lost packet, a firmware hang. Code without a timeout hangs forever; code with one must retry or fail clearly.

def query_with_retry(inst, message, attempts=3):
    for attempt in range(attempts):
        try:
            return inst.query(message)
        except pyvisa.VisaIOError:
            if attempt == attempts - 1:
                raise


def test_one_lost_answer_is_retried(scpi_bench, rm):
    daq = rm.open_resource(scpi_bench.resource("daq"), **OPTS)
    scpi_bench.daq.inject("no_response", count=1)
    assert query_with_retry(daq, "*IDN?").startswith("Keysight")


def test_dead_instrument_gives_up(scpi_bench, rm):
    daq = rm.open_resource(scpi_bench.resource("daq"), **OPTS)
    scpi_bench.daq.inject("no_response", count=10)
    with pytest.raises(pyvisa.VisaIOError):
        query_with_retry(daq, "*IDN?")

no_response drops the answer; the command itself still runs, as when a reply gets lost on the way.

The instrument is slow

Real life: auto-ranging, long integration times, a busy network.

def test_slow_answer_within_timeout(scpi_bench, rm):
    daq = rm.open_resource(scpi_bench.resource("daq"), **OPTS)
    scpi_bench.daq.inject("delay", seconds=0.3, count=1)
    start = time.monotonic()
    daq.query("*IDN?")
    assert time.monotonic() - start >= 0.3


def test_too_slow_is_a_timeout(scpi_bench, rm):
    daq = rm.open_resource(scpi_bench.resource("daq"), **OPTS)   # timeout 0.5 s
    scpi_bench.daq.inject("delay", seconds=1, count=1)
    with pytest.raises(pyvisa.VisaIOError):
        daq.query("*IDN?")

The connection drops

Real life: the instrument reboots, someone pulls the cable, a switch restarts.

def test_reconnect_after_drop(scpi_bench, rm):
    psu = rm.open_resource(scpi_bench.resource("psu"), **OPTS)
    scpi_bench.psu.inject("disconnect")
    with pytest.raises(Exception):     # PyVISA reports a closed socket in several ways
        psu.query("*IDN?")
    psu.close()
    psu = rm.open_resource(scpi_bench.resource("psu"), **OPTS)
    assert psu.query("*IDN?").startswith("Rohde&Schwarz")

The next message after inject("disconnect") closes the connection; the instrument keeps its settings, like a real one behind a reconnected cable.

The wrong instrument answers

Real life: two benches swapped IP addresses, or a replacement unit of a different model. A test station should refuse to run.

def check_identity(inst, model):
    idn = inst.query("*IDN?")
    if idn.split(",")[1] != model:
        raise RuntimeError(f"expected {model}, found {idn}")


def test_station_refuses_wrong_model(scpi_bench, rm):
    psu = rm.open_resource(scpi_bench.resource("psu"), **OPTS)
    scpi_bench.psu.inject("idn", value="Rohde&Schwarz,HMP2020,123,1.0")
    with pytest.raises(RuntimeError, match="HMP2020"):
        check_identity(psu, "HMP4040")

The reading drifts

Real life: an instrument out of calibration, a bad contact, a warm-up effect. Your limit check has to catch it.

def test_offset_trips_the_limit_check(scpi_bench, rm):
    psu = rm.open_resource(scpi_bench.resource("psu"), **OPTS)
    daq = rm.open_resource(scpi_bench.resource("daq"), **OPTS)
    psu.write("INST OUT1; VOLT 12; OUTP:SEL ON;:OUTP:GEN ON")
    scpi_bench.daq.inject("offset", value=0.5)
    volts = float(daq.query("MEAS:VOLT:DC? (@101)"))
    assert not 11.9 <= volts <= 12.1     # the station must report FAIL here


def test_noisy_meter_needs_averaging(scpi_bench, rm):
    daq = rm.open_resource(scpi_bench.resource("daq"), **OPTS)
    psu = rm.open_resource(scpi_bench.resource("psu"), **OPTS)
    psu.write("INST OUT1; VOLT 12; OUTP:SEL ON;:OUTP:GEN ON")
    scpi_bench.daq.inject("noise", sigma=0.01)       # 1 % standard deviation
    readings = [float(daq.query("MEAS:VOLT:DC? (@101)")) for _ in range(50)]
    assert abs(sum(readings) / len(readings) - 12) < 0.1

offset and noise act on multimeters and on the built-in multimeter of a DAQ.

An error appears out of nowhere

Real life: an instrument reports a hardware error, an overtemperature, or a command from another program failed. Code that never reads SYST:ERR? does not notice.

def errors(inst):
    found = []
    while not (entry := inst.query("SYST:ERR?")).startswith("0,"):
        found.append(entry)
    return found


def test_errors_are_collected(scpi_bench, rm):
    psu = rm.open_resource(scpi_bench.resource("psu"), **OPTS)
    scpi_bench.psu.inject("error", code=-222)
    assert errors(psu) == ['-222,"Data out of range"']

Oscilloscopes

A scope has faults of its own: no_trigger, and signal, which changes what is at an input in the middle of a test. See Oscilloscope.

Faults outside Python

In a running bench (scpi-sim serve --bench ... with control: in the bench file) the same faults come from the command line or any program that can write a line of JSON to the control port:

$ scpi-sim fault daq no_response count=1
ok
$ scpi-sim fault --list
psu: -
daq: no_response count=1
$ scpi-sim fault daq --clear
ok

Useful for manual testing of a GUI or test sequencer: start the bench, start your application, and break instruments while you watch what it does.