Closing behavioral gaps between pythonSoftIOC and traditional C IOCs: field-change callbacks, universal set(), and optional analog conversion

[BoredDadDev]

Hi everyone,
Thanks for creating and sharing pythonSoftIOC -- I find it intuitive, well designed, and easy to use; but it can always be improved, like anything. Hence, I'd like to discuss a few potential additions to the API and behavior, before submitting PRs. Thank you in advance for taking the time to consider this.

I'm a Controls Analyst at the Canadian Light Source (CLS), and I've been using pythonSoftIOC and working to introduce Python-based IOCs to our team. The main barrier is that experienced EPICS developers have doubts when PVs created with pythonSoftIOC don't behave the same way as PVs hosted by a traditional C IOC.

I believe that by closing some of these behavioral gaps, we can make pythonSoftIOC significantly easier to adopt by traditional EPICS programmer.

I have working code for all three proposals below and can submit PRs. I'd like to discuss the ideas first to get feedback before doing so.

---

Executive Summary (TL;DR)

Three additive, backward-compatible changes -- no modifications to EPICS Base, no breaking changes to existing applications:

1. on_field_change(field, callback) -- lets Python react when a CA/PVA client writes to any record field (SCAN, PROC, DRVL, DRVH, DESC, ...). Uses asTrapWriteRegisterListener, the same hook as the existing caput logger. ~200 lines added.

2. Universal set() -- set() now works regardless of the SCAN setting. Currently it silently does nothing when SCAN is not "I/O Intr". The fix: check scanIoRequest()'s return value and fall back to scanOnce() when the I/O Intr list is empty. ~20 lines changed.

3. Optional analog conversion (convert=True) -- builder.aIn() / builder.aOut() gain an opt-in convert=True flag that lets the EPICS RVAL->VAL conversion chain run (LINR, ESLO, EOFF, SMOO, breakpoint tables). Default is False -- fully backward compatible. Includes a egu_to_raw() helper for the inverse calculation. ~130 lines added.

Test results: 387 passing, 16 skipped (Windows-only). All three proposals have dedicated integration tests.

---

The current limitations

I understand that the current behavior is a known design consequence of how EPICS records are used in Python-based IOCs. EPICS provides the communication layer and the record infrastructure, but in pSIOC the "driving engine" is Python -- not the EPICS database scanning engine. PVs are essentially shared network variables, and many of the fields that control the original EPICS scanning and processing pipeline are not connected in any meaningful way to the Python side.

For example, in my applications I use periodic asyncio tasks for updates (say, every 1 second). The SCAN field, however, stays at I/O Intr -- which is a requirement for set() to publish immediately. If an operator or another application changes SCAN to something else, set() silently stops working. And if someone writes to .PROC on a Passive record expecting fresh data, they get whatever stale value was last stored -- Python had no way to know processing was requested.

Here are the specific concerns I hear from colleagues used to traditional EPICS:

1. SCAN field is misleading. It says "I/O Intr" but the actual update rate is controlled by Python code that the client can't see or influence through the PV fields. Changing SCAN to "1 second" or "Passive" either does nothing (if DISP=1) or silently breaks the record.

2. PROC doesn't trigger a fresh read. In a traditional IOC, writing to .PROC causes the record to process -- device support reads from the source, and the record publishes a fresh value. In pSIOC, .PROC does trigger dbProcess(), but Python has no notification that processing was requested, so there's no opportunity to read fresh data before the record publishes.

3. No way to react to many field changes. If a client writes to fields like SCAN, PROC, DRVL, DRVH, DESC, etc., Python code has no notification mechanism. The existing on_update callback only fires for VAL writes on output records.

Although it's possible to work around these issues with different patterns than a traditional IOC, an IOC should behave predictably for standard operations regardless of how it is implemented.

What I'd Like to Propose

I've investigated the EPICS Base and pythonSoftIOC code, and found approaches that address these concerns without modifying EPICS Base and without breaking any existing behavior. I'd like to discuss three changes.

---

Proposal 1: Field-Change Callbacks (on_field_change)

The gap: Python code cannot react when a CA or PVA client writes to many record fields -- including operationally important ones like SCAN, PROC, DRVL, DRVH, DISA, and others. The existing callbacks (on_update and validate) only cover VAL writes on output records.

The approach: Register a C-level asTrapWriteListener that fires after every external field write, then forward the notification to per-record, per-field Python callbacks.

Resulting Python API:

record = builder.aIn("TEMPERATURE", initial_value=25.0)

def on_scan_changed(record_name, field_name, new_value):
    log.info(f"{record_name}.{field_name} -> {new_value}")

record.on_field_change("SCAN", on_scan_changed)
record.on_field_change("*", log_all_writes)  # wildcard for any field

# Multi-field subscription -- one callback for several fields:
record.on_field_change(["DRVH", "DRVL", "HIHI"], on_limits_changed)

# De-registration:
record.remove_field_callback("SCAN", on_scan_changed)  # one
record.clear_field_callbacks("DRVH")    # all callbacks on a field
record.clear_field_callbacks()           # all callbacks on the record

# Read-only view of registered callbacks:
record.field_callbacks  # -> {"SCAN": [...], "*": [...], ...}

The most obvious use cases are SCAN and PROC -- allowing Python to adjust its update strategy when SCAN changes, or to read fresh data when PROC is poked. But on_field_change is general-purpose and enables many other patterns:

• Forward DRVL/DRVH changes to hardware limit switches
• Audit logging with user/host identification (available from the asTrapWriteMessage)
• React to DISA (disable) changes for interlock management
• Propagate DESC updates to a naming service
• Any field a client might write that Python needs to act on

How it works:

The mechanism uses asTrapWriteRegisterListener() -- the same API that the existing caput logger (EpicsPvPutHook) already uses. A new C function (FieldWriteHook) is registered as a second listener. It fires after the field write completes, reads the new value as DBR_STRING, acquires the GIL, and calls a single Python dispatch function. The Python layer demultiplexes by record name and field name to the registered callbacks.

static void FieldWriteHook(struct asTrapWriteMessage *pmessage, int after)
{
    if (!after || !py_field_write_callback)
        return;

    struct dbChannel *pchan = pmessage->serverSpecific;
    const char *channel_name = dbChannelName(pchan);  // e.g. "MYREC.SCAN"

    char value_str[MAX_STRING_SIZE + 1];
    long len = 1, opts = 0;
    dbGetField(&pchan->addr, DBR_STRING, value_str, &opts, &len, NULL);

    PyGILState_STATE gstate = PyGILState_Ensure();
    PyObject *result = PyObject_CallFunction(
        py_field_write_callback, "ss", channel_name, value_str);
    Py_XDECREF(result);
    if (PyErr_Occurred()) PyErr_Print();
    PyGILState_Release(gstate);
}

Backward compatibility: Fully backward compatible. The hook only fires if a Python callback has been registered via a new register_field_write_listener() call made during IOC startup. Records with no registered callbacks are unaffected. The existing caput logger (EpicsPvPutHook) continues to work -- asTrapWrite supports multiple listeners.

Performance impact: One dbGetField (DBR_STRING) + GIL acquire + Python function call per external field write. The GIL is released immediately after dispatch. Records without registered callbacks incur only the C-level null-pointer check (if (!py_field_write_callback) return;). No impact on internal set() calls or record processing -- asTrapWrite only fires for CA/PVA client writes.

The printf caput logger (EpicsPvPutHook) can be disabled by passing log_puts=False to iocInit() to eliminate per-put console overhead while still loading the access-security file needed for field-change callbacks.

---

Proposal 2: Universal set() via scanOnce()

Proposal 1 opens the door to reacting to SCAN changes -- Python can now detect that a client changed SCAN from "I/O Intr" to "5 second" and adjust the source polling rate accordingly. But this creates a new problem: once the record is no longer on I/O Intr, set() breaks. set() calls trigger(), which calls scanIoRequest(), and scanIoRequest() only processes records on the I/O Intr scan list. If we let users meaningfully change SCAN (which Proposal 1 makes possible), we must also make set() work regardless of the SCAN setting.

Root cause: trigger() uses scanIoRequest() exclusively. This function iterates the I/O Intr scan list -- when the record has been moved to a different scan list (Passive, periodic, Event), the list is empty and scanIoRequest() is a no-op. The value stored by set() sits unpublished until something else triggers dbProcess().

The approach: Use scanIoRequest()'s return value to detect whether it actually queued processing. scanIoRequest() returns a non-zero bitmask if records were queued, or zero if the scan list was empty. If it returns zero (meaning the record is not on I/O Intr), fall back to scanOnce() -- a public EPICS Base API declared in dbScan.h -- which queues the record for immediate processing regardless of SCAN setting.

From EPICS Base dbScan.c:

// scanIoRequest returns 0 when the I/O Intr scan list is empty
unsigned int scanIoRequest(IOSCANPVT piosh)
{
    unsigned int queued = 0;
    for (prio = 0; prio < NUM_CALLBACK_PRIORITIES; prio++) {
        if (ellCount(&piosh->iosl[prio].scan_list.list) > 0)
            if (!callbackRequest(&piosh->iosl[prio].callback))
                queued |= 1 << prio;
    }
    return queued;
}

// scanOnce always works, regardless of SCAN
int scanOnce(struct dbCommon *precord) {
    return scanOnceCallback(precord, NULL, NULL);
}

The change to trigger():

def trigger(self):
    '''Trigger immediate record processing.

    Uses scanIoRequest for I/O Intr records (fast path).
    Falls back to scanOnce for records on any other SCAN setting.
    '''
    queued = False
    if self.__ioscanpvt:
        queued = imports.scanIoRequest(self.__ioscanpvt)
    if not queued and hasattr(self, '_record') and self._record is not None:
        imports.scan_once(self._record)

Only one path fires:

• When SCAN = "I/O Intr": scanIoRequest returns non-zero -> queued is truthy -> scanOnce is skipped. This is the existing fast path, unchanged.
• When SCAN = anything else: scanIoRequest returns 0 (scan list empty) -> queued is falsy -> scanOnce fires, queuing immediate processing.

No duplicate processing. No duplicate monitors. One path or the other, never both.

The C wrapper for scanOnce is minimal:

#include <dbScan.h>

static PyObject *scan_once(PyObject *self, PyObject *args)
{
    long record_ptr;
    if (!PyArg_ParseTuple(args, "n", &record_ptr))
        return NULL;
    dbCommon *precord = (dbCommon *)record_ptr;
    Py_BEGIN_ALLOW_THREADS
    scanOnce(precord);
    Py_END_ALLOW_THREADS
    Py_RETURN_NONE;
}

The scanIoRequest wrapper also needs a small change -- currently it discards the return value (restype = None). The fix is to set restype = c_uint so Python can see whether processing was actually queued.

Behavior after the change:

SCAN setting	set() before	set() after
I/O Intr	Immediate	Immediate (unchanged, same code path)
Passive	❌ Broken -- value stored but never published	✅ Immediate via scanOnce()
1 second	⚠️ Delayed up to 1s	✅ Immediate via scanOnce()
5 second	⚠️ Delayed up to 5s	✅ Immediate via scanOnce()
Event	⚠️ Delayed until next event	✅ Immediate via scanOnce()

The SCAN field shows its true value to clients at all times. There is no interception, no write-back, no flicker. If a client sets SCAN to "5 second", it stays "5 second". Python's set() publishes immediately via scanOnce(), and the EPICS periodic scan also fires every 5 seconds as expected. Both mechanisms coexist cleanly because dbScanLock serializes access and recGblCheckDeadband() prevents duplicate monitors when the value hasn't changed.

Backward compatibility: Fully backward compatible. The scanIoRequest path is unchanged -- just its return value is now captured. The scanOnce call only fires when scanIoRequest didn't queue anything. All existing applications that use SCAN = "I/O Intr" (the current default) will see no behavioral difference.

Performance impact: One scanOnce() call per set() when SCAN is not "I/O Intr". This queues a dbProcess() on the onceTask thread -- the same cost as any single record processing. When SCAN IS "I/O Intr", no additional cost at all (only scanIoRequest fires, as before).

Combined with Proposal 1, this enables the full pattern:

record = builder.aIn("TEMPERATURE", initial_value=25.0)

async def poll_temperature():
    while True:
        value = await read_from_hardware()
        record.set(value)           # Always publishes immediately, any SCAN
        await asyncio.sleep(rate)   # Python controls the actual polling rate

def on_scan_changed(rec_name, field, new_value):
    # Client changed SCAN -- adjust our async polling rate to match
    nonlocal rate
    rate = scan_to_seconds(new_value)  # e.g. "5 second" -> 5.0

record.on_field_change("SCAN", on_scan_changed)

Now the SCAN field genuinely controls the update rate: the client changes SCAN, Python is notified via on_field_change, adjusts its polling, and every set() publishes immediately via scanOnce(). The EPICS periodic scan also fires at the expected rate. From the client's perspective, this record behaves exactly like a record in a traditional C IOC.

---

Proposal 3: Optional RVAL/Conversion Support for ai/ao (considering adding)

The gap: pythonSoftIOC's ai and ao device support returns NO_CONVERT (rc=2) from _process(). This bypasses the entire RVAL<->VAL conversion chain. About 15 standard fields on ai/ao records are therefore inert: LINR, ESLO, EOFF, EGUF, EGUL, ROFF, ASLO, AOFF, SMOO (ai), OROC/OVAL (ao).

This is by design for the typical case where Python already computes engineering-unit values. But it creates a barrier for:

• Hardware-facing IOCs where Python reads raw ADC counts and the calibration (LINR/ESLO/EOFF) should live in the EPICS record -- adjustable at runtime by operators without restarting the IOC.
• Migration from C IOCs where existing records already have LINR/ESLO/EOFF/SMOO calibrations that operators know how to use.
• Breakpoint tables -- EPICS's built-in typeKdegC, typeJdegC, etc. are simply inaccessible without this.

The approach: An optional convert=True parameter on builder.aIn() / builder.aOut(). Default is False -- fully backward compatible.

# Python provides raw ADC counts (integers) -> EPICS applies calibration
sensor = builder.aIn("TEMPERATURE",
    convert=True,
    LINR="SLOPE",
    ESLO=0.001,    # VAL = RVAL * 0.001 + EOFF
    EOFF=0.0,
    SMOO=0.1,
    EGU="kPa",
)
sensor.set(32768)             # writes RVAL=32768 -> C converts -> VAL=32.768
sensor.get()                  # -> 32.768  (engineering, what clients see)
sensor.get(raw=True)          # -> 32768.0 (float -- ai stores values as c_double
                              #             regardless of the type passed to set())

convert is a mutable property -- can be toggled at any time after iocInit. get(raw=False/True) is defined on all record types for API consistency (get(raw=True) is a no-op for non-convert records).

Implementation: ai._process() writes int(value) to RVAL and returns EPICS_OK instead of NO_CONVERT. The C aiRecord.c conversion chain runs unchanged -- not a single line of EPICS Base is modified. For ao, init_record returns EPICS_OK so the C convert() runs at initialization; it runs unconditionally on every process cycle regardless.

Important design note: This is only meaningful when Python provides integer-scale values (ADC counts, DAC codes, register integers). If Python already computes engineering-unit floats, convert=False (default) is correct -- the float-to-int cast in RVAL would lose sub-unit precision.

That said, convert=True can also serve the case where Python provides engineering-unit values but operators still need to trim them with EOFF. The egu_to_raw() helper method performs the inverse conversion so Python can write a desired VAL and let the operator's calibration fields apply on top:

# Python has a float temperature reading; operator wants to trim with EOFF
sensor = builder.aIn("TEMPERATURE",
    convert=True,
    LINR="SLOPE",
    ESLO=1.0,
    EOFF=0.0,     # operator can adjust this via caput at runtime
    EGU="degC",
)

# Write a desired engineering value; operator's EOFF will still add on top
sensor.set(sensor.egu_to_raw(measured_temp, ignore_operator_offset=True))

# Or write an exact value regardless of operator offset:
sensor.set(sensor.egu_to_raw(measured_temp))  # VAL == measured_temp exactly

The cost is one extra int() truncation per update cycle. For the common case where Python provides raw integer counts, pass the integer directly to set() -- no helper needed. For the "operator-adjustable offset on a Python float" use case, a separate calibration PV at the application layer remains the zero-overhead alternative.

I'm including this here because it completes the set of standard ai/ao behaviors that are otherwise unreachable, and because in practice the feedback I hear is that ai/ao records "don't behave normally" -- partly because SMOO, OROC, and calibration fields silently do nothing. Even if most users never need convert=True, having it available lowers the barrier to adopting pSIOC in hardware-integration roles.

---

Known Limitations

Periodic SCAN causes extra record processing

When SCAN is set to a periodic rate (e.g. "1 second"), the EPICS scan thread calls dbProcess() on its own schedule. Python's set() also calls scanOnce() -> dbProcess() each time a new value is published. This means the record can be processed by both paths:

• The EPICS periodic scan fires every N seconds, calling dbProcess().
• Python's set() calls scanOnce(), which also queues dbProcess().

dbScanLock serializes access so there is no race condition, and recGblCheckDeadband() prevents duplicate CA monitors when the value hasn't changed between scans. So clients won't see phantom updates -- but the record does undergo extra processing cycles (the periodic scan re-processes the same value Python already published).

This is an acceptable trade-off: the goal is to make set() work regardless of SCAN, and the extra periodic processing is the same thing that happens in a traditional C IOC when hardware is also pushing values.

Important: If a client sets SCAN to a periodic rate and you do not reset it, the record starts processing on the EPICS periodic scan schedule in addition to Python's set() + scanOnce() processing. This double processing is a known limitation. dbScanLock prevents race conditions and recGblCheckDeadband suppresses duplicate monitors, but the wasted processing cycles are real. Users who allow clients to change SCAN should either use auto_reset_scan (see below) or manually reset SCAN from the Python callback.

Mitigation -- auto_reset_scan (implemented)

iocInit() now accepts auto_reset_scan=True. When enabled, the C-level hook resets SCAN back to "I/O Intr" immediately after forwarding the written value to the Python callback -- unless the client wrote "Passive" (a deliberate "stop updating" command that is exempt from the reset).

In this model, SCAN acts as a latched command: the client writes "5 second" -> Python sees "5 second" in the callback and adjusts its source polling rate -> SCAN is restored to "I/O Intr" -> the record stays on the I/O Intr scan list at all times. scanIoRequest remains the only processing path -- no periodic-scan contention, no extra processing cycles.

The reset is done inside dbScanLock / dbScanUnlock with a dbPut to the SCAN field using DBR_ENUM. EPICS Base's SPC_SCAN special processing (scanDelete / scanAdd) fires, correctly moving the record back to the I/O Intr scan list. Internal dbPut does not trigger asTrapWrite, so there is no callback loop.

softioc.iocInit(dispatcher, auto_reset_scan=True)

def on_scan(rec_name, field, new_scan):
    # new_scan is the value the client wrote, e.g. "5 second".
    # SCAN has already been reset to I/O Intr by the C hook.
    adjust_polling_rate(new_scan)

record.on_field_change("SCAN", on_scan)

With auto_reset_scan=False (the default), SCAN stays at whatever the client wrote. This preserves backward compatibility but means users are responsible for managing the periodic-scan contention themselves if they care about it.

---

Summary

Proposal	What it fixes	Lines of code	EPICS Base changes
1. Field-change callbacks	Python blind to external field writes	~200 added	None -- uses asTrapWriteRegisterListener
2. Universal set()	set() broken for non-I/O-Intr SCAN	~20 added	None -- uses scanOnce()
3. Analog conversion (optional)	LINR/ESLO/EOFF/SMOO/OROC dead on ai/ao; egu_to_raw() helper	~130 added	None -- uses existing RVAL/NO_CONVERT mechanism

All changes are additive, backward compatible, and use only public EPICS Base APIs. No existing application requires modification.

Test results:

• PR 1: 5 dedicated tests (field callbacks) -- 311 total passing
• PR 2: 25 dedicated tests (universal set, callback refinements) -- 363 total passing
• PR 3: 24 dedicated tests (analog conversion, get/get(raw=), egu_to_raw) -- 387 total passing, 16 skipped (Windows-only)

I'd love to hear thoughts, concerns, or architectural suggestions before I submit the PRs.

Thanks for your time!
Emilio H.
Canadian Light Source.

[Araneidae]

I fixed some Mojibake (em-dash — came through as —). Will discuss this with @AlexanderWells-diamond and @coretl when we have the chance.

[AlexanderWells-diamond]

There's still quite a few places where → appears. I'm not sure what character(s) this is supposed to be. Also × and ↔

[Araneidae]

Mojibake Decoder tells me that → is supposed to be → and × is ×. Think I'll leave @BoredDadDev to fix those!

P.S. Don't put Unicode into source code, it just causes unnecessary extra hassle later on!

[AlexanderWells-diamond]

Thank you for this very thorough proposal. Your reasoning for them is sound; we have had many people tripped up by the way SCAN works in pythonSoftIOC. The value conversion is certainly something we could make use of EPICS base's facilities for. I've only given this a once-over but have a few questions/edge case considerations:

• Do your SCAN changes work if the user tries to set SCAN on initialization to a different value?
• From a glance at scanIoRequest() it looks like it scans for any record being on the I/O Intr list, not the specific record we are currently interested in? So if there is any 1 record still in I/O Intr mode, then the check ends up not doing what is expected?

[BoredDadDev]

Hi everyone, thanks for your quick replies. I apologize for the unreadable characters -- I think I've got rid of them all now in the original post. I'll submit the first PR soon, in case having the actual code helps the discussion. And by the way, the non-ASCII characters were only present in the Markdown documents, not the actual code files, so no risk of breaking anything else. Cheers. Emilio.
Update: the firt PR is #205
I also submitted the next in the series: #206