Python bindings

[leonardocarreras]

Just as context for #884 , this is my understanding and how I think it could be shaped (helped by discussions and drawing with LLM). Text and drawing for the moment follows verbatim. Disclaimer: I might be a bit biased on my opinion because of the way we do it in DPsim and somehow I like.

Current PR code architecture

flowchart TB
    U["User Python code"]

    subgraph L4["Layer 4: Python safety layer (binding.py, 518 lines)"]
        W["villas.node.binding.Node<br/>_SampleSlice · _ensure_capacity<br/>_resolve_range · bounds checks"]
        ST["binding.pyi · python_binding.pyi<br/>hand-written, pack_from arg order drifted"]
    end

    subgraph L3["Layer 3: pybind11 module (capi_python_binding.cpp)"]
        B["python_binding<br/>28 free functions: node_* / sample_*<br/>no py::arg, no overload_cast, no GIL release"]
        SA["SamplesArray<br/>manual refcnt, new/delete[]<br/>get_block returns raw void*, unchecked"]
        NS["ns_to_timespec<br/>tv_sec / tv_nsec swapped"]
    end

    subgraph L2["Layer 2: C shim (include/villas/node.h)"]
        C["typedef void *vnode<br/>typedef void *vsample"]
        A["lib/node_capi.cpp<br/>(Node *)n reinterpret casts"]
    end

    subgraph L1["Layer 1: C++ core"]
        N["villas::node::Node"]
        SM["villas::node::Sample<br/>PtrUnique holder defined but unused"]
    end

    U --> W
    W -->|"PyCapsule: opaque, any capsule accepted"| B
    B --- SA
    B --- NS
    B -->|"void * and void **"| C
    C --> A
    A --> N
    A --> SM
    ST -.-> W

    LOSS["all type information discarded here"]
    REGAIN["safety re-implemented in Python here"]
    LOSS -.-> C
    REGAIN -.-> W

    classDef bad fill:#5a1f1f,stroke:#c0392b,color:#fff
    classDef note fill:none,stroke:#c0392b,stroke-dasharray:3 3,color:#c0392b
    class SA,NS,C bad
    class LOSS,REGAIN note

Loading

Layer 2 throws away everything the compiler knew; layer 4 exists to rebuild it in Python.

Proposed architecture

flowchart TB
    U["User Python code"]
    NP["numpy.ndarray, zero copy"]

    subgraph PKG["villas.node: PEP 420 namespace package"]
        SUG["__init__.py, thin sugar<br/>context manager · iteration · from_json<br/>~50 lines"]
        TY["_native.pyi + py.typed<br/>generated by pybind11-stubgen at build"]
        EXIST["node.py · formats.py · sample.py<br/>existing pure-Python modules"]
    end

    subgraph NAT["villas.node._native: pybind11, binds C++ directly"]
        PN["py::class_&lt;Node&gt;<br/>read · write · start · stop · toJson<br/>call_guard&lt;gil_scoped_release&gt;"]
        PS["py::class_&lt;Sample, Sample::PtrUnique&gt;<br/>py::buffer_protocol<br/>refcount via holder"]
        PE["py::enum_&lt;State&gt;<br/>py::class_&lt;Signal&gt;"]
    end

    subgraph CORE["libvillas.so.1"]
        N["villas::node::Node"]
        SM["villas::node::Sample"]
        SN["villas::node::SuperNode<br/>paths · hooks · config"]
    end

    subgraph FFI["C shim: retained, but off the Python path"]
        C["villas/node.h<br/>for ctypes · Rust · LabVIEW"]
    end

    U --> SUG
    U --> NP
    SUG --> PN
    SUG --> PS
    NP <-->|"buffer protocol"| PS
    TY -.-> SUG
    PN --> N
    PN --> SN
    PS --> SM
    PE --> N
    C --> N

    classDef good fill:#1f4a2f,stroke:#27ae60,color:#fff
    class PS,PN good

Loading

Two layers instead of four. SamplesArray, sample_pack, sample_unpack, sample_details, get_block, _ensure_capacity, _resolve_range and _SampleSlice all cease to exist: the holder does lifetime, the buffer protocol does data access.

Build and packaging

Current

flowchart LR
    subgraph NOW["Current: two owners, neither complete"]
        direction TB
        P1["pip install villas-node<br/>setuptools, pure Python"] --> D1["site-packages/villas/node/"]
        C1["cmake --install<br/>absolute sysconfig purelib<br/>ignores CMAKE_INSTALL_PREFIX"] --> D1
        D1 --> R1["binding.py present<br/>.so missing, ImportError"]
    end

Loading

Proposed

flowchart LR
    subgraph PROP["Proposed: one artifact per consumer"]
        direction TB
        SBC["scikit-build-core<br/>cmake.source-dir = binding<br/>build.targets = _native<br/>install.components = python"]
        SBC --> CMK["python/binding/CMakeLists.txt<br/>dual mode: in-tree target villas<br/>or pkg-config libvillas"]
        CMK --> REL["install DESTINATION villas/node<br/>relative, staged into wheel"]
        REL --> CIBW["cibuildwheel + auditwheel<br/>vendors libvillas.so.1"]
        CIBW --> PYPI["PyPI wheel, self contained"]
        CMK --> DISTRO["cmake --install with prefix<br/>RPM / DEB / Nix, system libvillas"]
    end

    classDef bad fill:#5a1f1f,stroke:#c0392b,color:#fff
    classDef good fill:#1f4a2f,stroke:#27ae60,color:#fff
    class R1,C1 bad
    class PYPI,DISTRO good

Loading

Prerequisite for the proposed build: pkg_check_modules(libvillas) cannot work until packaging/libvillas.pc.in is fixed, since it currently emits -lz instead of -lvillas and a relative libdir. install(EXPORT VILLASNodeConfig) is declared at lib/CMakeLists.txt:143 but never installed, so find_package is not an alternative today either.