Sarus: Privacy-Preserving Multi-Vendor Perception Fusion via Homomorphic Encryption

Paper Link: https://arxiv.org/abs/2607.19146

This is a research implementation of Sarus. Five heterogeneous detectors produce canonical car, person, and stop sign detections. Vendors encode those detections as Gaussian moments, encrypt them with CKKS, and a public-key-only server aggregates matching sparse spatial bins. The vendor decrypts the aggregate and recovers fused bounding boxes and confidence values.

The repository keeps a plaintext path alongside the homomorphic path so every run can quantify the numerical approximation introduced by CKKS.

Pipeline

flowchart TD
    A["dataset"] --> B["CV detectors"]
    B --> C["Per-frame JSON and annotations"]
    C --> D["Vendor CKKS payloads"]
    D --> E["Public-only server aggregation"]
    E --> F["Vendor decryption and bin merge"]

Loading

Stage 1: CV prediction

cv_predictor.py reads every supported image in dataset/VTTI, loads yolov8, yolov9, detr50, detr101, and rt-detr, and creates one directory per input image under dataset/sarus-data.

Stage 2: homomorphic fusion

fuse.py processes every frame and every combination in configs/model_combinations.json. A combination directory such as detr50_detr101/ contains encrypted vendor payloads, plaintext reference payloads, the public-server aggregate, and both rendered fusion results.

For a detection with confidence p, vendor trust alpha, center (cx, cy), and variance proxy (sigma_x^2, sigma_y^2), SARUS forms the seven-moment vector

m = [w, w*cx, w*cx^2, w*sigma_x^2, w*cy, w*cy^2, w*sigma_y^2],
w = alpha*p.

Repository layout

sarus/
├── configs/
│   ├── cv.json
│   ├── fusion.json
│   └── model_combinations.json
├── dataset/
│   ├── VTTI/                 # input images
│   └── sarus-data/           # generated prediction and fusion artifacts
├── he_keys/
│   ├── ctx_server_pk.bin
│   └── ctx_vendors_skpk.bin  # contains the secret key; Git-ignored
├── src/sarus/
│   ├── cv_predictor/
│   └── homomorphic_fusion/
├── tests/
├── cv_predictor.py
├── fuse.py
├── HE_keygen.py
└── main.py

Installation

Create a Python 3.11 Conda environment, install the complete CV and HE dependency set, and install SARUS in editable mode:

conda create -n sarus python=3.11
conda activate sarus
pip install -r requirements.txt
pip install -e .

The final command registers the sarus console command from pyproject.toml while keeping the local source tree editable. Confirm it is available with:

sarus --help

Ultralytics and Hugging Face download model weights on first use. For reproducible or offline execution, pre-download the exact checkpoints listed in configs/cv.json and point each checkpoint value to the corresponding local path.

Key Generation

Generate a deployment-specific key pair directly from the repository:

python HE_keygen.py --output-dir he_keys

After completing the installation above, the equivalent command is:

sarus keygen --output-dir he_keys

Usage

Run the complete pipeline:

sarus run

If without installing the package, run the repository-level equivalent:

python main.py

Run the stages independently:

sarus predict
sarus fuse

The repository-level compatibility scripts are equivalent:

python cv_predictor.py
python fuse.py

Useful scoped runs include:

sarus predict --models detr50 detr101 --device cuda:0
sarus fuse --combination detr50_detr101
sarus fuse --frame frame_0001 --combination detr50_detr101

Run sarus <command> --help for all options. Paths supplied on the command line override the corresponding JSON values.

Output

For an input named frame_0001.png, prediction creates:

dataset/sarus-data/frame_0001/
├── frame_0001.png
├── yolov8.json       ├── yolov8.png
├── yolov9.json       ├── yolov9.png
├── detr50.json       ├── detr50.png
├── detr101.json      ├── detr101.png
└── rt-detr.json      └── rt-detr.png

The detr50_detr101 fusion combination creates:

detr50_detr101/
├── detr50.bin
├── detr50.png
├── detr50_plain.bin
├── detr101.bin
├── detr101.png
├── detr101_plain.bin
├── fused.bin
├── HE-Fused.png
├── plain_fused.bin
├── Plaintext-Fused.png
└── fusion_results.json

dataset/sarus-data/fusion_metrics.json summarizes bounding-box IoU and absolute differences in centers, standard deviations, and fused probabilities over all processed frame/combination pairs.

Validation

PYTHONPATH=src python -m unittest discover -s tests -v
python -m compileall -q src tests *.py

A full integration run requires OpenCV, PyTorch, Transformers, Ultralytics, TenSEAL, model checkpoints and a matched key pair.

Security and research limitations

• The server context is checked at runtime and execution stops if it contains a secret key.
• Detection values and trust-normalization mass are encrypted, but sparse class/bin identifiers, payload size, and message timing remain visible to the server.
• CKKS is approximate. Always report fusion_metrics.json and the cryptographic parameters alongside experimental results.
• .bin experiment payloads use pickle and must only be loaded from trusted local runs.
• The current vendor context models a shared secret among vendors. Distributed key generation and threshold decryption are outside this implementation.