OpenSynaptic

2-N-2 high-performance IoT protocol stack — standardises sensor readings into UCUM units, compresses them via Base62 encoding, wraps them in a binary packet, and dispatches over pluggable transporters (TCP / UDP / UART / LoRa / MQTT / CAN).

📚 Documentation Hub

🌍 All documentation has been moved to our comprehensive Wiki. Choose your language and start exploring:

English Documentation

📖 Full Wiki Navigation — Complete English documentation index
🏠 Wiki Home — Start here for role-based navigation
📄 v1.3.0 Release Notes — Package install equivalence, cross-platform wheel fixes & UCUM data bundling
🧪 v1.3.0 Test Report — 1275 tests, 1273 pass, 0 fail across 6 suites

中文文档

📖 完整导航 — 完整的中文文档索引
🏠 Wiki 首页 — 中文版首页
📄 v1.3.0 发布公告 — pip 安装等效性、跨平台 wheel 修复与 UCUM 数据打包
🧪 v1.3.0 测试报告 (EN) — 1275 项测试，1273 通过，0 失败

Try In 30 Seconds

pip install -e .
os-node demo --open-browser

Windows (no Activate.ps1 required):

.\run-main.cmd demo --open-browser

Default user config path: ~/.config/opensynaptic/Config.json
First run launches onboarding wizard (--yes / --no-wizard supported)

CLI Completion (Tab)

py -3 -m pip install argcomplete
powershell -ExecutionPolicy Bypass -File .\scripts\enable_argcomplete.ps1

Manual (without script):

Invoke-Expression (register-python-argcomplete os-node --shell powershell)

Restart PowerShell after activation.

Architecture

%%{init: {'flowchart': {'rankSeparation': 60, 'nodeSeparation': 50}}}%%
graph LR
    A["Sensors List"] -->|Raw Data| B["Standardizer<br/>(UCUM Normalisation)"]
    B -->|Standardised| C["Compressor<br/>(Base62 Encoding)"]
    C -->|Compressed| D["Fusion Engine<br/>(FULL/DIFF Packet)"]
    D -->|Binary Packet| E["Dispatcher<br/>(TCP/UDP/LoRa/MQTT/CAN)"]
    E -->|Transmitted| F["Remote Node"]
    
    style A fill:#e1f5ff
    style B fill:#fff3e0
    style C fill:#f3e5f5
    style D fill:#e8f5e9
    style E fill:#fce4ec
    style F fill:#e0f2f1

Pipeline Overview (Text):

sensors list
    → OpenSynapticStandardizer.standardize()   # UCUM normalisation
    → OpenSynapticEngine.compress()            # Base62 solidity compression
    → OSVisualFusionEngine.run_engine()        # binary packet (FULL / DIFF strategy)
    → OpenSynaptic.dispatch(medium="UDP")      # physical send via transporter

src/opensynaptic/
├── core/
│   ├── __init__.py             # Public core facade + active backend loader
│   ├── pycore/
│   │   ├── core.py             # Orchestrator – OpenSynaptic class
│   │   ├── standardization.py  # UCUM normalisation
│   │   ├── solidity.py         # Base62 compress / decompress
│   │   ├── unified_parser.py   # Binary packet encode/decode, template learning
│   │   ├── handshake.py        # CMD byte dispatch, device ID negotiation
│   │   └── transporter_manager.py # Auto-discovers pluggable transporters
│   ├── rscore/                 # Rust core backend + build/check helpers
│   ├── transport_layer/        # L4 protocol managers and protocols/
│   ├── physical_layer/         # PHY protocol managers and protocols/
│   ├── layered_protocol_manager.py # 3-layer protocol orchestration
│   ├── coremanager.py          # Core selection/runtime manager
│   └── loader.py               # Core/plugin loader
├── services/
│   ├── service_manager.py      # Plugin mount / load / dispatch hub
│   ├── plugin_registry.py      # Built-in plugin mapping + config default sync
│   ├── tui/                    # Terminal UI service (section-aware, interactive)
│   ├── web_user/               # Lightweight web UI + user management API
│   ├── dependency_manager/     # Dependency check/repair/install plugin
│   ├── env_guard/              # Environment guard service
│   ├── transporters/           # Application-layer transporter service
│   ├── db_engine/              # Database integration service
│   └── test_plugin/            # Built-in component & stress test suite
├── utils/
│   ├── constants.py            # LogMsg enum, MESSAGES, CLI_HELP_TABLE
│   ├── logger.py               # os_log singleton
│   ├── paths.py                # OSContext, read_json, write_json, get_registry_path
│   ├── base62/                 # Base62 codec bindings/utilities
│   ├── security/               # crc/xor/session-key helpers
│   └── c/                      # Native loader/build helpers
├── CLI/
│   └── app.py                  # Argparse CLI (os-node entrypoint)
plugins/
└── id_allocator.py             # uint32 ID pool, persisted to data/id_allocation.json
libraries/
└── Units/                      # UCUM unit definition JSON files
scripts/
├── integration_test.py
├── udp_receive_test.py
├── audit_driver_capabilities.py
├── services_smoke_check.py
└── cli_exhaustive_check.py
Config.json                     # Single source of truth for all runtime settings

Prerequisites

Python 3.11+
Optional: mysql-connector-python, psycopg[binary], aioquic (see pyproject.toml)

Install

pip install -e .

Windows PowerShell Startup Note

If you see this error when activating a virtual environment:

Activate.ps1 cannot be loaded because running scripts is disabled on this system.

Use the project wrappers and run without activation:

.\scripts\venv-python.cmd -m pip install -e .
.\scripts\venv-python.cmd -m pytest tests/unit tests/integration -q
.\scripts\venv-python.cmd -u src/main.py --help
.\run-main.cmd --help

If you need activation in the current shell only:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
& ".\.venv\Scripts\Activate.ps1"

First-Run Native Auto Repair

On first run, OpenSynaptic now auto-attempts native C binding repair if required runtime libraries are missing.

Trigger point: first run startup preflight and node initialization failure fallback.
What it does: runs the same native build pipeline as native-build, then retries node startup once.
If compiler/toolchain is missing, it returns structured guidance and records environment hints through env_guard.

Disable this behavior only when needed:

$env:OPENSYNAPTIC_AUTO_NATIVE_REPAIR = "0"

Why OpenSynaptic?

OpenSynaptic is not a replacement for MQTT or CoAP — it solves a different problem. While MQTT/CoAP handle transport, OpenSynaptic focuses on what you do with sensor data before it hits the wire.

The Real Problem

When you deploy IoT sensors from different vendors, you face three universal headaches:

Problem	Example
Unit chaos	Sensor A sends `"pressure": 101.3, "unit": "kPa"`, Sensor B sends `"p": 14.7, "u": "psi"`
Verbose encoding	`{"sensor_id": "temp_01", "value": 23.5, "unit": "celsius"}` → 62 bytes for 4 bytes of data
Transport fragmentation	Need TCP for reliability, UDP for speed, LoRa for range, CAN for automotive — five different codebases

What OpenSynaptic Does Differently

Layer	Traditional Approach	OpenSynaptic
Semantic normalization	Application code	✅ Built-in UCUM
Payload compression	Optional library (zlib/lz4)	✅ Built-in Base62 + DIFF (60–80% reduction vs. JSON)
Transport abstraction	Rewrite per medium	✅ Single API: TCP/UDP/LoRa/CAN/MQTT
Binary encoding	Custom or Protobuf	✅ Zero-copy pipeline

Performance Comparison (Apples-to-Apples)

The numbers below compare CPU processing time only — what happens inside your application before network transmission. Network latency (which dominates end-to-end delay) is excluded because it depends on your infrastructure, not your protocol choice.

Metric	MQTT + JSON (paho-mqtt)	CoAP (aiocoap)	OpenSynaptic
Processing latency (single sensor, Python)	~150–300 μs	~80–200 μs	9.7 μs
Throughput (single core)	~8K–15K ops/s	~10K–20K ops/s	1.2M ops/s†
Payload size (temperature: 23.5°C)	~60 bytes	~40 bytes	~16 bytes
Compression ratio (vs. JSON)	N/A	N/A	60–80% reduction

OpenSynaptic: batch_fused mode, 16 processes, R5 9600X. See Performance at a Glance.

⚠️ Important: These are protocol + serialization benchmarks, not end-to-end network latency. MQTT/CoAP add 1–10 ms for broker/network round trips — OpenSynaptic would add the same when deployed over real networks. The advantage is in processing efficiency, not physics-defying network speed.

When to Use What

If you need...	Use...
Standard IoT cloud connectivity (AWS IoT, Azure IoT Hub)	MQTT
REST-like request/response over UDP	CoAP
Sensor normalization + compression + multi-transport in one stack	OpenSynaptic
Maximum processing throughput on constrained hardware	OpenSynaptic
Already have a working MQTT/CoAP deployment	Keep it. Add OpenSynaptic for preprocessing.

OpenSynaptic as a Preprocessor

OpenSynaptic doesn't force you to abandon existing infrastructure. Use it as a preprocessing layer before MQTT:

# Standardize and compress sensor data, then send via MQTT
node = OpenSynaptic()
packet, _, _ = node.transmit(sensors=[["temp", "OK", 23.5, "cel"]])
mqtt_client.publish("sensors/data", packet.hex())  # 16 bytes instead of 60

⚡ Performance at a Glance

Color legend: R5 9600X = active, i7-9750H = crit, i7_9750H = active.

Note: The chart above uses user-provided run data. Because run profiles differ (total, processes, threads, batch, chain_mode), treat it as an engineering reference rather than a strict A/B benchmark.

gantt
  title CPU Processing Time by Percentile (us)
  dateFormat x
  axisFormat %Q
  section AVG
  R5_9600X 10.7us   : a1, 0, 11
  RK3588 31.5us   :crit, a2, 0, 32
  i7_9750H 21.8us     :active, a3, 0, 22

  section P95
  R5_9600X 36.8us   :b1, 0, 37
  RK3588 88.4us   :crit, b2, 0, 88
  i7_9750H 54.7us     :active, b3, 0, 55

  section P99
  R5_9600X 68.6us   : c1, 0, 69
  RK3588 186.1us  :crit, c2, 0, 186
  i7_9750H 89.7us     :active, c3, 0, 90

  section P99.9
  R5_9600X 109.8us  : d1, 0, 110
  RK3588 461.2us  :crit, d2, 0, 461
  i7_9750H 192.9us    :active, d3, 0, 193

  section P99.99
  R5_9600X 109.8us  : e1, 0, 110
  RK3588 811.6us  :crit, e2, 0, 812
  i7_9750H 338.8us    :active, e3, 0, 339

  section MAX
  R5_9600X 141.5us  : f1, 0, 142
  RK3588 2012.9us :crit, f2, 0, 2013
  i7_9750H 436.4us    :active, f3, 0, 436

Chart scale note: bars are shown in approximate microseconds (us) for readability, grouped by percentile for faster cross-CPU comparison.

Note: This simplified view shows only total per-packet CPU processing latency from user-provided batch_fused runs. If you need stage-level timing (standardize_ms / compress_ms / fuse_ms), use --pipeline-mode legacy in single-process mode.

Legacy Mode (Precise Per-Stage Timing)

For accurate per-stage breakdown, use --pipeline-mode legacy with single process only:

python -u src/main.py plugin-test --suite stress --total 1 \
  --chain-mode core --pipeline-mode legacy --processes 1 --threads-per-process 1

Legacy Mode Pipeline Timing Breakdown:

pie title Per-Stage Latency Distribution
    "Standardize (3.8%)" : 3.8
    "Compress (6.3%)" : 6.3
    "Fuse (89.7%)" : 89.7

Key Insight: Fusion (binary packet construction) dominates latency in legacy mode; batch_fused optimization reduces this bottleneck significantly.

⚠️ Legacy Mode Caveats:

Throughput artificially low (~1-5K pps) due to global lock in result collector, not representative of actual system speed
Latency data accuracy , but don't use pps metric for performance tuning
Use batch_fused (above) for realistic performance profiling

Full Benchmark Report

Use Cases

timeline
    title Deployment Scenarios
    Smart Agriculture (Offline)
        : $30 SBC = Local Cloud
        : LoRa Sensor Network
        : Zero Internet Dependency
        : Real-time Data Aggregation
    
    Industrial IoT (Unified)
        : Single Protocol Stack
        : Replace Proprietary Solutions
        : 50% Cost Reduction
        : Multi-transport Flexibility
    
    Privacy-First Smart Home
        : Local Data Sovereignty
        : SBC = Home Gateway
        : Cloud-agnostic Control
        : Mobile App Integration

Detailed Scenarios:

Smart Agriculture (Offline)

Deploy a $30 SBC as a local cloud, aggregating data from LoRa sensors. No internet required.

Industrial IoT (Unified)

Replace multiple proprietary protocols with a single OpenSynaptic stack, reducing integration cost by 50%.

Privacy-First Smart Home

Keep all sensor data on a local SBC; control via mobile app without exposing data to public cloud.

Minimal Usage

from opensynaptic.core import OpenSynaptic

node = OpenSynaptic()                        # reads Config.json automatically
node.ensure_id("192.168.1.100", 8080)        # request device ID from server
packet, aid, strategy = node.transmit(sensors=[["V1", "OK", 3.14, "Pa"]])
node.dispatch(packet, medium="UDP")

from opensynaptic.core import get_core_manager

manager = get_core_manager()
print(manager.available_cores())             # e.g. ['pycore', 'rscore']
manager.set_active_core('pycore')
OpenSynaptic = manager.get_symbol('OpenSynaptic')

CLI Quick Reference

All commands are available via os-node (installed entrypoint), ./run-main.cmd (Windows), or python -u src/main.py:

Command Categories:

Runtime: run, restart, snapshot, ensure-id, transmit, inject, decode, watch, tui
️ Config: config-show, config-get, config-set, core, transporter-toggle
Plugin: plugin-list, plugin-load, plugin-cmd, web-user, deps
Testing: plugin-test, native-check, native-build, rscore-build, rscore-check
Monitor: transport-status, db-status, help

All Commands

Category	Command	Description
Runtime	`run`	Persistent run loop with heartbeat
Runtime	`restart`	Gracefully restart the run loop (stop + auto-start new process)
Runtime	`snapshot`	Print node/service/transporter JSON snapshot
Runtime	`ensure-id`	Request device ID from server
Runtime	`transmit`	Encode one sensor reading and dispatch
Runtime	`inject`	Push data through pipeline stages and inspect output
Runtime	`decode`	Decode a binary packet (hex) or Base62 string back to JSON
Runtime	`watch`	Real-time poll a module's state (config / registry / transport / pipeline)
Runtime	`tui`	Render TUI snapshot (add `--interactive` for live mode)
Config	`config-show`	Display Config.json or a specific section
Config	`config-get`	Read a dot-notation key path from Config
Config	`config-set`	Write a typed value to a Config key path
Config	`core`	Show/switch core backend (`pycore` / `rscore`)
Config	`transporter-toggle`	Enable or disable a transporter in Config
Plugin	`plugin-list`	List mounted service plugins
Plugin	`plugin-load`	Load a mounted plugin by name
Plugin	`plugin-cmd`	Route a sub-command to a plugin's CLI handler
Plugin	`web-user`	Run web_user plugin directly from CLI
Plugin	`deps`	Run dependency_manager plugin directly from CLI
Testing	`plugin-test`	Run component or stress tests
Testing	`native-check`	Check native compiler/toolchain availability
Testing	`native-build`	Build native C bindings (optionally include RS core)
Testing	`rscore-build`	Build and install Rust RS core shared library
Testing	`rscore-check`	Check RS core DLL/runtime readiness and active core
Monitor	`transport-status`	Show all transporter layer states
Monitor	`db-status`	Show DB engine status
Monitor	`help`	Print full help

Full usage examples → CLI README

Config.json

Config.json at the project root is the single source of truth. Key fields:

Key	Type	Default	Effect
`assigned_id`	uint32	`4294967295`	Device ID; `4294967295` = unassigned
`engine_settings.precision`	int	`4`	Base62 decimal places
`engine_settings.active_standardization`	bool	`true`	Toggle UCUM normalisation stage
`engine_settings.active_compression`	bool	`true`	Toggle Base62 compression stage
`RESOURCES.transporters_status`	map	`{}`	Legacy merged compatibility map (mirrors layer-specific status maps)
`security_settings.drop_on_crc16_failure`	bool	`true`	Drop packets with bad CRC

Full schema → docs/CONFIG_SCHEMA.md

Testing

Test Suite Options:

Suite	Purpose	Command
`component`	Unit-level component tests	`plugin-test --suite component`
`stress`	High-volume performance tests	`plugin-test --suite stress --workers 8 --total 200`
`integration`	End-to-end integration tests	`plugin-test --suite integration`
`all`	Complete test coverage	`plugin-test --suite all`

Quick Commands:

# Windows shortcut (no Activate.ps1 needed)
.\run-main.cmd plugin-test --suite component

# Component tests (unit-level)
python -u src/main.py plugin-test --suite component

# Stress test (high throughput)
python -u src/main.py plugin-test --suite stress --workers 8 --total 200

# All suites combined
python -u src/main.py plugin-test --suite all

# If tests fail, repair dependencies
python -u src/main.py deps --cmd check
python -u src/main.py deps --cmd repair

# Then retry the test suite

# Additional testing scripts
python scripts/integration_test.py
python scripts/udp_receive_test.py --protocol udp --host 127.0.0.1 --port 8080 --config Config.json
python scripts/audit_driver_capabilities.py
python scripts/services_smoke_check.py

# Exhaustive test suites (full protocol, plugin, security infra, orthogonal)
python scripts/exhaustive_business_logic.py
python scripts/exhaustive_plugin_test.py
python scripts/exhaustive_security_infra_test.py
python scripts/exhaustive_orthogonal_test.py

# Unified local test runner (all suites in one command)
python test.py              # run all 6 suites
python test.py --fast       # skip slow suites (business logic + plugin)
python test.py --suite infra  # run a single suite by name
python test.py --list       # list all available suites

Comprehensive Repeatable Pipeline:

# Full-scale reproducible validation (recommended default)
python -u scripts/extreme_validation_pipeline.py --scale full

# Lightweight CI smoke pass
python -u scripts/extreme_validation_pipeline.py --scale smoke

# Maximum workload, fail on any step
python -u scripts/extreme_validation_pipeline.py --scale extreme --strict

The runner writes a single aggregated JSON report to:

data/benchmarks/extreme_validation_report_latest.json

It also updates per-suite benchmark artifacts under data/benchmarks/ (compare, stress, protocol matrix, CLI exhaustive report).

Full test report: See TEST_REPORT_v1.3.0.md — 1275 tests across 6 suites, 1273 pass, 0 fail.

Native And Rust Build

Backend Options:

Backend	Type	Toolchain Required	Best For
`pycore`	Pure Python	Optional (C compiler)	Balanced, easy setup
`rscore`	Rust + FFI	Required (Rust + MSVC/Clang)	Maximum performance

Quick Start:

Check what you have:

python -u src/main.py native-check      # Check toolchain availability

Build native bindings (for pycore):

python -u src/main.py native-build      # Build C bindings

Build Rust backend (optional, for rscore):

python -u src/main.py rscore-build      # Build Rust core
python -u src/main.py rscore-check      # Verify installation

Switch active core (persistent):

# Use Rust core
python -u src/main.py core --set rscore --persist

# Or switch back to Python core
python -u src/main.py core --set pycore --persist

Windows Shortcuts (no Activate.ps1 needed):

.\run-main.cmd native-check
.\run-main.cmd native-build

Adding a Transporter

See docs/TRANSPORTER_PLUGIN.md.

API Reference

See docs/API.md.

Core facade and loader reference -> docs/CORE_API.md

Documentation Hub

Repository docs map: docs/Navigation.md
Start here: docs/Home.md
Architecture walkthrough: docs/ARCHITECTURE.md
Config schema: docs/CONFIG_SCHEMA.md
Transporter/plugin extension: docs/TRANSPORTER_PLUGIN.md
Core internals: docs/internal/internal-PYCORE_INTERNALS.md
Rust core references: docs/RSCORE_API.md, docs/PYCORE_RUST_API.md
ID lease docs: docs/ID_LEASE_SYSTEM.md, docs/ID_LEASE_CONFIG_REFERENCE.md

Plugin Config Auto-Sync

Built-in plugin settings are stored in Config.json at:

RESOURCES.service_plugins.<plugin_name>

These entries are auto-created with defaults if missing; plugins remain manual-start and do not auto-run at process

Name		Name	Last commit message	Last commit date
Latest commit History 180 Commits
.github/workflows		.github/workflows
data		data
libraries		libraries
scripts		scripts
src		src
tests		tests
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Config.json		Config.json
LICENSE		LICENSE
README.md		README.md
README.zh.md		README.zh.md
RELEASE_NOTES_v1.3.0.md		RELEASE_NOTES_v1.3.0.md
RELEASE_NOTES_v1.3.1.md		RELEASE_NOTES_v1.3.1.md
RELEASE_NOTES_v1.4.0.md		RELEASE_NOTES_v1.4.0.md
SECURITY.md		SECURITY.md
TEST_COVERAGE.md		TEST_COVERAGE.md
TEST_REPORT_v1.3.0.md		TEST_REPORT_v1.3.0.md
_config.yml		_config.yml
analyze_transmissions.py		analyze_transmissions.py
config.json.lock		config.json.lock
index.html		index.html
pyproject.toml		pyproject.toml
qodana.yaml		qodana.yaml
run-main.cmd		run-main.cmd
run-main.sh		run-main.sh
test.py		test.py

Folders and files

Latest commit

History

Repository files navigation

OpenSynaptic

📚 Documentation Hub

English Documentation

中文文档

Try In 30 Seconds

CLI Completion (Tab)

Table of Contents

Architecture

Prerequisites

Install

Windows PowerShell Startup Note

First-Run Native Auto Repair

Why OpenSynaptic?

The Real Problem

What OpenSynaptic Does Differently

Performance Comparison (Apples-to-Apples)

When to Use What

OpenSynaptic as a Preprocessor

⚡ Performance at a Glance

Legacy Mode (Precise Per-Stage Timing)

Use Cases

Smart Agriculture (Offline)

Industrial IoT (Unified)

Privacy-First Smart Home

Minimal Usage

CLI Quick Reference

All Commands

Config.json

Testing

Native And Rust Build

Adding a Transporter

API Reference

Documentation Hub

Plugin Config Auto-Sync

About

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 15

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages