Skip to content
BranchesTags

Repository files navigation

Virtual Shelly Pro 3EM (Home Assistant–backed)

Overview

  • Emulates a Shelly Pro 3EM device. It polls power/voltage/current/PF values from Home Assistant (HA) and exposes Shelly-like RPC over HTTP, WebSocket, and UDP, with mDNS discovery and simple energy integration/persistence.
  • Useful for gateways/apps that expect a Shelly 3‑phase power meter on the LAN (e.g., Shelly app, b2500 integrations, custom dashboards).

Features

  • HTTP API (FastAPI + Uvicorn) with:
    • GET /shelly: Gen2 device info (id, app, ver, model, gen, mac, sn, auth flags).
    • GET /ui: Minimal web interface showing current values, recent client IPs, and endpoint metrics (auto-refreshes every 5s).
    • POST /rpc: JSON‑RPC 2.0 envelope for Shelly.* and EM*/EMData* methods.
    • GET /rpc?method=... and GET /rpc/{method}: returns the method result directly (no envelope), matching Shelly GET semantics.
    • GET /healthz and GET /: simple health and info.
  • WebSockets:
    • WS /rpc: JSON‑RPC over WebSocket (common for Shelly Gen2).
    • Fan‑out WS servers on TCP ports 6010–6022 echoing RPC responses (configurable).
    • On /rpc connect: sends NotifyFullStatus; then periodic NotifyStatus broadcasts (throttled by WS_NOTIFY_INTERVAL).
  • UDP RPC (Shelly‑style, compatible with tomquist/b2500‑meter):
    • Supports EM.GetStatus (3‑phase) and EM1.GetStatus (single total) with identical rounding/decimal behavior.
    • Listens on configurable UDP ports (defaults include 1010 and 2220).
  • Modbus TCP server (pymodbus):
    • Exposes key Shelly Pro 3EM measurements (voltage, current, power, PF, energy totals) as 32-bit floats.
    • Read-only input/holding registers share the same data; writing 1 to holding register 4200 resets the virtual energy counters.
    • Defaults to TCP port 502 with unit ID 1 (configurable via env vars).
  • mDNS service advertisements (_http._tcp and _shelly._tcp) for discovery.
  • Energy counters integrated from power over time and persisted hourly at /data/state.json (via Docker volume).
  • LightGBM forecasts replace phase and total power in Shelly responses. Raw readings are retained for energy integration, while the dashboard reports validation MAPE and model status.
  • Simple /metrics endpoint with Prometheus‑style counters for HTTP/WS/UDP events.

Quick Start (Docker Compose)

  1. Adjust docker-compose.yaml environment to match your HA URL, token, and entity IDs.
  2. Ensure your host can use host networking (recommended for UDP + mDNS).
  3. Bring it up:
    • docker compose up -d
  4. Verify:
    • curl http://<device-ip>/shelly
    • curl "http://<device-ip>/rpc/EM.GetStatus?id=0"
    • curl http://<device-ip>/healthz

Manual Run (no Docker)

  • Python 3.11+
  • LightGBM requires the OpenMP runtime (libgomp.so.1 on Linux; e.g. install libgomp1 on Debian/Ubuntu).
  • pip install -r requirements.txt
  • uvicorn app:app --host 0.0.0.0 --port 80

Configuration (env vars)

  • Home Assistant
    • HA_BASE_URL: e.g. http://homeassistant:8123 or http://192.168.x.x:8123
    • HA_TOKEN: HA Long‑Lived Access Token (read‑only)
    • POLL_INTERVAL: seconds between polls (default 2.0)
    • INPUT_SOURCE: source for phase values. Supported: home_assistant (default) or shelly_webapi
    • HA_SMOOTHING_ENABLE: average each sensor reading over the last 3 values (default true; set to false to disable)
    • Entity IDs (override as needed): A_POWER, B_POWER, C_POWER, A_VOLT, B_VOLT, C_VOLT, A_CURR, B_CURR, C_CURR, A_PF, B_PF, C_PF
    • Shelly upstream source (used when INPUT_SOURCE=shelly_webapi)
      • SHELLY_BASE_URL: base URL of one upstream Shelly (for example http://192.168.1.120)
      • SHELLY_BASE_URLS: comma-separated list of Shelly base URLs to aggregate (takes precedence over SHELLY_BASE_URL)
      • SHELLY_TIMEOUT: HTTP timeout in seconds for Shelly requests (default 3.0)
      • SHELLY_API_MODE: API mode for all configured Shelly devices (auto, gen2_rpc, gen1_emeter; default auto)
      • SHELLY_API_MODES: optional comma-separated API modes per configured URL (same order as SHELLY_BASE_URLS; overrides SHELLY_API_MODE per device)
  • Device identity
    • DEVICE_ID: Device identifier reported in RPC/mDNS (default shellypro3em-virtual-001)
    • APP_ID: Shelly application identifier (default shellypro3em)
    • MODEL: Hardware model string (default SHPRO-3EM)
    • FIRMWARE: Firmware version reported as ver (default 1.0.0-virt)
    • FW_ID: Firmware build identifier reported as fw_id (defaults to the same as FIRMWARE)
    • MAC, SN, MANUFACTURER
    • GENERATION: Shelly generation advertised via HTTP and mDNS TXT records (default 2)
  • HTTP/WebSocket
    • HTTP_PORT: container listens on this port and advertises it via mDNS (default 80).
    • WS_PORT_START, WS_PORT_END: WS fan‑out TCP range (default 6010–6022)
    • WS_NOTIFY_INTERVAL: throttle seconds for WS /rpc NotifyStatus (default 2.0).
    • WS_NOTIFY_EPS: coalescing threshold in watts; only broadcast when change ≥ EPS (default 0.1).
    • CORS_ENABLE: enable CORS middleware (true|false, default false).
    • CORS_ORIGINS: comma‑separated allowed origins (default *).
    • Request‑side scaling (divide power among active client IPs minus one, so multiple batteries share the load without oscillating)
      • REQUEST_SIDE_SCALING_ENABLE: true|false (default true)
      • REQUEST_SIDE_SCALING_CLIENTS: integer override for asking-device count (default 0 = auto by active IPs); the divisor is this count minus one, with a minimum of one
    • Total power offsets
      • POSITIVE_POWER_OFFSET: watts subtracted when total power is positive (default 10.0)
      • NEGATIVE_POWER_OFFSET: watts subtracted when total power is negative (default 10.0)
      • An exact zero total is left unchanged.
  • UDP RPC
    • UDP_PORTS: comma‑separated list (e.g. 1010,2220) for old/new Shelly Pro 3EM styles
    • UDP_MAX: max UDP payload size (bytes)
  • mDNS
    • MDNS_ENABLE: true|false
    • MDNS_HOSTNAME: instance name (defaults to DEVICE_ID)
    • MDNS_IP: optional explicit IP to advertise (helps with multi‑homed hosts)
  • Modbus TCP
    • MODBUS_ENABLE: true|false
    • MODBUS_PORT: TCP port for the Modbus server (default 502)
    • MODBUS_BIND: Bind address (default 0.0.0.0)
    • MODBUS_UNIT_ID: Unit identifier (default 1)
  • Payload shape
    • STRICT_MINIMAL_PAYLOAD: when true, HTTP/WS EM.GetStatus returns only {a_act_power,b_act_power,c_act_power,total_act_power} (some gateways prefer this).
  • Persistence
    • STATE_PATH: defaults to /data/state.json (mounted via volume in Compose).
    • Energy state and forecast observations are buffered in memory and flushed to disk once every 60 minutes to limit disk churn. A shutdown between flushes can therefore lose up to one hour of new data.
  • Forecasting
    • FORECAST_ENABLE: enable history collection and forecasts (default true). Until a model is available, current readings are returned.
    • FORECAST_HORIZON_STEPS: number of polling steps ahead to predict (default 1; a step is POLL_INTERVAL). Changing it invalidates the active model and triggers a complete cold retrain for the new target window.
    • FORECAST_HISTORY_PATH: SQLite observation store (default /data/power_history.sqlite3); new observations are written in hourly batches.
    • FORECAST_MODEL_DIR: promoted LightGBM models and metrics (default /data/forecast_model).
    • FORECAST_TRAIN_HOUR: local hour for daily child-process training (default 2).
    • FORECAST_MIN_SAMPLES: minimum supervised samples required to train (default 1000).
    • FORECAST_WINDOW_SIZE: number of consecutive readings in the input vector used to make each prediction (default 5). Changing it invalidates the active model and forces a cold retrain.
    • FORECAST_VALIDATION_FRACTION: newest chronological fraction used for validation (default 0.2).
    • FORECAST_HISTORY_DAYS: observation retention period (default 30).
    • FORECAST_MAPE_FLOOR_WATTS: denominator floor used by MAPE around zero (default 10).
    • FORECAST_SERVE_INTERVAL: seconds between serving consecutive power consumers (default 0.25). Requests are served in arrival order using the latest available reading instead of waiting for a new source poll.
    • Training uses all observations retained by FORECAST_HISTORY_DAYS (30 days by default). Daily training warm-starts each phase model from the active LightGBM booster. A candidate is atomically promoted only when its mean phase validation MAPE is under 10% and beats the incumbent on the same validation slice. Until a qualifying model exists for the configured horizon and input window, current readings are returned as the fallback.

APIs

  • HTTP JSON‑RPC (POST /rpc): send a JSON‑RPC 2.0 request, e.g.
    • { "id": 1, "method": "EM.GetStatus", "params": {"id": 0} }
  • HTTP GET RPC:
    • /rpc?method=EM.GetStatus&id=0
    • /rpc/EM.GetStatus?id=0
    • Response is the method result object, e.g. { "a_act_power": 123.4, ... }
  • WebSocket RPC:
    • Connect ws://<ip>/rpc and send the same JSON‑RPC envelopes as POST /rpc.
    • On connect you’ll receive a NotifyFullStatus; during operation NotifyStatus messages are broadcast.
  • Shelly‑style UDP RPC (b2500 compatible):
    • Send to UDP port 1010 or 2220 (configurable via UDP_PORTS)
    • Request (example): {"id":1,"src":"cli","method":"EM.GetStatus","params":{"id":0}}
    • Response (example): {"id":1,"src":"<DEVICE_ID>","dst":"unknown","result":{"a_act_power":X.X,"b_act_power":Y.Y,"c_act_power":Z.Z,"total_act_power":T.TTT}}
    • Also supports EM1.GetStatus -> { "result": { "act_power": T.TTT } }
    • If payload includes "jsonrpc":"2.0", responds with JSON‑RPC envelope (fallback handler).

Example Commands

  • HTTP GET:
    • curl http://<ip>/shelly
    • curl "http://<ip>/rpc/EM.GetStatus?id=0"
  • HTTP POST JSON‑RPC:
    • curl -s http://<ip>/rpc -H 'Content-Type: application/json' -d '{"id":1,"method":"EM.GetStatus","params":{"id":0}}'
  • WebSocket JSON‑RPC (using websocat):
    • websocat ws://<ip>/rpc
    • Then send: {"id":1,"method":"EM.GetStatus","params":{"id":0}}
  • UDP (netcat):
    • echo -n '{"id":1,"src":"cli","method":"EM.GetStatus","params":{"id":0}}' | nc -u -w1 <ip> 2220

Home Assistant polling

  • Each POLL_INTERVAL, HA sensors are fetched. Missing or unknown/unavailable values are treated as None (or 0.0 for power). Phase power values feed energy integration (kWh) over time.
  • Power consumers are served one at a time, in arrival order, with FORECAST_SERVE_INTERVAL between them. Each consumer receives the latest available result immediately after that short delay; requests do not wait for a fresh source poll.
  • Energy counters persist to STATE_PATH. You can reset counters via RPC: EMData.ResetCounters.

Shelly Web API polling

  • When INPUT_SOURCE=shelly_webapi, each poll auto-detects the API style per configured Shelly device. It first tries Gen2 RPC (/rpc/EM.GetStatus?id=0, fallback /rpc?method=EM.GetStatus&id=0) and falls back to legacy Gen1 /emeter/0..2 when needed.
  • For multiple devices (SHELLY_BASE_URLS), per-phase values are aggregated:
    • a/b/c_act_power and a/b/c_current: summed across Shelly devices.
    • a/b/c_voltage and a/b/c_pf: averaged across Shelly devices that returned values.

Shelly app notes

  • Use host networking in Docker for best compatibility with mDNS/UDP. Ensure the phone and host are on the same subnet/VLAN.
  • If discovery fails:
    • Try adding by IP in the app.
    • Set MDNS_IP to the actual LAN IP and restart.
    • Ensure firewall allows UDP 5353 (mDNS) and the chosen UDP RPC ports (e.g., 1010/2220).
    • Keep HTTP on port 80 (default in image) as some apps assume it.

Troubleshooting

  • Check health: curl http://<ip>/healthz (image includes a Docker HEALTHCHECK)
  • Inspect metrics: curl http://<ip>/metrics
  • Open the UI: http:///ui (auto-refresh every 5s)
  • Inspect logs: docker logs -f shelly3em-virtual
  • Verify endpoints hit by the app (look for GET /shelly, /rpc calls, WS /rpc handshakes).
  • Confirm UDP replies with netcat; try both 1010 and 2220 depending on your consumer.

Security

  • Do not commit real HA tokens. Use a read‑only long‑lived token.
  • This service is intended for trusted LANs. It provides unauthenticated endpoints by design to mimic Shelly devices.

Modbus register map (summary)

  • Instantaneous values (float32, big-endian; accessible via holding or input registers):
    • 3000/3001: Line frequency (Hz)
    • 3002-3007: Phase A/B/C voltages (V)
    • 3010-3015: Phase A/B/C currents (A)
    • 3020-3027: Phase A/B/C active power and total active power (W)
    • 3030-3035: Power factor for phases A/B/C
  • Energy counters (float32 kWh):
    • 3100-3105: Import energy per phase (A/B/C)
    • 3106-3111: Export energy per phase (A/B/C)
    • 3112/3113: Total import energy; 3114/3115: Total export energy
  • Diagnostics and metadata:
    • 3200/3201: Current UNIX timestamp; 3202/3203: uptime seconds
    • 3300+: Device ID, model, firmware, MAC encoded as ASCII (2 chars per register)
    • 3400: Device ready flag (1); 3401: phase count (3)
  • Commands:
    • Write 1 to holding register 4200 to reset accumulated energy counters.

Limitations

  • Not a full Shelly implementation; only a subset of RPC is supported.
  • Energy math is approximate and depends on poll timing from HA.
  • mDNS behavior can vary across networks/containers; host networking is recommended.

License

  • See repository license(s) of dependencies. This project is provided as‑is for personal/home use.

Shoutouts

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

3 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages