Skip to content

Device Simulator

Jump to bottom
Willem edited this page Aug 24, 2026 · 7 revisions

Device Simulator

Simulates RAMSES devices at the communication endpoint (MQTT or serial) for testing without physical RF hardware.

⚠️ IMPORTANT: Use Only on Dedicated System or Container

The Device Simulator should only be used on a dedicated Home Assistant instance or container, separate from your production system. The simulator responds to RAMSES protocol messages and can interfere with real RF traffic if deployed alongside production hardware. See Deployment for setup instructions.

Overview

The Device Simulator feature allows you to:

  • Simulate RAMSES devices (FAN, REM, etc.) without physical hardware
  • Test ramses_cc integration behavior in a controlled environment
  • Replay captured traffic scenarios
  • Debug protocol exchanges without real RF interference
  • Test new features and edge cases safely

Architecture

The simulator operates at the communication endpoint level:

  • MQTT Mode: Responds to messages on RAMSES/GATEWAY_SIM/{gwy_id}/rx and publishes responses to /tx
  • Serial Mode: [WIP] Intercepts serial traffic (for HGI80 testing) — not yet implemented
  • Zero Crosstalk: Uses separate MQTT topic namespaces (RAMSES/GATEWAY_SIM/* vs RAMSES/GATEWAY/*)

Key Components

Scenario Engine

Orchestrates simulated device behavior:

  • Periodic Emitters: Autonomous I-frame bursts for device discovery ✅
  • Response Engine: RQ→RP auto-response based on device database ✅
  • Device Playback: ✅ Replay captured conversations with auto-activation and emitter control
  • Device Unavailability: Simulates devices going offline/coming back online ✅
  • HVAC Device Loss: Simulates HVAC controller failure ✅

Device Database

Stores simulated device profiles (database structure implemented, data curation ongoing):

  • Response Templates: Parameter values and response patterns per device type
  • Periodic Emissions: I-frame payloads for autonomous device behavior
  • Hardware Variants: Different device personalities
  • Conversations: Multi-device exchange blocks for playback

Communication Endpoint

  • MqttEndpoint: MQTT-based communication with ramses_rf/ramses_cc ✅
  • SerialEndpoint: [WIP] Direct serial communication (for HGI80 testing) — not yet implemented

Use Cases

1. Integration Testing

Test ramses_cc integration without real hardware:

ramses_cc → RQ → Simulator → RP → ramses_cc

2. Regression Testing

Replay captured traffic to verify fixes:

# Load a scenario from captured traffic
scenario = load_scenario("fan_discovery.yaml")
engine.run_scenario(scenario)

3. Protocol Debugging

Isolate protocol-level issues:

  • Test specific parameter codes
  • Simulate edge cases (invalid payloads, timeouts)
  • Verify QoS behavior

4. Feature Development

Develop and test new features safely:

  • Test new parameter handling
  • Verify device discovery logic
  • Test error conditions

Configuration

Make sure to ONLY use the sim on a dedicated container or system. It will change your settings on Ramses RF.

See Deployment for setup instructions.

Enable Feature

  1. Navigate to Settings → Devices & Services → Ramses Extras
  2. Select Device Simulator from feature list
  3. Click Configure
  4. Set the simulator gateway ID (e.g., 18:001234)

MQTT Configuration

The simulator uses HA's built-in MQTT integration with the same broker, username, and password as your production HA instance. However, it uses a different topic namespace to ensure zero crosstalk:

  • Production: RAMSES/GATEWAY/{gwy_id}/rx / /tx
  • Simulator: RAMSES/GATEWAY_SIM/{gwy_id}/rx / /tx

When the Device Simulator is enabled, it automatically adapts the MQTT topic namespace for ramses_cc to use RAMSES/GATEWAY_SIM. No manual MQTT reconfiguration is needed for the simulator itself.

ramses_cc configuration (sim-only YAML — the ha-sim container uses a YAML config for reproducible test setups; production uses the UI config flow):

# In ramses_cc config (ha-sim only — do NOT use YAML config in production)
ramses_cc:
  gateway_id: "18:001234"
  mqtt:
    host: localhost
    port: 1883
  transport: "mqtt"

Note: Ensure the MQTT integration is enabled in HA before enabling the Device Simulator.

Device Profiles

[WIP] Load device profiles from the device database (profile management is ongoing work):

device_simulator:
  profiles:
    - class: FAN
      device_id: "37:170000"
      variant_id: "001"
      excluded_codes: []
      enabled: true

Scenarios

Built-in Scenarios

  • Device Playback: Replay captured conversations ✅ (see Device Playback below)
  • Discovery Test: [WIP] Simulates device discovery flow (stub implementation)
  • Flooding Test: [WIP] Tests system under high message load (stub implementation)
  • Timeout Test: [WIP] Simulates delayed/missing responses (stub implementation)
  • Device Suite: [WIP] Activate multiple devices by type/brand simultaneously (stub implementation)
  • Rotate Log Files: [WIP] Simulate log rotation scenarios (crashes, device loss) from user reports
  • Device Unavailability: Simulates devices going offline ✅
  • HVAC Device Loss: Simulates HVAC controller failure ✅

Discovery Conversations

Discovery conversations allow you to simulate the device discovery process for heat or HVAC device types.

Running Heat Discovery

Workflow:

  1. Load the system with fresh_start_allow_unknown_devices (all options selected in ramses_cc)
  2. Run the heat_discover_all conversation

Steps:

  1. Navigate to the Device Simulator card in HA
  2. Go to the "Scenarios" tab
  3. Find the "Conversation Playback" section
  4. Select the heat_discover_all conversation
  5. Enable the following settings:
    • auto-answer: enabled (to respond to discovery RQ frames)
    • answer_unknown_devices: enabled (to respond to devices not pre-activated)
    • skip_answers: disabled (use recorded responses from the conversation)
    • auto_activate_devices: enabled (default)
    • auto_start_emitter: disabled (default, to avoid interference)
  6. Start playback

What happens:

  • The conversation sends discovery RQ frames to all heat device types (CTL, TRV, DHW, THM, etc.)
  • Devices are auto-activated as they appear in the conversation
  • The simulator responds with discovery payloads from the device database
  • Devices are discovered by ramses_cc

Expected warnings during discovery:

  • Unexpected verb/code for src (X) to Tx - Expected during discovery; these are discovery responses that the dispatcher doesn't normally expect during steady-state operation
  • expecting len 12, got: 7 for JST 3EF1 - Expected; we use 7-byte payloads from regression logs instead of padding (which causes "Bad frame" errors)

Running HVAC Discovery

Similar workflow using the hvac_discover_all conversation for HVAC device types (FAN, REM, CO2, etc.).

Device Playback

The Device Playback scenario allows you to replay captured RAMSES protocol conversations for testing and debugging.

How It Works

When you start a device playback:

  1. Conversation Loading: The simulator loads the conversation (either from the device database or from pasted log content)
  2. Device Mapping: Peer slugs (e.g., "FAN", "REM") are mapped to device IDs (e.g., "32:150000", "37:170000")
  3. Auto-Activation: Devices from the device_map are automatically activated in the simulator (controlled by auto_activate_devices parameter)
  4. Emitter Control: Periodic emitter tasks can be started or suppressed (controlled by auto_start_emitter parameter)
  5. Playback: Conversation frames are sent with original timing (or modified by speed/delay settings)
  6. Response Handling: Depending on settings, responses can come from the recorded conversation or from the auto-answer engine

Key Settings

auto-answer Toggle

When enabled, the simulator automatically responds to RQ (request) frames using the device database. This is the primary response mechanism for playback scenarios.

  • Enabled: Simulator responds to RQ frames with RP frames from the device database
  • Disabled: Simulator does not respond to RQ frames (useful for testing timeout behavior)

skip_answers Parameter

When starting a playback with skip_answers=true (or checking the "Skip recorded answers" checkbox in the UI):

  • The simulator skips recorded RP frames from the conversation
  • Instead, it relies on the auto-answer toggle to generate responses dynamically
  • This allows you to test the simulator's response engine with real RQ frames while using device database payloads for responses

Use case: You have a captured log of RQ frames, but you want to test if the simulator's auto-answer can respond correctly using its device database.

answer_unknown_devices Toggle

When enabled, the simulator will respond to RQ frames for devices that are not in its active device list.

  • The device type is inferred from the device ID prefix using the device database (e.g., 32: → FAN, 37: → FAN)
  • This is useful for playback scenarios where devices are discovered dynamically
  • If device type inference fails, the simulator will not respond to the RQ frame

Use case: You're playing back a conversation where a device appears that wasn't pre-activated. With this toggle enabled, the simulator can still respond to RQ frames for that device.

auto_activate_devices Parameter

Controls whether devices from the device_map are automatically activated before playback starts.

  • true (default): Devices are automatically activated
  • false: Devices must be manually activated via the Devices tab before playback

Device activation flow:

  1. For each device in the device_map that is not already active
  2. Device type is inferred from device ID prefix
  3. Device is activated via async_activate_device
  4. The device can now respond to RQ frames via auto-answer

auto_start_emitter Parameter

Controls whether the periodic emitter task is started for auto-activated devices.

  • false (default): Devices are activated without starting their periodic emitter (no autonomous I messages)
  • true: Devices are activated with their periodic emitter started (autonomous I messages begin)

Use case: You want to test RQ→RP responses without background autonomous emissions interfering. Set auto_start_emitter=false.

Example Workflow

Scenario: Test if the simulator can respond to a fan info request using its device database.

  1. Import the conversation log (or select from device database)
  2. Enable auto-answer toggle
  3. Enable answer_unknown_devices toggle
  4. Check "Skip recorded answers" (skip_answers=true)
  5. Set auto_activate_devices=true (default)
  6. Set auto_start_emitter=false (default)
  7. Start playback

What happens:

  • REM device sends RQ to FAN device
  • FAN device is auto-activated
  • Simulator's auto-answer responds to the RQ using device database payload
  • No autonomous emissions from REM or FAN (emitter not started)

Parameters

Parameter Type Default Description
conversation string required Conversation ID to play
log_content string null Pasted ramses.log content to import and play
name string "imported_log" Name for imported log
save_yaml boolean true Persist imported conversation to YAML
device_map dict auto-inferred Map peer slugs to device IDs
speed float null Playback speed multiplier
loops int 1 Number of repetitions
inter_message_delay float null Fixed delay between messages
skip_answers boolean false Skip recorded RP frames, use auto-answer
auto_activate_devices boolean true Auto-activate devices from device_map
auto_start_emitter boolean false Start periodic emitter for devices

Device Database

The Device Database stores simulated device profiles, response templates, and conversation scenarios. It is built from real RAMSES protocol captures and can be extended with user-submitted logs.

Database Structure

device_db/
  heat/
    CTL.yaml    # Evohome controller + variants
    TRV.yaml    # HR91/HR92 radiator valve
    OTB.yaml    # R8810A/R8820 OpenTherm bridge
  hvac/
    FAN.yaml    # CVE-RF, VMD, VMC, PIV variants
    CO2.yaml    # VMS-12C39, VMS-17C01, etc.
    REM.yaml    # VMN-15LF01, VMI-15MC01, etc.
    HUM.yaml    # VMS-23HB33, VMS-17HB01
    RFS.yaml    # CCU-12T20 spIDer gateway
  conversations/
    fan+rem.yaml      # speed change, boost, etc.
    fan+co2.yaml      # DCV reaction
    fan+rfs.yaml      # RFS discovery sequence
    imported/         # User-imported conversations

Adding Messages from Logs

You can extend the device database with real protocol captures from user reports, forum posts, or your own testing.

Method 1: UI Import (for Conversations)

  1. Navigate to the Device Simulator card in HA
  2. Go to the "Scenarios" tab
  3. Find the "Conversation Playback" section
  4. Paste your ramses.log content into the import field
  5. Enter a name for the conversation
  6. Check "Save to database" to persist it
  7. Click "Import" to parse and save

The conversation will be saved to device_db/conversations/imported/<name>.yaml and can be replayed like any built-in conversation.

Method 2: Service Call (for Conversations)

service: device_simulator_import_user_log
data:
  name: "fan_speed_change_forum_post_123"
  log_content: |
    2024-01-15 12:34:56.789 082 I 37:170000 --:------ 31DA 007FFF...
    2024-01-15 12:34:57.123 082 I 32:150000 --:------ 31DA 007FFF...
  save_yaml: true

Method 3: Offline Database Build (for Auto-Answer)

For adding payloads to the auto-answer response database, use the offline build script:

cd /home/willem/dev/ramses_extras
source ~/venvs/extras/bin/activate
python custom_components/ramses_extras/features/device_simulator/scripts/build_device_db.py

This script:

  • Reads ramses_rf source code for schema definitions
  • Mines regression test packets for example payloads
  • Generates YAML device type files with response templates
  • Computes intervals for periodic emissions

Method 4: Audit and Enrich (for Both)

The audit tool validates and enriches the database with real log data:

cd /home/willem/dev/ramses_extras
source ~/venvs/extras/bin/activate

# Extract payloads from log files and update DB
python custom_components/ramses_extras/features/device_simulator/tools/audit_device_db.py \
    --extract-from-logs ~/docker_files/hass/config/ramses_log

# Audit specific device
python custom_components/ramses_extras/features/device_simulator/tools/audit_device_db.py \
    --device FAN

# Apply LLM-assisted annotation
python custom_components/ramses_extras/features/device_simulator/tools/audit_device_db.py \
    --llm-audit custom_components/ramses_extras/features/device_simulator/tools/audit_all.yaml

# Apply audit results back to DB
python custom_components/ramses_extras/features/device_simulator/tools/audit_device_db.py \
    --apply-audit custom_components/ramses_extras/features/device_simulator/tools/audit_all.yaml

Log Import Formats

The parser accepts two formats interchangeably:

Classic ramses.log:

2024-01-01 12:00:00.123 082 I 20:123456 --:------ 31DA 0001020304

Tab-separated dumps (HA packet-log exports):

2026-04-18T18:51:46.915588	RP	2349	01:150000	18:000730	013 0807C000FFFFFFFFFFFFFFFFFF

Database Content Types

1. Autonomous Emissions

Periodic I-frame messages that devices send without being asked:

  • Code (e.g., 31DA for HVAC state)
  • Verb (always I)
  • Trigger type (periodic)
  • Interval (seconds between emissions)
  • Example payloads (hex strings)

2. Responses

1:1 reactions to incoming RQ or W frames:

  • Code (e.g., 31DA)
  • RQ verb (request type)
  • RP verb (response type)
  • Delay (ms to wait before responding)
  • Example payloads

3. Conversations

Multi-device exchange blocks:

  • Peers (device types involved)
  • Frames with relative timestamps
  • Source/destination/device code/payload per frame
  • Scheme (payload variant, e.g., itho/orcon)

Auditing the Database

The audit tool helps validate and enrich the database:

Modes:

  • --audit: Parse all DB payloads, write audit file
  • --device FAN: Audit single device
  • --extract-from-logs: Mine log files for real payloads
  • --llm-audit: Use LLM to classify and annotate
  • --apply-audit: Apply audit results back to DB YAMLs

Audit file structure:

# audit_fan.yaml
device_type: FAN
domain: hvac
entries:
  - code: "31DA"
    verb: "I"
    payload: "007FFF007FFF00EFFF..."
    audit: "Y"  # Y=keep, N=remove, Remark=needs review
    notes: "Valid payload, matches schema"

Best practices:

  • Run --extract-from-logs after capturing new device traffic
  • Review audit file manually before --apply-audit
  • Use --llm-audit for initial classification, then human review
  • Commit audit YAMLs to version control for traceability

Custom Scenarios

[WIP] Create custom scenarios in YAML format (not fully implemented):

# custom_scenario.yaml
name: "Custom Test"
description: "Test specific behavior"
steps:
  - delay: 1.0
    action: emit
    device_id: "37:170000"
    code: "31DA"
    payload: "000001002000..."
  - delay: 0.5
    action: wait
    expected_code: "2411"

Device Tab

The Device Simulator includes a Lovelace card for real-time monitoring:

Features

  • Message Log: View RQ/RP/I/W frames in chronological order ✅
  • Device Status: See active simulated devices ✅
  • Scenario Control: Start/stop scenarios ✅ (Device Playback, Device Unavailability, HVAC Device Loss implemented; others are stubs)
  • Parameter Inspection: View decoded parameter values ✅
  • Profile Management: Load and switch device profiles ✅
  • Auto-Answer Toggle: Enable/disable automatic RQ→RP responses ✅
  • Answer Unknown Devices Toggle: Enable/disable responses for devices not in active list ✅

Message Log Columns

  • Time: Timestamp with millisecond precision
  • Direction: in (received) / out (sent)
  • Verb: RQ/RP/I/W
  • Code: RAMSES parameter code (e.g., 2411, 31DA)
  • Payload: Hex payload
  • Decoded: Human-readable parameter value

Deployment

⚠️ Safety Requirements

DO NOT deploy the Device Simulator on your production Home Assistant instance. The simulator:

  • Responds to RQ frames with fake data
  • Can interfere with real RF traffic
  • Uses a separate HGI ID that may conflict
  • Should run in an isolated environment

Recommended Setup: Dedicated Container

Use the provided ha-sim Docker Compose setup:

# 1. Set config path and create directory
export HA_SIM_CONFIG=/path/to/your/ha-sim/config
mkdir -p $HA_SIM_CONFIG

# 2. Copy docker-compose.yml and configuration.yaml
cp /path/to/ramses_extras/deployments/ha-sim/docker-compose.yml $HA_SIM_CONFIG/../
cp /path/to/ramses_extras/deployments/ha-sim/configuration.yaml.example $HA_SIM_CONFIG/configuration.yaml

# 3. Start the container
cd $HA_SIM_CONFIG/..
docker compose up -d

# 4. Deploy components to simulator
cd /path/to/ramses_rf    && HA_SIM_CONFIG=$HA_SIM_CONFIG make install-sim
cd /path/to/ramses_cc    && HA_SIM_CONFIG=$HA_SIM_CONFIG make install-sim
cd /path/to/ramses_extras && HA_SIM_CONFIG=$HA_SIM_CONFIG make install-sim
docker restart ha-sim

Note: The HA_SIM_CONFIG environment variable can be set in your shell profile (e.g., ~/.bashrc) for convenience. See deployments/ha-sim/README.md for detailed instructions.

Port Configuration

Instance Purpose Port Container
Production Real hardware 8123 hass
Simulator Device testing 8124 ha-sim

MQTT Topic Namespaces

  • Production: RAMSES/GATEWAY/{gwy_id}/rx / /tx
  • Simulator: RAMSES/GATEWAY_SIM/{gwy_id}/rx / /tx

This ensures zero crosstalk between instances.

Alternative: Separate Physical System

For advanced users, deploy on a separate physical machine:

  1. Install Home Assistant on a dedicated system
  2. Install ramses_cc, ramses_rf, ramses_extras
  3. Configure Device Simulator
  4. Ensure separate MQTT broker or use topic namespaces

Troubleshooting

Simulator Not Responding

Check:

  1. MQTT broker is running and accessible
  2. ramses_cc is configured with correct gateway ID
  3. Device Simulator feature is enabled
  4. Auto-answer is enabled (default)
# In Device Simulator config
auto_answer: true  # Default: true

Messages Not Appearing in Device Tab

Check:

  1. Device Simulator card is added to dashboard
  2. Browser cache cleared (simulator UI updates require reload)
  3. WebSocket connection is active

Conflicts with Production Hardware

If you see interference:

  1. Stop immediately - verify you're on dedicated instance
  2. Check MQTT topic namespaces are separate
  3. Verify HGI IDs are different
  4. Ensure simulator is NOT deployed on production HA

Limitations

  • Not a Hardware Emulator: Simulates protocol responses, not RF physics
  • Device Database Coverage: Limited to profiles in device database
  • Serial Mode: Requires physical serial connection or loopback
  • Real-Time Constraints: Timing accuracy depends on system load
  • MQTT Mode vs Real RF Behavior: Binding handshakes (1FC9/3150) cannot be simulated in MQTT mode

MQTT Mode vs Real RF Behavior

In real RAMSES RF systems, devices communicate directly via radio:

  • Binding frames (e.g., 1FC9/3150) are sent directly between devices (CTL → TRV)
  • The HGI (Home Gateway Interface) sees these frames because it listens to all RF traffic
  • No routing through the gateway is required

In the simulator's MQTT mode:

  • Binding frames (1FC9/3150) are rejected by ramses_rf as "Invalid address set" even when routed via HGI
  • Direct device-to-device I-frames bypass the gateway and are rejected
  • This is a fundamental limitation of the MQTT-based simulation

Implications for testing:

  • Binding handshake frames should be omitted from conversation YAMLs in MQTT mode
  • Devices will still be discovered via regular discovery frames (RQ/RP)
  • Binding relationships won't be established automatically
  • If binding is required for testing, it must be configured manually or via other means
  • This limitation affects testing of features that depend on binding relationships (e.g., fan control with bound REM)

Future improvements:

  • The simulator could be enhanced to handle direct device-to-device I-frames correctly
  • ramses_rf would need to accept direct device-to-device frames via MQTT
  • This would require proper addressing logic and frame forwarding in the simulator
  • Alternatively, a serial-based simulation mode could better replicate real RF behavior

Known Issues

1. RF Restart Required to Find Bound REM

On a clean system (no cached messages, empty database), ramses_rf cannot find the REM that is defined as bound for a FAN during startup. This causes errors when using fan parameter services.

Workaround: Restart ramses_rf (only) again. It will find the REM and use it as the bound REM.

Root Cause: The REM is not known by RF on a clean system. When RF starts, it cannot find the REM that is defined as bound for a FAN.

Future Fix: This could be solved by active lookup when using get/set fan params and setting it the hard way. This may be resolved by the ongoing system overhaul.

2. RQ Messages Not Logged Correctly

Outbound RQ messages are not logged to the packet log file. The log_all_mqtt setting only adds mq Tx: / mq Rx: entries to the HA log file — it does not touch the packet log file or the add_msg_handler pipeline.

Root Cause: The packet log uses a separate dedicated logger (set_pkt_logging in logger.py) that only receives frames that pass through the protocol's _dispatch_msg → add_handler pipeline — exclusively inbound packets. Outbound RQ never enters this path for MQTT (no echo).

There is a BlockMqttFilter class defined in logger.py that would block MQTT/Sent traffic from the packet log, but it is never actually applied to any handler (dead code).

3. RQ/RP Message Order Issues

The order of RQ/RP messages in the log may appear incorrect. This can be due to RQ message cycles and retries from the protocol layer.

Development

Automated End-to-End Testing with ha_sim_test

The tools/ha_sim_test package is an automated test suite that exercises all ramses_cc + ramses_extras features end-to-end on the ha-sim container via the HA websocket + REST API at 100x simulator speed.

A full run takes ~6 minutes and executes 449 checks across 73 recipes covering schema management, device lifecycle, HVAC/FAN handling, raw packet injection, concurrency/stress testing, and log monitoring.

cd /home/willem/dev/ramses_extras
source ~/venvs/extras/bin/activate
python -m tools.ha_sim_test

See the dedicated wiki pages for details:

The complete technical reference (log monitor design, wait scaling internals, services tested, troubleshooting) is in docs/HA_SIM_TEST_TOOL.md in the repo.

Building Device Database

Build response templates from ramses_rf regression test packets:

cd /home/willem/dev/ramses_extras
source ~/venvs/extras/bin/activate
make build-device-db

Note: The database scaffolding is implemented, but data curation (validating and expanding payload coverage) is ongoing. Use the audit tools (see Device Database) to enrich the database with real log data.

Adding New Scenarios

Create scenario files in custom_components/ramses_extras/features/device_simulator/scenarios/ (infrastructure exists, scenario implementation is ongoing):

# my_scenario.py
from .base import ScenarioDefinition, ScenarioResult

async def run(context: ScenarioContext, params: dict[str, Any]) -> ScenarioResult:
    """Execute the scenario."""
    # Scenario logic here
    return ScenarioResult(
        scenario_id="my_scenario",
        success=True,
        details={"message": "Completed successfully"}
    )

SCENARIO_DEFINITION = ScenarioDefinition(
    scenario_id="my_scenario",
    label="My Scenario",
    toggleable=False,
    description="Test specific behavior",
    run=run,
)

Scenarios are automatically discovered and registered by the framework.

Related Features

  • Ramses Debugger: Use with Device Simulator for protocol analysis
  • Sensor Control: Test sensor mapping with simulated devices
  • HVAC Fan Card: Monitor simulated device state via UI

References

Back to: Home Prev: Feature Catalog

Ramses Extras Wiki

Features

Misc

Architecture

Reference

Clone this wiki locally