Skip to content

Development Guide

Jump to bottom
AshipaEk0 edited this page Jun 23, 2026 · 4 revisions

Development Guide – Epilykos

This guide explains how to extend Epilykos with new visual blocks, data sources, and metrics. Intended for developers comfortable with JavaScript, Node.js, SQLite, and basic CSS/HTML.

Prerequisites

  • Node.js v22 via /usr/bin/node — serialport native bindings are compiled against v22. Running with a different Node version (e.g., v25) will produce NODE_MODULE_VERSION mismatch errors. Always explicitly use /usr/bin/node.
  • Epilykos installed (Docker or local)
  • JavaScript ES6+ (ES modules)
  • Async/await patterns
  • Basic SQL

Architecture Overview

Backend (modules/)

Module Responsibility
database.js SQLite connection, getConfig(), setConfig(), WAL mode, essentialKeys
sessionAuth.js Session-based auth, CSRF, rate limiting
ha.js Home Assistant polling
mqtt.js MQTT broker connections
modbus.js Modbus TCP/Serial with profile-based register mapping
rs232.js RS232 serial (Voltronic QPIGS, SolaX AA55, VE.Direct streaming)
dongle.js WiFi dongle TCP (Solarman/Modbus TCP, Felicity TCP)
external.js REST API polling
bms.js Bluetooth BMS bridge
pvoutput.js PVOutput.org data export (outbound)
history.js Legacy history snapshots, role-based metric mapper
grid.js Grid status, timeline
solar.js Solar forecast (Solcast/Open-Meteo), daily energy computation
savings.js PV savings calculation
metrics.js Current metrics, history
metricsManager.js User-created metrics CRUD
dashboard-config.js Layout config (JSON)
backup.js Database backup/restore with WAL checkpointing, daily snapshots
logger.js Winston logging
utils.js Grid state parser

Frontend (public/js/)

All ES modules via <script type="module">.

Module Purpose
main.js Entry — theme, config, WebSocket, polling
dashboard.js CSS Grid + SortableJS, per-block config
editor.js GridStack visual layout editor (/editor)
updater.js Dispatches updateWithState(state) to blocks
components/index.js componentBuilders registry (18 blocks)
components/*.js Block builders + updaters
api.js Centralised fetch with CSRF
charts.js Chart.js with multi-instance Maps
forecast.js Multi-instance sparkline + weather
tables.js Multi-instance data tables
grid.js Grid card helpers
utils/blockId.js Block ID generation
utils/uid.js Scoped DOM ID helper

Data Flow

Polling (30s) → buildDashboardState() → WebSocket broadcast
    → Frontend updateWithState(state)
    → Each block updater reads state.metrics
Fallback: polling /api/dashboard-state every 60s

Dashboard Layout

  • 12-column CSS Grid with align-items: start (blocks keep independent heights)
  • colSpan — grid-column span (1–12). Default 12.
  • rowSpan — explicit height in px (0 = auto). Default 0.
  • bgColor — per-block background colour
  • transparent — per-block transparency toggle
  • SortableJS drag-to-reorder (auth-only, lock/unlock toggle)
  • Multi-instance — all blocks support duplicates via uid(base, blockId) scoped IDs

Editor (/editor)

  • GridStack visual layout designer
  • Drag blocks from palette, reposition, resize
  • Click ✕ to remove, + New / Delete / Rename dashboards
  • Save & Exit persists layout
  • Block positions saved as colSpan/rowSpan in dashboard config

Adding a New Block Type

Step 1: Create the Builder

Create public/js/components/myBlock.js:

import { uid } from '../utils/uid.js';

export function buildMyBlock(block = {}) {
  const id = block.id || '';
  const config = block.config || {};
  const metrics = config.metrics || {};
  const myMetric = metrics.value || 'default_metric';

  const container = document.createElement('div');
  container.className = 'my-block card';
  container.dataset.metricMap = JSON.stringify({ value: myMetric });
  container.dataset.blockId = id;

  container.innerHTML = `
    <h3>${escapeHtml(config.title || 'My Block')}</h3>
    <div class="my-value" data-metric="${escapeHtml(myMetric)}">--</div>
  `;
  return container;
}
function escapeHtml(s) { const d = document.createElement('div'); d.textContent = s; return d.innerHTML; }

Pattern: Use uid(base, id) for all DOM IDs. Store metric config on dataset.metricMap. Use data-metric attributes. Provide defaults for backward compat.

Step 2: Create the Updater

export function updateMyBlock(state) {
  document.querySelectorAll('.my-block').forEach(container => {
    const id = container.dataset.blockId || '';
    let mm; try { mm = JSON.parse(container.dataset.metricMap); } catch(e) { return; }
    const m = state.metrics || {};
    const val = m[mm.value]?.value;
    if (val === undefined || val === null) return;
    const el = container.querySelector('.my-value');
    if (el) el.textContent = val.toFixed(1);
  });
}

Pattern: Iterate ALL containers of your type. Scope queries within each. Check for undefined/null values.

Step 3: Register

Edit components/index.js:

import { buildMyBlock, updateMyBlock } from './myBlock.js';
// Add to componentBuilders:
'my-block': buildMyBlock,

Step 4: Register Updater

Edit updater.js:

import { updateMyBlock } from './components/myBlock.js';
// Add to updateWithState:
if (activeLayout.some(b => b.type === 'my-block')) updateMyBlock(state);

Step 5: Register in Settings

Edit settings.js. Add to the type dropdown:

<option value="my-block">My Block</option>

Add config panel (in the config button handler):

} else if (block.type === 'my-block') {
  const currentMetrics = block.config?.metrics || {};
  configPanel.innerHTML = `
    <label>Title</label><input type="text" class="config-title" value="${escapeHtml(block.config?.title||'')}" style="width:100%;margin-bottom:0.5rem;">
    <label>Value Metric</label><select class="config-metric" data-role="value" style="width:100%;margin-bottom:0.5rem;">${generateMetricOptionsHtml(currentMetrics.value)}</select>
    <button class="fetch-btn save-config">Save</button>`;
}

Add save handler:

} else if (block.type === 'my-block') {
  block.config.title = configPanel.querySelector('.config-title')?.value || '';
  block.config.metrics = {};
  configPanel.querySelectorAll('.config-metric').forEach(s => { const r = s.dataset.role; if (s.value) block.config.metrics[r] = s.value; });
}

Important: Always use generateMetricOptionsHtml() for metric dropdowns — never hardcode metric options. This function pulls from the live allMetrics list loaded from /api/metrics/list.

Step 6: Add CSS

Add to style.css:

.my-value { font-size: var(--fs-large); font-weight: 600; color: var(--text); }

Step 7: Add to Editor Palette

The editor (editor.js) auto-discovers all builders from componentBuilders. Add a display name:

const names = { 'my-block': 'My Block', ... };

Adding a New Data Source

Every data source module follows the same pattern. Here's the canonical integration:

1. Module File (modules/name.js)

Export these functions:

module.exports = {
  loadProfiles,           // Load protocol profiles (if applicable)
  pollName,               // Main poll function called from pollAllSources()
  testNameConnection,     // Test button handler for settings UI
  shutdownName,           // Graceful close (for persistent connections)
  restartName,            // Reconnect on settings save (if applicable)
  availableProfiles,      // [{ id, name, protocol }] (if applicable)
};

Critical: availableProfiles must be a mutated array, never reassigned. The pattern let availableProfiles = []; module.exports = { availableProfiles } captures the ARRAY REFERENCE at require-time. Mutate in-place:

// ✅ Correct — mutates the existing array
availableProfiles.length = 0;
availableProfiles.push(item);

// ❌ Wrong — reassigns, export is now stale
availableProfiles = newArray;

2. Metric Storage Pattern

Always guard against non-numeric values — the metrics and latest_metrics tables have value REAL columns that silently reject strings:

const now = Math.floor(Date.now() / 1000);
for (const [metric, value] of Object.entries(results)) {
  if (typeof value !== 'number' || isNaN(value)) continue;
  getMetricInsert().run(now, metric, value);
  getLatestUpsert().run(metric, value, now);
}

3. Prepared Statements — Lazy Module-Scoped Pattern

Never call db.prepare() inside a poll function — each call re-compiles the SQL:

let stmt = null;
function getStmt() {
  if (!stmt) stmt = db.prepare('INSERT INTO metrics (timestamp, metric, value) VALUES (?, ?, ?)');
  return stmt;
}

4. Server Integration (server.js)

Six insertion points:

Location Change
~31 (imports) const { pollName, testNameConnection, ... } = require('./modules/name');
~67 (init) Call init/loadProfiles function after existing profile loaders
~231 (poll loop) await pollName(); in pollAllSources()
~620 (API endpoints) Add 3 endpoints: profiles list, ports list, test connection
~722 (settings hook) Add restart trigger on config save
~1054 (shutdown) Add SIGTERM/SIGINT handlers calling shutdownName()

5. Essential Keys

Add the config key to the essentialKeys array in modules/database.js (~line 154). Without this, saved config won't persist in the database.

6. Shutdown Cleanup

If your module opens persistent connections (serial ports, TCP sockets), add shutdown handlers. Never just process.exit(0) — the SQLite WAL won't be checkpointed, risking data loss.

process.on('SIGTERM', () => { shutdownName(); shutdownName2(); wsServer.close(); db.close(); process.exit(0); });
process.on('SIGINT', () => { /* same */ });

Settings UI Patterns

Pre-Save Testing

Test buttons (Test Broker, Test Topic, Test Forecast) should work BEFORE the user saves. Pattern: read input fields from the DOM, pass as query params with fallback to saved config.

Server endpoint:

const value = req.query.broker || (() => {
  const devices = JSON.parse(getConfig('mqtt_devices') || '[]');
  return devices.find(d => d.enabled)?.broker;
})();

Client handler:

const value = card.querySelector('input[name$="[broker]"]').value.trim();
if (!value) return showStatus(el, 'Enter a value first', 'error');
const res = await fetch(`/api/test-endpoint?${new URLSearchParams({ broker: value })}`);

Global Metric Exclusion

When a metric is mapped in any data source (MQTT, HA, Modbus, RS232, Dongle), it is removed from all other source dropdowns to prevent accidental double-mapping. This is handled by:

  • getAllUsedMetrics() — collects all selected metric names across all device cards
  • refreshAllMetricDropdowns() — async: fetches latest metrics from API, then rebuilds all dropdowns
  • Must be called on: page load, add/remove device, add/remove mapping, dropdown change

Always use createMetricDropdown(selectedMetric, excludeMetrics) when dynamically creating metric <select> elements — never clone from DOM or use document.getElementById.

Unsaved Changes Indicator

The settings page tracks dirty state via a Set of changed field names. After save, clear both the visual element and the Set:

dirtyCount.textContent = ''; dirtyCount.classList.remove('show');
document.dispatchEvent(new CustomEvent('stg-save-complete'));
// In settings.html: document.addEventListener('stg-save-complete', () => { dirtyFields.clear(); });

Dashboard Editor: Preview Visibility

Some block types (forecast-banner, forecast-info, forecast-sparkline, weather-block) start with display: none — they show content only after fetching data. In the editor, force them visible with placeholder content so users can see what they're adding:

if (isHiddenBlock) {
  content.style.display = '';
  // Fill placeholder data or show a centred stub with the block type name
}

Adding Profile-Based Sources (Modbus, RS232, Dongle)

Sources that use protocol profiles (JSON files in profiles/) follow an extended pattern with user-mapped metric dropdowns.

Profile JSON Structure

Profiles define raw data addresses, types, scales, and human-readable labels — but NOT metric names (those are user-mapped in settings):

[
  {
    "name": "Infinisolar V",
    "protocol": "voltronic-qpigs",
    "baudRate": 2400,
    "dataBits": 8,
    "parity": "none",
    "stopBits": 1,
    "commands": [{
      "name": "QPIGS",
      "fields": [
        { "index": 0, "label": "Grid Voltage", "unit": "V", "scale": 1 },
        { "index": 1, "label": "Grid Frequency", "unit": "Hz", "scale": 0.01 }
      ]
    }]
  }
]

Uniform Metric Mapping (5-Step Conversion)

When converting a source from hardcoded metric names to user-mapped dropdowns:

Step What Where
1 Add label fields to profiles profiles/<source>/*.json
2 Support device.mappings in poll module modules/<source>.js
3 Add /api/<source>/profile/:id endpoint server.js
4 Add mapping section to device card public/settings.js
5 Collect mappings on save public/settings.js

Poll Module with User Mappings

const userMappings = device.mappings || null;
let itemsToPoll = profile.registers; // or .fields, .metrics
if (userMappings) {
  itemsToPoll = profile.registers.filter(r => userMappings[String(r.address)] !== undefined);
}
// In the loop:
const metricName = userMappings ? userMappings[String(reg.address)] : reg.metric;
if (metricName) results[metricName] = value;

Key: The identifier used in device.mappings lookup must match what row.dataset.address stores in the UI exactly (e.g., "0x0065" for hex-formatted dongle registers).

Database Backup & Restore

Architecture

Function Trigger What it does
backupDatabase(res) GET /api/backup Creates temp snapshot → streams to user → cleans up
restoreDatabase(filePath) POST /api/restore Validates schema → backs up current → replaces → verify → rollback on fail
startSnapshotScheduler() Server startup Checks hourly; creates snapshot if last >24h old
listSnapshots() GET /api/snapshots UI list for one-click restore

WAL Checkpointing is Mandatory

Before any file copy of the database, always call:

PRAGMA wal_checkpoint(TRUNCATE);

Without this, the .db file may have an incomplete transaction log, making the backup corrupt. A file command reporting "SQLite 3.x database" is NOT sufficient — always verify with PRAGMA integrity_check;.

Restore Safety Chain

  1. Validate uploaded DB has required tables (config, metrics, latest_metrics)
  2. Checkpoint + copy current DB to .bak
  3. Shutdown: close DB, disconnect all MQTT clients
  4. Copy uploaded file over DB path
  5. Re-initialize DB + MQTT
  6. Verify restored DB is usable (SELECT 1 FROM metrics LIMIT 1)
  7. On success → delete .bak; on failure → restore .bak → re-init → rethrow

Never test destructive operations (restore, DB wipe) on production. Always copy the production database to the local dev environment first.

Performance Best Practices

Prepared Statements

All modules should use lazy-module-scoped prepared statements:

let stmt = null;
function getStmt() {
  if (!stmt) stmt = db.prepare('INSERT ... VALUES (?, ?, ?)');
  return stmt;
}

Never call db.prepare() inside a poll function — it re-compiles SQL every cycle.

Compression Middleware

Add compression() middleware to Express. Place it AFTER morgan() but BEFORE routes:

const compression = require('compression');
app.use(compression());

This reduces 10KB JSON payloads to ~1.5KB over the wire.

Static Asset Cache Headers

app.use(express.static('public', { maxAge: '5m' }));

Without this, browsers re-download CSS/JS on every page navigation.

CDN Scripts — Use defer

Scripts loaded from CDN (Chart.js, GridStack) should use defer so they don't block HTML parsing:

<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js" defer></script>

Always version-pin CDN URLs (chart.js@4.4.0 not chart.js@latest).

Query Consolidation

When a function runs 3+ queries against the same table, consolidate into one:

-- Instead of 3 separate queries:
SELECT MAX(CASE WHEN state = 1 THEN timestamp END) AS last_on,
       MAX(CASE WHEN state = 0 THEN timestamp END) AS last_off,
       state AS last_state, timestamp AS last_change_ts
FROM grid_status ORDER BY timestamp DESC LIMIT 1

Deployment

Docker Workflow

The production site runs in Docker (irunmole/epilykos:latest, built automatically from main), container name epilykos, port 8095 → :3000.

Note: irunmole/epilykos:dev images are also built automatically (from the dev branch) by CI. These are intended for development/staging — production always runs :latest.

Static files (public/*.{html,css,js}) — no restart needed, just docker cp and refresh browser:

docker cp ~/epilykos-dev/public/<file> epilykos:/app/public/<file>

Server-side files (modules/*.js, profiles/*.json, server.js) — MUST restart:

docker cp ~/epilykos-dev/<file> epilykos:/app/<file>
docker restart epilykos
sleep 3 && curl -s -o /dev/null -w "%{http_code}" https://<domain>  # expect 200

Docker Restart Zombie Process

docker restart sends SIGTERM to PID 1. If PID 1 is npm start (which doesn't forward signals to child processes), the Node.js child can survive the restart and keep serving stale code from the old require.cache. If in doubt, use:

docker rm -f epilykos
cd ~/docker/epilykos && docker compose up -d

Stale Code Detection

The most common bug in local Node.js dev: the running process has old code. Node caches modules via require.cache once loaded — editing a file on disk has zero effect on a running process.

Detect it:

SERVER_PID=$(ss -tlnp | grep ":3002" | grep -oP 'pid=\K\d+')
stat /proc/$SERVER_PID | head -3    # process start time
cd ~/epilykos-dev && git log -1 --format="%ai"  # last commit time
# If process start < commit time → stale

Verify freshness after restart:

ss -tlnp | grep ":PORT"   # PID changed?
curl -s "http://localhost:PORT/api/endpoint"   # new endpoint works?

Production-Only Patches

Some fixes need to run on production but should NOT be committed to git (e.g., DB config overrides like all_time_pv_savings_override):

  1. Edit the file in ~/epilykos-dev/
  2. docker cp to container + docker restart
  3. Verify the API returns the expected result
  4. git checkout -- <file> to restore the original in the dev clone

Common Pitfalls

1. serialport v8 Import Trap (Critical)

In serialport v8.x, the SerialPort constructor is the default export, not a named export:

// ❌ Wrong — SerialPort is undefined at runtime
const { SerialPort } = require('serialport');

// ✅ Correct
const SerialPort = require('serialport');
const ReadlineParser = require('@serialport/parser-readline');

Everything silently fails with the destructured import — no error, just no data.

2. Metric Value NaN Guarding

The metrics and latest_metrics tables have value REAL columns. Non-numeric values are silently rejected by SQLite. Always guard:

if (typeof value !== 'number' || isNaN(value)) continue;

Multi-step binary decoders can return intermediate ack objects — those string values get silently rejected without this guard.

3. availableProfiles Reference Trap

let availableProfiles = [];
module.exports = { availableProfiles };
// Later:
availableProfiles = loadProfiles();  // ❌ Reassigns — export is now stale

Always mutate in-place: availableProfiles.length = 0; availableProfiles.push(...).

4. CSS Attribute Selector Spacing

/* ❌ Space means "descendant of" */
[name^="ha"] [name$="[url]"]
/* This matches: <div name="ha_container"><input name="something[url]"> — NOT the input itself */

/* ✅ No space, explicit tag — matches the input directly */
input[name$="[url]"]

5. essentialKeys — Config Won't Persist Without It

Every new config key used by a data source MUST be added to the essentialKeys array in database.js. Without this entry, saved settings won't be stored in the database.

6. Never Drop Existing Exports When Adding New Ones

When extending module.exports, preserve ALL existing exports. A common mistake:

module.exports = { startPolling, restartPolling, getProfileById };
// ❌ `stopPolling` was in the original but got dropped → SIGTERM crash

Always read the full module.exports line before editing. Prefer adding to the existing object.

7. db.prepare() Inside Poll Functions

Each call to db.prepare() compiles the SQL statement. Inside a poll function that runs every 30 seconds, this adds up. Use the lazy-module-scoped pattern instead (see Performance section above).

8. BMS Async Polling Race Condition

If a setInterval poll function is async, the interval can fire while a previous invocation is still awaiting. Add a mutex:

let pollingActive = false;
async function pollBms() {
  if (pollingActive) return;
  pollingActive = true;
  try { /* poll */ } catch(e) { logger.error(e); }
  pollingActive = false;
}

9. WebSocket Zombie Clients

Always add error handlers alongside close handlers:

ws.on('error', () => { wsClients.delete(ws); ws.terminate(); });
ws.on('close', () => { wsClients.delete(ws); });

// Also on send:
client.send(message, err => { if (err) { wsClients.delete(client); client.terminate(); }});

10. Settings Page MutationObserver Freeze

A MutationObserver on #settings-form with { childList: true, subtree: true } fires synchronously on every appendChild call. When building device lists, this can create N cascading reflow cycles. Fix: debounce with setTimeout(..., 50).

11. Chart.js Opaque Fills Obscuring Datasets

When one dataset has much higher values than others, gradient fills (0.8 opacity) can completely cover other lines. Set subtle gradient opacity (0.03–0.1) and avoid fill: true at the chart options level. Per-dataset fills should be opt-in.

Key Patterns

uid() Scoping

import { uid } from '../utils/uid.js';
const id = block.id || '';
// In HTML: id="${uid('my-element', id)}"
// In updater: document.getElementById(uid('my-element', id))

Metric Configurability

// Builder: store on container
container.dataset.metricMap = JSON.stringify(metrics);

// Updater: read from container
let mm; try { mm = JSON.parse(container.dataset.metricMap); } catch(e) { return; }
const val = state.metrics?.[mm.some_role]?.value;

Multi-Instance Updaters

export function updateMyBlock(state) {
  document.querySelectorAll('.my-block').forEach(container => {
    // scope all queries within container
    const id = container.dataset.blockId || '';
    // ...
  });
}

Per-Block Config

Each block in the dashboard editor has:

  • Type dropdown, Width dropdown, Height dropdown
  • Background colour picker (block.bgColor)
  • Transparency checkbox (block.transparent)
  • Config button for per-block settings (metrics, title, etc.)

Git Conventions

  • Branch: All pushes go to the dev branch. main is updated via PR merges from dev.
  • Commits: Use conventional commits: feat:, fix:, refactor:, docs:, chore:
  • No .md files in the repo except readme.md. Research docs and planning notes live locally only (not committed).
  • Remove accidentally-tracked files: git rm --cached <file> && git commit

Server Port Configuration

The server uses process.env.PORT || 3000 — not a --port CLI flag. Always set via environment variable:

PORT=3002 /usr/bin/node server.js

Dev Instances

The project has a convention of running dev instances on specific ports directly (not Docker) so code changes are picked up immediately without docker cp:

PORT=3002 /usr/bin/node server.js &   # main dev

Coding Conventions

  • Use const for immutable values
  • escapeHtml() for user input in innerHTML
  • data-* attributes for metadata
  • JSON.stringify/parse for storing objects in data attributes
  • Check for undefined/null before displaying values
  • Polling functions never throw — log and continue
  • Chart.js instances stored in Maps keyed by canvas ID
  • Always use generateMetricOptionsHtml() for metric dropdowns — never hardcode

Complete Examples

See gaugeCard.js (full-circle SVG gauge), multiValueCard.js (unlimited metrics), textCard.js (static HTML content), iframeCard.js (URL embed), or flowCardSquare.js (2×2 grid layout) for complete examples of different block patterns.

Clone this wiki locally