Skip to content

Architecture

Jump to bottom
Brian Rodriguez edited this page Dec 9, 2025 · 3 revisions

Architecture Guide

Complete technical overview of how XY-MD02 WebApp works.

System Architecture

┌──────────────────────────────────────────────────────────────┐
│                      XY-MD02 WebApp                          │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌──────────────┐      ┌──────────────┐                      │
│  │  Web Browser │◄────►│   Waitress   │                      │
│  │  (User UI)   │ HTTP │  WSGI Server │                      │
│  └──────────────┘ 8050 │  (8 threads) │                      │
│        ▲               └───────┬──────┘                      │ 
│        │ WebSocket/HTTP        │                             │
│        │ (Dash callbacks)      │                             │
│        │                       ▼                             │
│  ┌─────┴────────────────────────────────┐                    │
│  │         Dash Application             │                    │
│  │  ┌──────────┐      ┌──────────────┐  │                    │
│  │  │ Layout   │◄────►│  Callbacks   │  │                    │
│  │  │ (UI)     │      │  (Logic)     │  │                    │
│  │  └──────────┘      └──────┬───────┘  │                    │
│  │                           │          │                    │
│  │                           ▼          │                    │
│  │                    ┌─────────────┐   │                    │
│  │                    │  Database   │   │                    │
│  │                    │  Module     │   │                    │
│  │                    └──────┬──────┘   │                    │
│  └───────────────────────────┼──────────┘                    │
│                              │                               │
│                              ▼                               │
│         ┌─────────────────────────────────┐                  │
│         │    SQLite Database (WAL mode)   │                  │
│         │  ┌─────────────────────────┐    │                  │
│         │  │ measurements_20251208   │    │                  │
│         │  │ measurements_20251209   │    │                  │
│         │  │ measurements_20251210   │    │                  │
│         │  │         ...             │    │                  │
│         │  └─────────────────────────┘    │                  │
│         └────────────┬────────────────────┘                  │
│                      ▲                                       │
│                      │ Write every 30s                       │
│                      │                                       │
│         ┌────────────┴────────────┐                          │
│         │  Modbus Reader Thread   │                          │
│         │  (Background Daemon)    │                          │
│         └────────────┬────────────┘                          │
│                      │ Read every 1s                         │
│                      ▼                                       │
└──────────────────────┼───────────────────────────────────────┘
                       │ Modbus RTU
                       │ (RS485 Serial)
                       ▼
              ┌─────────────────┐
              │  XY-MD02 Sensor │
              │  (Temp/Humidity)│
              └─────────────────┘

Component Breakdown

1. Modbus Reader Thread (modbus_reader.py)

Purpose: Continuously read sensor data and store in database

Operation:

  • Runs as background daemon thread (non-blocking)
  • Reads temperature and humidity registers every 1 second
  • Validates readings (temperature: -50 to 100°C, humidity: 0-100%)
  • Calculates derived values (dewpoint, absolute humidity)
  • Buffers 30 measurements in memory
  • Batch writes to database every 30 seconds
  • Performs cleanup every 24 hours (if DATA_RETENTION_DAYS > 0)

Error Handling:

  • Retries on communication errors
  • Skips invalid readings
  • Continues running on errors (logged to console)
  • Reconnects to serial port automatically

Key Functions:

read_modbus_data()  # Main thread loop
_initialize_instrument()  # Lazy initialization of Modbus connection

2. Database Layer (database.py)

Purpose: Manage SQLite database with optimized storage

Features:

  • Table-per-day partitioning: measurements_YYYYMMDD format
  • WAL mode: Better concurrent read/write performance
  • Indexed timestamps: Fast filtering on time ranges
  • Smart queries: Only scan relevant tables

Key Functions:

init_database()  # Setup with optimizations (WAL, PRAGMA settings)
get_table_name(date)  # Generate table name for specific date
ensure_table_exists(cursor, table_name)  # Create table if missing
build_union_query(...)  # Build optimized UNION ALL across tables
get_tables_for_timerange(...)  # Determine relevant tables
cleanup_old_data()  # Delete tables older than retention period

Database Schema (per table):

CREATE TABLE measurements_20251209 (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp INTEGER NOT NULL,      -- Unix timestamp
    temperature REAL NOT NULL,       -- Celsius
    humidity REAL NOT NULL,          -- Percentage
    dewpoint REAL,                   -- Celsius (calculated)
    absolute_humidity REAL           -- g/m³ (calculated)
);
CREATE INDEX idx_measurements_20251209_timestamp ON measurements_20251209(timestamp);

3. Waitress WSGI Server (app.py)

Purpose: Production-ready web server

Configuration:

serve(
    server,                    # Dash application
    host='0.0.0.0',           # Listen on all interfaces
    port=8050,                # Default port
    threads=8,                # Concurrent request handlers
    channel_timeout=60,       # Idle connection timeout
    cleanup_interval=10,      # Connection cleanup
    asyncore_use_poll=True    # Windows optimization
)

Why Waitress:

  • Production-ready (not Flask dev server)
  • Cross-platform (pure Python, no C compiler needed)
  • Multi-threaded for multiple concurrent users
  • Proper HTTP error handling
  • Logging control (we suppress to ERROR level)

Startup Sequence:

  1. Load environment variables from .env
  2. Initialize database (init_database())
  3. Run cleanup if retention policy active
  4. Start Modbus reader thread
  5. Initialize Dash app with layout
  6. Register callbacks
  7. Start Waitress server (blocks until CTRL+C)

4. Dash Application (layout.py + callbacks.py)

Purpose: Interactive web interface

Layout Components (layout.py):

  • Header: Title + language selector
  • 5 Value Cards: Temperature, humidity, dewpoint, abs. humidity, comfort
  • 5 Time-Series Graphs: Synchronized plots with unified hover
  • Time Range Selector: Dropdown (1min to all data)
  • Psychrometric Chart: Mollier diagram with current conditions
  • Historical Replay: Modal + slider for time-travel analysis
  • Stores: Hidden components for state management

Callbacks (callbacks.py):

Callback Trigger Purpose
update_language Language dropdown change Update all UI text to selected language
update_graph 1-second interval / time range change Main graphs, current values, comfort
update_psychrometric_chart 1-second interval Live psychrometric diagram
toggle_modal Button clicks Show/hide historical range selector
handle_range_selection Preset/custom range Store selected time range
load_historical_data Range selection Query and prepare historical dataset
update_historical_view Slider movement Update chart with historical point

Update Intervals:

  • Main graphs: 1000ms (1 second)
  • Graph updates check n_intervals to trigger

5. Adaptive Downsampling

Purpose: Maintain performance with large datasets

Implementation (callbacks.py - update_graph):

if time_range_minutes > 60 or original_points > 5000:
    df = df.set_index('timestamp')
    
    # Select interval based on time range
    if time_range_minutes == -1:  # All data - dynamic
        total_days = (df.index.max() - df.index.min()).days
        if total_days > 365:
            interval = '1D'   # Daily averages
        elif total_days > 90:
            interval = '6h'   # 6-hour averages
        # ... etc
    elif time_range_minutes <= 360:
        interval = '1min'     # 1-6 hours
    elif time_range_minutes <= 1440:
        interval = '5min'     # 6-24 hours
    # ... etc
    
    df = df.resample(interval).mean()
    df = df.dropna().reset_index()

Performance Impact:

Original Points After Downsampling Reduction
18,720 (6h) 360 98.1%
374,400 (1wk) 672 99.8%
1,612,800 (1mo) 720 99.96%
27,374,400 (1yr) 365 99.999%

Key Principle: Keep displayed points between 1,000-2,000 for instant rendering while preserving visual trends.

6. Comfort Score System

Purpose: Quantify thermal comfort using Humidex

Formula (Environment Canada):

1. Dewpoint (Td) = T - ((100 - RH) / 5.0)
2. Vapor pressure (e) = 6.11 × exp(5417.7530 × (1/273.16 - 1/(273.15+Td)))
3. Humidex = T + 0.5555 × (e - 10)

Scoring:

if humidex < 20:      return 0  # Too cold
elif humidex < 27:    return 4  # Comfortable (cool)
elif humidex < 30:    return 5  # Comfortable
elif humidex < 35:    return 6  # Optimal
elif humidex < 40:    return 3  # Slightly uncomfortable
elif humidex < 46:    return 2  # Uncomfortable
elif humidex < 54:    return 1  # Unhealthy
else:                 return 0  # Heatstroke risk

Implementation:

  • Calculated for each datapoint in update_graph() callback
  • Applied AFTER downsampling (not averaged)
  • Displayed in comfort card (current value) and graph (historical)

7. Psychrometric Chart

Purpose: Visual representation of air properties

Components:

  • Saturation line (100% RH)
  • RH curves (10%, 20%, ..., 90%)
  • Comfort zone (20-26°C, 30-60% RH) - shaded green
  • Current point (⭐ star marker)
  • Axes: Temperature (°C) vs. Humidity Ratio (g/kg)

Variants:

  • create_psychrometric_chart(): For current conditions
  • create_psychrometric_chart_historical(): For historical replay (different label)

Update:

  • Every 1 second with latest sensor reading
  • On-demand during historical replay slider movement

Data Flow Examples

Example 1: Real-time Update

[Every 1 second]
1. Browser triggers callback via interval component (n_intervals += 1)
2. update_graph() callback executes:
   - Queries database for selected time range
   - Applies downsampling if needed
   - Calculates comfort scores
   - Returns figure + current values
3. Browser receives updates
4. Plotly re-renders graphs (with existing zoom/pan preserved)
5. Value cards update with new numbers

Example 2: Time Range Change

[User selects "Last 6 hours" from dropdown]
1. Browser triggers callback (time_range_dropdown value changes)
2. update_graph() callback executes:
   - Calculates cutoff time (now - 6 hours)
   - Builds UNION query across relevant tables
   - Retrieves ~18,720 datapoints
   - Applies 1-minute downsampling → 360 points
   - Returns downsampled figure
3. Browser receives figure
4. Plotly renders 360 points instantly
5. Console shows: "Downsampled: 18720 → 360 punten (-98.1%)"

Example 3: Historical Replay

[User clicks "Choose range", selects "Last 1 hour", moves slider]
1. toggle_modal() opens modal
2. handle_range_selection() stores choice
3. load_historical_data() executes:
   - Queries database for 1 hour
   - Resamples to 1-minute intervals (60 points)
   - Generates slider marks
   - Stores data in browser (dcc.Store)
4. Slider appears with 60 positions
5. User drags slider to position 30 (halfway)
6. update_historical_view() executes:
   - Retrieves stored data
   - Gets temperature/humidity at position 30
   - Calls create_psychrometric_chart_historical()
   - Returns updated chart
7. Browser shows conditions at selected moment

Performance Characteristics

Memory Usage

  • Modbus thread: ~1-5 MB (30-measurement buffer)
  • Database: Varies by data volume (1 year ≈ 500 MB)
  • Web server: ~50-100 MB per connected user
  • Browser: ~50-200 MB (depends on graph complexity)

CPU Usage

  • Idle: < 1%
  • Modbus reading: < 5% (1 read/second)
  • Database write: < 10% spike every 30 seconds
  • Graph update: < 20% spike when user interacts
  • Downsampling: < 30% spike for large datasets (milliseconds)

Network

  • Initial page load: ~1-2 MB (includes all libraries)
  • Graph update: ~10-100 KB (depends on point count)
  • No constant polling: Updates only on user action or interval

Disk I/O

  • Modbus writes: Every 30 seconds (batch of 30 measurements)
  • Graph queries: On-demand (when user changes time range)
  • Cleanup: Once per 24 hours (if retention enabled)

Scalability

Concurrent Users

  • Waitress handles up to 8 concurrent requests
  • Each user's browser updates independently
  • Database WAL mode allows concurrent reads during writes

Data Volume

  • Tested: Years of data (27+ million records)
  • Performance: Instant rendering with downsampling
  • Storage: ~500 MB per year of 1-second readings
  • Cleanup: Automatic with DATA_RETENTION_DAYS

Response Time

  • Page load: < 2 seconds
  • Graph update: < 500ms (with downsampling)
  • Historical query: < 1 second (1 month of data)
  • All data query: < 2 seconds (years of data)

Debugging

Enable Debug Output

Set in .env:

DEBUG_LOGGING=True

Console Output:

📊 Query voor laatste 360 minuten (vanaf 2025-12-09 14:00:00)
   → Aantal relevante tabellen: 1
   → Opgehaalde datapunten: 18720
   → Downsampled: 18720 → 360 punten (-98.1%)
   Debug comfort #1: T=21.5°C, RH=55.0%, Humidex=24.3, Score=5
   Debug comfort #2: T=21.6°C, RH=54.8%, Humidex=24.4, Score=5

Information Logged:

  • Database queries (table count, timestamps)
  • Downsampling statistics (before/after, reduction %)
  • Comfort calculations (first 5 points)
  • Modbus communication (readings, errors)
  • Table creation/cleanup events

Common Debug Tasks

Check database contents:

sqlite3 src/modbus_sensor_data.db
.tables  # List all tables
SELECT COUNT(*) FROM measurements_20251209;

Verify Modbus communication:

  • Look for continuous "Temperature: X°C, Humidity: Y%" messages
  • Check for "No response from slave" errors
  • Verify sensor LED is on (powered)

Test downsampling:

  • Select "All data" from dropdown
  • Check console for "Downsampled: X → Y punten"
  • Should see dramatic reduction (>98%)

Profile performance:

  • Enable DEBUG_LOGGING
  • Change time ranges and note query times
  • Check browser developer tools (F12) → Network tab

Future Enhancements

Potential improvements (not yet implemented):

  1. WebGL Rendering: For even larger datasets (millions of points)
  2. Server-Side Rendering: Pre-render graphs on server
  3. Progressive Loading: Load summary first, details on zoom
  4. Data Export: Built-in CSV/Excel export functionality
  5. Alerts: Email/push notifications for out-of-range conditions
  6. Multiple Sensors: Support multiple XY-MD02 units
  7. Database Vacuum: Automatic SQLite VACUUM on schedule
  8. Real-time Alerts: Browser notifications for comfort changes
  9. Mobile App: Native iOS/Android applications
  10. API Endpoint: REST API for external integrations

Security Considerations

Current Implementation:

  • No authentication (runs on localhost by default)
  • Database is local file (no remote access)
  • Waitress runs as single-user process

Production Deployment:

  • Use reverse proxy (nginx/Apache) for HTTPS
  • Implement authentication (Flask-Login, OAuth)
  • Restrict APP_HOST to specific interface
  • Enable firewall rules for APP_PORT
  • Regular backups of database file
  • Monitor disk space for database growth

Maintenance

Daily:

  • Automatic cleanup (if DATA_RETENTION_DAYS > 0)
  • Log rotation (if logs are enabled)

Weekly:

  • Check database size
  • Review console logs for errors

Monthly:

  • Update dependencies (Dependabot PRs)
  • Vacuum database: sqlite3 src/modbus_sensor_data.db "VACUUM;"
  • Check sensor calibration

Yearly:

  • Full backup of database
  • Review data retention policy
  • Update Python version if needed

Clone this wiki locally