WebMonitor

WebMonitor is a small self-hosted service that periodically checks websites and RSS feeds, records changes, and can notify you by SMS. It includes a web dashboard to manage monitors, inspect the latest RSS items, and browse HTML snapshots of pages (similar in spirit to a minimal Wayback Machine for your own monitors).

What it does

• Web page monitors fetch a URL (or an optional separate “fetch” URL), normalize the visible text, and compare a content hash to the previous run. When the hash changes, optional OpenAI validation can filter noisy diffs before an alert. On a confirmed change, it can save a snapshot of the HTML under data/sites/, downloading linked images into the same edition folder.
• RSS / Atom monitors treat new feed entries (stable ids from the feed) as changes, not a hash of the whole feed text. The last 100 items per feed are stored as JSON under data/rss_feeds/ for the article list in the UI.
• Health checks (HTTP success) are optional per monitor; you can alert when a site appears down.
• SMS is sent through a configurable HTTP gateway (same style as a small companion SMS API). Sends are rate-limited per hour to protect your gateway and bill.
• A change log (JSON) stores recent alerts with optional debug text for troubleshooting.

Project layout

Path	Purpose
app.py	Flask application entry (run locally or behind Gunicorn).
code/	Python package: engine, fetchers, storage, SMS, OpenAI, archives, etc.
config/	Local secrets and settings (not tracked if you use the provided .gitignore). Copy from *.example.json.
data/	Runtime data: monitors.json, state.json, change_log.json, sms_rate.json, rss_feeds/, sites/ (snapshots).
templates/, static/	Dashboard UI.
deploy/webmonitor.service	Systemd unit template used by the Ubuntu install script.
scripts/install-ubuntu-24.04.sh	Installs to /opt/webmonitor (by default) as a systemd service.

Paths are resolved relative to the directory that contains app.py (application root).

Requirements

• Python 3.10+ recommended (3.12 on Ubuntu 24.04).
• Dependencies are listed in requirements.txt (Flask, Requests, APScheduler, Feedparser, OpenAI client, BeautifulSoup, Gunicorn for production, etc.).

Quick start (development)

cd WebMonitor
python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

1. Copy example configs into config/ (filenames expected by the app):

   • sms_config.example.json → config/sms_config.json (optional; SMS)
   • openai_config.example.json → config/openai_config.json (optional; AI validation)

2. Optionally set OPENAI_API_KEY and/or FLASK_SECRET_KEY in the environment.

3. Start the app:

   python app.py

   By default it listens on all interfaces port 8080 (see PORT below). Open the dashboard in a browser and add monitors, or place a data/monitors.json derived from monitors.example.json.

Configuration files (config/)

• sms_config.json — Gateway base URL, credentials, recipient, TLS options. You can override many fields with environment variables (see code/sms_client.py), e.g. SMS_BASE_URL, SMS_USERNAME, SMS_PASSWORD, SMS_RECIPIENT.
• openai_config.json — API key and model; overridable with OPENAI_API_KEY, OPENAI_MODEL, OPENAI_BASE_URL.

If these files are missing, SMS and OpenAI features are simply unavailable until configured.

Environment variables (common)

Variable	Role
PORT	Used only when running python app.py (default 8080).
FLASK_SECRET_KEY	Flask session security; set in production.
MONITOR_TICK_SECONDS	Background scheduler tick interval (default 30; monitors still respect their own interval).
WEBMONITOR_DISPLAY_TZ	IANA timezone name for the dashboard clock (default Europe/Amsterdam).
OPENAI_*, SMS_*	Overrides for OpenAI and SMS (see code).

Under Gunicorn/systemd, listen address is controlled by BIND in /etc/default/webmonitor (see below), not PORT.

Production install (Ubuntu 24.04)

The repository includes a script that installs dependencies, creates an unprivileged user, syncs the app under /opt/webmonitor, installs Python packages into a venv, and registers a systemd service webmonitor using Gunicorn (wsgi:app) with one worker so the in-process scheduler runs once.

cd /path/to/WebMonitor
sudo chmod +x scripts/install-ubuntu-24.04.sh
sudo ./scripts/install-ubuntu-24.04.sh

Optional:

• INSTALL_ROOT=/srv/webmonitor — install directory.
• RUN_USER=myuser — service user name (default webmonitor).

On first install the script creates /etc/default/webmonitor if missing, with BIND=0.0.0.0:8080. Adjust BIND (e.g. 127.0.0.1:8080 behind nginx) and add FLASK_SECRET_KEY there, then:

sudo systemctl restart webmonitor
sudo journalctl -u webmonitor -f

Re-running the script upgrades application files while excluding existing data/ and config/ from the rsync so your monitors and secrets stay in place.

Dashboard features (short)

• Row actions: RSS feeds — link to stored articles; web monitors — snapshot list, force snapshot (save HTML + assets without waiting for a content change), check now, edit, delete.
• Snapshots: opening an edition uses an iframe view so the WebMonitor chrome stays visible; you can still open the raw snapshot in a new tab.
• Change log — history of detected changes with optional debug payloads.

Security notes

• No built-in login — anyone who can reach the HTTP port can use the dashboard (add/delete monitors, trigger checks, SMS tests, etc.). Run only on trusted networks, bind to localhost (127.0.0.1) and use a reverse proxy with authentication, or use a VPN/firewall.
• Treat config/ as secret; keep it off public git remotes (see .gitignore).
• Set a strong FLASK_SECRET_KEY in production; the app logs a warning if the default dev key is used.
• SMS and OpenAI credentials belong in config/ or the environment, never committed in source.
• code/sms.py is an optional standalone SMS micro-app: it reads only SMS_* environment variables (no secrets in code). Its /message/... route is unauthenticated; default bind is 127.0.0.1 via SMS_BIND (see file docstring). Do not expose it publicly without a proxy and auth.
• Query-string messages on the dashboard (errors, test output) are rendered with Jinja’s default HTML escaping to reduce XSS risk.

License

No license file is included in this repository; add one if you distribute the project.