1.10.0

Release 1.10.0: the AI support agent, and Bale replaces SMS

An MCP server at POST /mcp so any agent that speaks the protocol can work the
support queue - read waiting tickets with the customer's balance, machine
state and account age attached, reply, and export one customer's own data,
only while that customer has an open ticket. Its address, key and tool list
are in the admin panel, with rotation handled by the root provisioner because
the key file is deliberately unreadable to the web process.

A signed webhook wakes an agent the moment a customer writes, and a skill and
unattended prompt tell it how to answer: what may never be promised, when to
escalate, and that nobody is reading its transcript.

Messages go through Bale rather than SMS. On the line this platform used, 12
of the last 25 messages were accepted, charged, and then filtered by the
operator before reaching the handset - including every verification code
measured on the day it was replaced. Each account links once by sharing its
contact with the bot, which is stronger evidence than a code proving somebody
could read a message.

Twelve findings from an external review of the 1.9.0 diff are fixed, among
them a contact-linking hole that could redirect another customer's
verification codes, a bot token written to the journal on every request, and
an installer that never created the credential files its own units declare.

1.9.0

Fixed

• Password sign-in had no abuse control of any kind. SMS codes were rate
  limited; passwords - the credential actually worth guessing - could be tried
  indefinitely, and the only trace was a counter on a dashboard. Failures are
  now counted per identifier in a rolling window (eight in fifteen minutes by
  default), which reduces an attacker from unlimited guesses to thirty-two an
  hour while a person who mistypes waits a quarter of an hour rather than
  being locked out.

  • Counted against the identifier as typed, matched or not. Throttling
    only real accounts would make the throttle itself an oracle: "this one
    slowed down, so it exists".
  • Refused before the password is checked, so a throttled caller learns
    nothing from how long the answer takes, and a correct password is refused
    too - otherwise the throttle is bypassed by whoever finally guesses right.
  • In the database, not in memory: an in-process counter resets on every
    deploy, which is exactly when someone watching would try.

• One SMS code could admit several sessions. verify read the code,
  checked it, then marked it consumed, with no lock in between. Concurrent
  requests carrying the same code all found it unconsumed and all passed.
  Reachable by a double-submitted form or a replayed request, not only by an
  attacker. The row is now locked before it is read.

  • Proven rather than asserted: with the lock removed, the new test shows
    six callers consuming one code.

• Phone numbers could be tested for membership. /api/auth/phone-available
  answered whether a number belonged to a customer, and requesting a signup
  code for a registered number returned a distinct 409 - so anyone could
  enumerate customers one number at a time without possessing any of them.
  The endpoint is removed and the code request now answers identically either
  way, sending the number's owner a sign-in reminder instead. Only the person
  holding the phone learns anything.

Changed

• Signup validates phone format in the browser and stops there. Whether a
  number is already registered is reported by SMS to its owner, or at submit.

Removed

• The 1.7 and 1.8 review documents. Every finding is either resolved, recorded
  in docs/DECISIONS.md, or closed by an explicit product decision; a work
  queue with nothing left in it is a file that only goes stale.
• web/js/disk.js, 74 lines of CSS for markup no page emits, seven imports
  the exporter extraction stranded, and one dead settings row.

1.8.0

Highlights

• Made balance mutations and AI usage settlement transactional and idempotent across failure and retry boundaries.
• Strengthened account security with verified phone changes, session revocation after identity or status changes, and correct rejected/suspended login handling.
• Clarified the first-run journey: OpenRouter works without compute, while workspace creation remains optional.
• Added supplier-limit visibility, actionable recovery paths, durable notification polling, and accessible field-level validation across primary customer and admin forms.
• Improved mobile navigation and converted billing, support, customer, published-port, and SSH-key tables into labelled phone layouts.
• Corrected managed-service vhost readiness, documentation drift, SMS delivery coverage, per-user balance-step notifications, and the 1.7 review backlog.

Payments remain operator-mediated, Telegram database backups remain unencrypted by owner decision, and published customer applications remain HTTP-only.

Verification: 624 backend tests, all frontend tests, and all static checks passed. The production health endpoint reports version 1.8.0.

1.7.0

Added

• Prometheus and Grafana, with 67 exported metric families and eight
  dashboards. The control plane exposes /internal/metrics behind a bearer
  token; Prometheus scrapes it every 60 seconds on loopback and Grafana serves
  the dashboards on loopback behind the administrator session. Each dashboard
  is embedded in the admin tab it is about — customers and credit under Users,
  the pool under Storage, capacity under Tariffs — rather than one page nobody
  can read. docs/METRICS.md lists every series and is generated from a live
  scrape, so it describes what the endpoint emits rather than what it was
  meant to.

  • Grafana has its own login. It previously trusted anonymous access
    behind the reverse proxy, which quietly merged two different systems' idea
    of "administrator": anything that reached it was already a Viewer. The
    credential is shown, masked, on the admin overview.
  • Labels are bounded at the call site: route templates never paths,
    status classes never codes, provisioner verbs from the fixed
    allowlist. A scanner walking random URLs cannot mint series.
  • The worker is a separate process, so it writes a metric snapshot the
    exporter folds in at render time. That merge is deliberately
    non-mutating — folding cumulative counters into the exporter's own
    registry would add one worker-lifetime per scrape.

• SMS notifications through Kavenegar, with a per-customer preference page.
  23 message templates, each verified by test to fit one segment — Persian
  and emoji are UCS-2, so a segment is 70 UTF-16 code units, not 70 characters,
  and counting the wrong unit silently doubles the bill. No line mixes Persian
  and Latin letters, which reads badly in an RTL client and shaped the wording
  as much as the layout.

  • Twelve of the 23 are switchable by the customer. Security messages and
    login codes are not: an interface that appears to silence a takeover
    warning is worse than one with no switch.
  • The provider key is worker-only via LoadCredential, like the OpenRouter
    management key. The internet-facing API queues messages into an outbox and
    cannot send.
  • A temporary allowlist gates real delivery while the feature is proven.
    Suppressed messages are still recorded as skipped, so the trial shows
    what would have been sent. Clear MMD_SMS_ALLOWLIST to remove it.

• Phone verification at signup, and sign-in by SMS code. Codes are hashed
  at rest, single-use, expiring, attempt-limited and rate-limited per phone,
  with the limit in the table rather than in memory so it survives a restart.
  Requesting a login code answers identically whether or not the number has an
  account.

• A worker watchdog (mmd-watchdog.timer) that texts every administrator
  when the loop stops ticking. Its own unit on purpose: a check inside the
  worker cannot report the worker being dead, and it sends directly because
  the process that drains the outbox is the one that stopped.

• Admin-configurable default model for every OpenRouter-backed service.
  One setting on the OpenRouter tab now governs Hermes, OpenClaw, OpenCode and
  Open WebUI. Each wants the id in a different shape — bare for Hermes and
  Open WebUI, openrouter/-prefixed for OpenClaw and OpenCode — which is a
  formatting detail handled per service rather than four settings to keep in
  step. Claude Code and Codex have their own starting model on their pricing
  tabs, written only at install and only when the customer has no config of
  their own.

• Sortable columns and in-place filtering on the admin users table.

Changed

• An account, an OpenRouter key and a workspace are now three lifecycles.
  Approval creates an account-scoped OpenRouter identity without creating
  compute; customers create or permanently delete their machine whenever they
  choose, and deleting or factory-resetting it preserves the account, balance
  and supplier key. Metering, credit blocking and re-enabling all work without
  a workspace. Full account deletion is the only action that revokes the key.

• OpenRouter keys are no longer restricted by model. Customers use the key
  outside the workspace, where a Hermes-shaped allowlist made it incomplete —
  non-agent endpoints such as TTS were refused. Existing keys keep their secret
  while the old allowlist is cleared once. Credit-derived dollar caps and
  zero-credit disabling remain.

• Phone is the sole contact identity, and sign-in accepts username or
  phone. Customer email had no delivery workflow yet appeared in signup,
  profile editing and destructive confirmations; the column is removed rather
  than left nullable.

• OpenCode and Open WebUI join Hermes and OpenClaw as managed agents, each
  spending the customer's existing OpenRouter key.

• Reserved infrastructure hostnames — ports, ssh, rdp, backup and
  others — cannot be registered as usernames, and the signup form no longer
  reveals a Hermes address while checking a name.

• Workspace disk reclaimed 34 GB on the host. The ZFS pool is a file vdev
  that had never been trimmed, so every block it had ever touched stayed
  allocated: 68 GiB apparent, 69 GiB written, for 33 GiB of data. zpool trim
  took the host from 97% full to 63%, and autotrim=on keeps it there. This is
  also why the 44.8 GiB reclaimed by thin provisioning in 1.6 never appeared —
  it was freed inside the pool and the file never shrank.

• Sample retention 7 days → 2. usage_samples was 96% of the database.
  Settlement only reads the hour it is closing; everything older existed for
  charts that Prometheus now draws. 48,836 rows pruned.

• The admin pages stopped drawing their own charts. The tariff page's host
  and per-workspace series, the storage tab's pool bar and distribution rows,
  and the per-customer credit chart were all redrawn on every visit with no
  history. They are dashboards now; the forms beside them stayed, because a
  form is something an operator changes rather than watches.

• The first-run checklist is three steps and stays finished once completed.
  It was recomputed from live state, so powering a machine down brought the
  whole thing back.

Fixed

• mmd_workspace_memory_bytes reported stale memory for stopped machines —
  the newest sample regardless of age, so a workspace switched off yesterday
  still claimed 429 MB. Memory is a gauge and now reports zero when the machine
  is off; CPU is a counter and correctly keeps its last value.

• OpenCode could not answer. It was installed with an OpenRouter key but no
  model, so it had nothing to call and replied with something that read like
  the prompt echoed back. Its config is now written at install.

• Nav icons that did not mean what their page did: a shield for the account, a
  bell for SMS, a speech bubble for support — which read as messaging, next to
  the SMS page — and a group of people for administration.

Removed

• /api/admin/metrics, /api/admin/metrics/per-user and
  /api/admin/capacity. All three had no caller left once the charts moved to
  Grafana; the per-user one loaded every sample in the window and grouped them
  in Python on each request. The same readings are exported as
  mmd_workspace_* and mmd_capacity_*.

• The پایش tab. Each dashboard now sits beside the tab it describes, so a
  separate monitoring page was a second place to look for the same thing.

MMD-DEV 1.5.0

Full Changelog: v1.4.0...v1.5.0

MMD-DEV 1.4.0

What's Changed

• Release 1.3.0: bug fixes, mostly reported by customers by @MMDBadCoder in #1

New Contributors

• @MMDBadCoder made their first contribution in #1

Full Changelog: v1.1.0...v1.4.0

MMD-DEV 1.1.0

Live on https://mmd-ai.ir with a real certificate.

Everything in this release either came from a customer ticket or was found while fixing one. Seven tickets came in during this cycle; all seven are closed.

---

Real TLS

The dashboard ran on a self-signed certificate bound to a bare IP — a permanent browser warning, which is exactly the habit phishing depends on. Now a Let's Encrypt certificate for the apex and www, HTTP→HTTPS on every path, HSTS, and renewal on a timer with a deploy hook that reloads nginx — without the hook a renewed certificate sits on disk while the expired one keeps being served, which is how a certificate that "renews automatically" still expires.

MMD_DOMAIN / MMD_ACME_EMAIL make it reproducible; with no domain it still falls back to self-signed so a fresh host comes up either way. One canonical origin: www, the bare IP and plain HTTP all 301 to https://mmd-ai.ir with the path preserved.

The trap in that

HSTS applies to a host on every port, not just 443. Measured, not assumed — with the dashboard's policy stored, Chrome rewrote http://mmd-ai.ir:28999 to HTTPS and the page failed, while http://ports.mmd-ai.ir:28999 loaded normally.

Customers publish their own services on high ports. Putting those on the apex would have force-upgraded every plain-HTTP app they run and broken it, in a way that looks like their bug. So connection addresses live on a subdomain, and the HSTS header deliberately omits includeSubDomains — a judgement call at the time, load-bearing now, with a test keeping it that way.

Billing

• Publishing a port is free. It hands out a firewall rule and a number from a range of ten thousand — not scarce enough to meter, and charging for it discouraged the thing the product exists for. Ledger rows written while it was charged keep their breakdown; history is not rewritten because a price changed.
• «زمان باقی‌مانده» reads in days. A funded account had several hundred hours left, and «۳۵۷٫۱ ساعت» is not a number anyone can act on.

Usage charts

• Cores and gigabytes, not percentages. "90%" reads identically on half a core and on three, and hides how much room is left. Scaled against the tier, so a quiet machine draws a low line rather than a dramatic one.
• A window picker — 5m / 15m / 1h / 6h / 24h, defaulting to 5 minutes and remembered per browser — replacing a fixed six-hour view. In front of a running machine the question is "what is it doing now".
• Sampling every 20 seconds instead of 60, because five minutes at 60 s is five points and not a chart. Settlement still runs at 5 minutes and reconciliation at 15 — tripling the metering rate must not silently triple how often money moves, and a test pins that arithmetic.
• usage_samples is finally pruned. It had grown without limit despite the docs claiming seven-day retention.
• The admin panel now distinguishes what is reserved from what is used, with host-wide charts for the latter.

Interface

• The header stays live. The support badge and the credit chip were drawn from a session object fetched once at page load, so reading a ticket left the counter unchanged until a hard reload — and the balance never moved at all.
• The unread badge on the ticket list is visible at last. It was in the DOM the whole time, but .badge was styled only inside the Connections tab bar, so it rendered as unstyled plain text.
• The reply box says «پیام شما» when your own message is last and «پاسخ شما» when support's is.
• Chart labels carry their colon, and the current value is labelled at all.
• The console logo goes to the homepage.
• File-manager paths no longer render as //home/dev.
• The browser terminal stopped clipping its own last line — the fit addon measured the border-box height and laid out ~12px more terminal than fit. At small font sizes a whole line was lost.

Operations

• Removed a CUPS print server found serving an unauthenticated web interface on 0.0.0.0:631 — on a host with no printers.
• Destroying a workspace no longer leaves its DNAT rules in the kernel.
• The server's public IP is derived at runtime rather than written into the repository, which also makes the verification suites correct on any other machine.

Two guards worth naming

No English prose may reach the Persian interface. A customer quoted a dashboard message that was in English, said "credits" where the product charges Toman, and used Western digits. The cause was structural: the API built finished sentences and the page printed them. It now returns codes and numbers; the interface writes the sentence. tests/test_no_english_prose.py walks the AST of every request handler and fails on prose, f-string pieces included.

A module must import what it uses. A shared chart module was added and one import line silently failed to apply, so the overview page threw a ReferenceError and rendered nothing for every customer. node --check passes on that file — the syntax is perfect — and loading the module passes too, because an undefined identifier is a runtime error, not a link error. tests/web/imports.test.mjs catches it, verified by reintroducing the bug.

The wider lesson, now written into the docs: a page is not verified until it has been rendered. Checking that a module parses, that its assets return 200, and that some other page loads cleanly proves nothing about it.

Tests

298 backend and 45 interface tests, up from 274 and 34 at 1.0.

Full changelog: v1.0.0...v1.1.0

MMD-DEV 1.0.0

Cloud development machines, billed by the hour in Toman.

Every customer gets what feels like their own Ubuntu server — root access, apt, Docker, AI coding tools preinstalled — reachable from a Persian web dashboard, and able to power fully off to near-zero cost without losing a byte. The whole product runs on one host.

This is 1.0: it is live and serving paying customers.

---

Why it is built the way it is

The host has no hardware virtualization — no /dev/kvm, no vmx/svm. That rules out every VM-based design on performance grounds: QEMU falls back to software emulation, and Firecracker, Kata, Cloud Hypervisor and incus launch --vm all require KVM.

Unprivileged Incus system containers on ZFS is what remains, and it turns out to satisfy the requirements well. Measured on this host, CPU and memory throughput inside a workspace are within noise of the host itself:

	host	workspace
CPU (sysbench, 1 thread)	4791 ev/s	5154 ev/s	107.6%
Memory (sysbench)	8211 MiB/s	8264 MiB/s	100.6%

The trade-off is stated rather than hidden: containers share the host kernel, so a kernel privilege-escalation bug crosses a boundary a hypervisor would resist. Admin approval of every signup is the compensating control, and it is load-bearing.

What a customer gets

• A real machine. Root, apt install anything, docker run anything.
• Power off to zero. No CPU, no RAM, no charge for either — while every file, package, config edit and Docker volume survives untouched.
• Ways in: browser terminal, SSH with managed public keys, a full XFCE desktop over RDP, and a file manager with editing, upload, download and recursive zip.
• Two permanent addresses. SSH and RDP ports reserved for the life of the account, so a saved config keeps working.
• AI tools already signed in. One click installs Claude Code and carries the platform's sign-in across.
• Publish a port so what they build stays reachable.
• Factory reset behind a confirmation that requires three separate things, one of which is the account password.
• Honest billing and support tickets built in.

Entirely Persian and right-to-left. Established technical terms stay Latin, because translating "Docker" helps nobody.

Billing

Disk bills every hour in every state, because a stopped workspace still holds its reservation. CPU and memory bill only while on, as a reservation component plus measured usage. Settlement is in arrears, and before each hour the balance must cover that hour at full capacity or the machine will not start. At zero credit the machine is archived, restorable for 30 days, then deleted.

Money is integer micro-Toman throughout — floats drift and the ledger stops reconciling. Charges are idempotent on (workspace_id, period_start, kind), so a worker restart cannot double-charge a real customer.

Security shape

The internet-facing service holds a restricted Incus certificate. Incus itself — not application logic — refuses it privileged containers, host-path disks and custom idmaps, so a compromise of the web app cannot reach the host. Everything genuinely privileged goes through a root daemon with no network listener, a SO_PEERCRED check and a fixed verb allowlist.

From inside a workspace, the host, the Incus API, cloud metadata, the provider LAN, RFC1918 and every other tenant are unreachable; internet and DNS work.

Known and accepted risks are written down in SECURITY.md rather than left to be discovered.

Verified, not asserted

• 274 backend tests and 34 interface tests, needing no infrastructure
• Verification suites that inspect the running host rather than the source
• 4 busy threads on a 1-core tier deliver exactly 1.00 cores
• A 2 GiB allocation against a 1 GiB tier stays capped at 809 MiB resident
• Powered off: cgroup gone, no processes, Incus reports 0 bytes memory
• Across a power cycle: packages, files, /etc edits, Docker images, volumes and containers all survive
• Inside, nproc = 1 and free = 1024 MiB while the host is 4 cores / 7936 MiB

Bugs fixed on the way here

Each was found in production, several reported by customers:

• An SSH lockout caused by the hardening script disabling password authentication when the only key present was the provider's
• Two workspaces could not run at once — Incus registers DNS names per network, not per project
• Docker's leftover nftables rules silently killed all workspace egress while DNS kept working
• apt install firefox failed and wedged dpkg, because Ubuntu ships a stub that installs a snap and snaps cannot run in an unprivileged container
• The reserved SSH and RDP ports were never actually allocated, so new customers saw blank addresses
• A slow stop raised ReadTimeout and parked the workspace in error, which nothing reconciled
• The browser terminal clipped its own last line, because the fit addon measured the border-box height
• English text reached the Persian dashboard, saying "credits" where the product charges Toman
• The file manager displayed //home/dev

Documentation

• AGENTS.md — for an AI agent picking this up cold: the ten traps that cost hours each and are invisible in the code
• docs/ARCHITECTURE.md — where each boundary sits and why
• docs/BILLING.md — the charge model, precisely
• docs/OPERATIONS.md — runbook
• docs/API.md — every endpoint
• docs/DEVELOPMENT.md — conventions
• docs/DECISIONS.md — every non-obvious decision and every bug that has bitten, with the reasoning. The most useful file in the repository

Known gaps

Stated so nobody mistakes them for oversights: single host with no HA and no off-host backup; Docker volumes are sparse zvols with no reservation; the AI feature shares one Claude subscription, which any customer with root can read; no SMTP, so notices are in-dashboard only; self-signed TLS until a domain exists.