Operations Alerting

Production Secrets & Observability Hardening (GH-107); see the ops runbook (local ops records)

; see the ops runbook (local ops records) Status: the logging fix (Scope A) and the alert-delivery/OnFailure wiring; see the ops runbook (local ops records) (Scope B) are versioned and shipped in this repository. The external; see the ops runbook (local ops records) uptime workflow (Scope C) is versioned and runs unconditionally on its; see the ops runbook (local ops records) schedule, but its alert leg — and every host-local alert leg described; see the ops runbook (local ops records) below — is only as live as the ALERT_WEBHOOK_URL value an operator; see the ops runbook (local ops records) configures. This document does not claim that value has been set anywhere; see the ops runbook (local ops records) yet; see "Outstanding operator action" at the end.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) Source: the 2026-08-28 server infrastructure baseline (Ops record, PR #103;; see the ops runbook (local ops records) original captured in git history). DR/off-host backup is tracked separately in #105 and is out of; see the ops runbook (local ops records) scope here.; see the ops runbook (local ops records) ; see the ops runbook (local ops records)

Scope A — secret-safe logging; see the ops runbook (local ops records)

; see the ops runbook (local ops records) Finding: external_contact_sync.main() (the daily; see the ops runbook (local ops records) wecom-external-contact-reconcile timer) called; see the ops runbook (local ops records) logging.basicConfig(level=logging.INFO). httpx/httpcore's own; see the ops runbook (local ops records) request/response logger prints the full request URL at INFO, and WeCom's; see the ops runbook (local ops records) API takes access_token (and, for the token-mint call, corpsecret) as a; see the ops runbook (local ops records) query-string parameter rather than a header — so every reconcile run wrote; see the ops runbook (local ops records) those secrets to journald verbatim (~11.7k lines/24h observed).; see the ops runbook (local ops records) ; see the ops runbook (local ops records) Fix: backend/app/log_safety.py; see the ops runbook (local ops records) installs a process-wide LogRecord factory from app.__init__, before any; see the ops runbook (local ops records) application entrypoint creates an outbound HTTP client. The factory composes; see the ops runbook (local ops records) with any existing factory and scrubs query credentials from the final message,; see the ops runbook (local ops records) deferred %s/mapping arguments, stack info, and URL-bearing exception text; see the ops runbook (local ops records) before a logger propagates to a handler or a formatter writes to a sink.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) It covers application, httpx, httpcore, and other normal Python logger; see the ops runbook (local ops records) records without changing their levels. The investigated outbound URL keys are; see the ops runbook (local ops records) WeCom access_token, corpsecret, and suite_access_token, plus Qiniu's; see the ops runbook (local ops records) short-lived signed-download token; key matching is case-insensitive. A; see the ops runbook (local ops records) handler-compatible filter reuses the same primitive where an embedding process; see the ops runbook (local ops records) needs an explicit handler filter.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) The helper remains explicitly called at app/main.py,; see the ops runbook (local ops records) app/services/wecom_org_authorization.py, and; see the ops runbook (local ops records) app/services/external_contact_sync.py:main() as an idempotent guard for; see the ops runbook (local ops records) embedded/import-order-customized processes.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) What still gets logged (operator diagnosability): main() keeps its; see the ops runbook (local ops records) existing structured summary line; see the ops runbook (local ops records) (external_contact_reconcile status=completed tenant_count=... total=...; see the ops runbook (local ops records) inserted=... failed=...) and per-tenant failure warnings; see the ops runbook (local ops records) (external_contact_reconcile tenant=%s result=failed; see the ops runbook (local ops records) error_class=tenant_processing_failed). Per-request HTTP diagnostics also; see the ops runbook (local ops records) remain available with method, host, path, protocol/status, and non-sensitive; see the ops runbook (local ops records) query parameters; only credential values appear as [REDACTED].; see the ops runbook (local ops records) ; see the ops runbook (local ops records) GH-139 exposure-surface audit:; see the ops runbook (local ops records) ; see the ops runbook (local ops records) | Path | HTTP client | Credential transport | URL-log exposure | Central boundary |; see the ops runbook (local ops records) | --- | --- | --- | --- | --- |; see the ops runbook (local ops records) | External-contact reconcile/refresh, display-name and group-chat sync (app/wecom_contacts.py) | httpx | access_token query parameter | Yes | Redacted |; see the ops runbook (local ops records) | WeCom token mint and OAuth user lookup (app/auth.py, app/routers/auth.py) | httpx | corpsecret and access_token query parameters | Yes | Redacted |; see the ops runbook (local ops records) | Third-party organization authorization | httpx | suite_access_token query parameter | Yes | Redacted |; see the ops runbook (local ops records) | Qiniu internal signed reads | httpx | signed token query parameter | Yes | Redacted |; see the ops runbook (local ops records) | Avatar fetch | httpx | no configured credential query; a provider-signed token URL is still covered | Possible | Redacted |; see the ops runbook (local ops records) | WeChat Pay, Alipay, DeepSeek | httpx | Authorization header or request body, not a credential query | No URL credential identified | Headers/bodies are not emitted by normal httpx request summaries; existing exception paths log a class/fixed message |; see the ops runbook (local ops records) | Branding DNS-over-HTTPS | httpx | none | No | Not applicable |; see the ops runbook (local ops records) | WeCom archive/media SDK | C SDK | SDK arguments, not a Python HTTP URL | No | Not applicable |; see the ops runbook (local ops records) | Self-service activation E2E helper | requests | no credential query; not a production systemd entrypoint | No identified production path | Not applicable |; see the ops runbook (local ops records) ; see the ops runbook (local ops records) No aiohttp, urllib, or urllib3 application call site was found. The; see the ops runbook (local ops records) application-side exception handlers in this surface continue to log only; see the ops runbook (local ops records) type(exc).__name__ or a fixed message; the centralized boundary also handles; see the ops runbook (local ops records) URL-bearing exception text before a handler can format it.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) Tests: backend/tests/test_gh107_log_safety.py.; see the ops runbook (local ops records) ; see the ops runbook (local ops records)

GH-139 historical-exposure disposition; see the ops runbook (local ops records)

; see the ops runbook (local ops records) A read-only, value-free production audit before the GH-139 rollout found the; see the ops runbook (local ops records) following retained matches. It emitted counts and dates only, never matching; see the ops runbook (local ops records) log text or credential values:; see the ops runbook (local ops records) ; see the ops runbook (local ops records)

• wecom-external-contact-reconcile.service journal records: 124,664 retained; see the ops runbook (local ops records) query-secret matches (2026-08-18 through 2026-08-27 UTC).; see the ops runbook (local ops records)
• rsyslog-managed /var/log/messages-*: 222,467 retained query-secret matches; see the ops runbook (local ops records) from 2026-08-05 and 2026-08-18 through 2026-08-28 local log dates.; see the ops runbook (local ops records)
• The affected producer was the external-contact reconciliation path. The; see the ops runbook (local ops records) journal uses SystemMaxUse=500M and MaxRetentionSec=14day; rsyslog is; see the ops runbook (local ops records) active and uses the system's weekly, four-rotation policy for its local; see the ops runbook (local ops records) messages files. No active journal-upload/log-shipper service or active; see the ops runbook (local ops records) rsyslog forwarding action was found. The versioned application backup script; see the ops runbook (local ops records) backs up database/media artifacts, not host logs; cloud/host snapshots remain; see the ops runbook (local ops records) an Ops confirmation item.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) Rotation decision: ROTATION REQUIRED. The pre-GH-107 request URLs exposed; see the ops runbook (local ops records) short-lived access tokens and the long-lived WeCom corpsecret values used by; see the ops runbook (local ops records) the external-contact and OAuth token-mint paths. A WeCom owner must rotate the; see the ops runbook (local ops records) corresponding configured app secrets, update the protected production runtime; see the ops runbook (local ops records) configuration through the approved secret channel, and restart/deploy without; see the ops runbook (local ops records) putting either old or new value in a command, ticket, PR, or log. The agent; see the ops runbook (local ops records) must not perform that console/configuration action.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) Log-handling decision: after rotation, an authorized Ops owner must choose; see the ops runbook (local ops records) between early removal of the identified rotated rsyslog files plus affected; see the ops runbook (local ops records) journal archives, or retention until their configured expiry. Selective; see the ops runbook (local ops records) redaction is not possible for existing journal/syslog records; early removal; see the ops runbook (local ops records) also removes unrelated operational evidence, so it requires explicit approval; see the ops runbook (local ops records) and a post-action count-only rescan. Any cloud snapshot or external-log copy; see the ops runbook (local ops records) must be handled under its owning retention/deletion process.; see the ops runbook (local ops records) ; see the ops runbook (local ops records)

Scope B — host-local alert delivery; see the ops runbook (local ops records)

; see the ops runbook (local ops records) The alert transport already existed before GH-107; see the ops runbook (local ops records) (ssl-renew/notify.sh; end-to-end wiring described in the ops runbook,; see the ops runbook (local ops records) now in local ops records): POST a JSON payload; see the ops runbook (local ops records) {domain, hostname, failed_stage, timestamp, error_summary} to; see the ops runbook (local ops records) ALERT_WEBHOOK_URL; if that variable is unset, log a [DEGRADED] line and; see the ops runbook (local ops records) exit 0 rather than fail the caller. lib/common.sh's filter_secrets; see the ops runbook (local ops records) redacts known secret-shaped tokens from any message before it reaches a; see the ops runbook (local ops records) log line or the webhook payload. What GH-107 adds is coverage for; see the ops runbook (local ops records) scheduled-workload failures, which nothing previously alerted on:; see the ops runbook (local ops records) ; see the ops runbook (local ops records) | Alert source | Mechanism | Status |; see the ops runbook (local ops records) | --- | --- | --- |; see the ops runbook (local ops records) | Disk / inode threshold | scripts/disk_usage_check.sh → notify.sh | pre-existing |; see the ops runbook (local ops records) | Memory / swap pressure | scripts/disk_usage_check.sh → notify.sh | pre-existing |; see the ops runbook (local ops records) | PostgreSQL connection/availability | scripts/disk_usage_check.sh → notify.sh (reports DEGRADED, not a false WARN, when unreachable) | pre-existing |; see the ops runbook (local ops records) | Backup job failure | scripts/backup_once.sh → notify.sh | pre-existing |; see the ops runbook (local ops records) | wecom-archive-worker failure | OnFailure= → wecom-job-failure-alert@.service → notify.sh | GH-107 |; see the ops runbook (local ops records) | wecom-external-contact-reconcile failure | same | GH-107 |; see the ops runbook (local ops records) | wecom-external-contact-refresh failure | same | GH-104 Follow-up A |; see the ops runbook (local ops records) | wecom-billing-lifecycle failure | same | GH-107 |; see the ops runbook (local ops records) | wecom-billing-notifications failure | same | GH-107 |; see the ops runbook (local ops records) | wecom-payment-recovery failure | same | GH-107 |; see the ops runbook (local ops records) | wecom-payment-reconciliation failure | same | GH-107 |; see the ops runbook (local ops records) ; see the ops runbook (local ops records) deploy/systemd/wecom-job-failure-alert@.service is the standard systemd; see the ops runbook (local ops records) "failure notification" recipe: a templated oneshot unit, listed in; see the ops runbook (local ops records) MANAGED_UNITS (so CD deploys it) but never enabled itself — systemd; see the ops runbook (local ops records) instantiates wecom-job-failure-alert@<unit-name>.service on demand when; see the ops runbook (local ops records) a unit with OnFailure=wecom-job-failure-alert@%n.service fails, and it; see the ops runbook (local ops records) calls notify.sh ERROR <unit-name> "systemd unit failed — see journalctl; see the ops runbook (local ops records) -u <unit-name> for detail". It never puts journal content or credentials; see the ops runbook (local ops records) in that message, only the unit name.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) Disk/backup checks were deliberately left alone (no OnFailure= added):; see the ops runbook (local ops records) they already call notify.sh themselves on their own richer WARN/failure; see the ops runbook (local ops records) paths; an OnFailure= alert would be a second, less specific alert for; see the ops runbook (local ops records) the same event.; see the ops runbook (local ops records) ; see the ops runbook (local ops records)

Configuring ALERT_WEBHOOK_URL (server-side); see the ops runbook (local ops records)

; see the ops runbook (local ops records) Store it in backend/.env (production runtime config — the same; see the ops runbook (local ops records) approved, gitignored file that already holds every other production; see the ops runbook (local ops records) secret) as ALERT_WEBHOOK_URL=https://.... Never commit it, never pass it; see the ops runbook (local ops records) on a command line that lands in shell history or ps, never log it —; see the ops runbook (local ops records) filter_secrets also strips ALERT_WEBHOOK_URL=... specifically if it; see the ops runbook (local ops records) ever appears in a captured command's output.; see the ops runbook (local ops records) ; see the ops runbook (local ops records) wecom-job-failure-alert@.service reads the same backend/.env via; see the ops runbook (local ops records) EnvironmentFile=-... (the leading - means a missing file degrades; see the ops runbook (local ops records) instead of failing the alert unit itself).; see the ops runbook (local ops records) ; see the ops runbook (local ops records)

Testing the alert path without a real outage; see the ops runbook (local ops records)

; see the ops runbook (local ops records)

CURRENT_STAGE=manual-test ./ssl-renew/notify.sh WARN "manual-test" "operator-triggered test alert, not a real incident"; see the ops runbook (local ops records)
```; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
Confirm one `[NOTIFY] [OK] webhook delivered (HTTP 2xx)` line and that the; see the ops runbook (local ops records)
message actually arrives at the configured destination.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
### Known compatibility gap — pick an approved destination; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
`notify.sh`'s payload is a **generic** JSON object. Chat-bot webhooks with; see the ops runbook (local ops records)
their own required envelope (e.g. a WeCom group-robot webhook expects; see the ops runbook (local ops records)
`{"msgtype": "text", "text": {...}}`, not this schema) will reject it; see the ops runbook (local ops records)
outright rather than degrade gracefully. Before pointing; see the ops runbook (local ops records)
`ALERT_WEBHOOK_URL` at a real destination, either:; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
- pick a receiver that accepts an arbitrary JSON body (many; see the ops runbook (local ops records)
  generic-webhook/alerting relays do), or; see the ops runbook (local ops records)
- put a small relay/transform in front of the actual chat channel.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
This decision (which destination, and any relay needed) is a product/ops; see the ops runbook (local ops records)
choice this document does not make.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
## Scope C — external uptime monitoring; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
[`.github/workflows/uptime-check.yml`](https://github.com/zuohaisu/wecom-archive/blob/main/.github/workflows/uptime-check.yml); see the ops runbook (local ops records)
runs on a GitHub Actions runner — outside the production ECS — on a; see the ops runbook (local ops records)
`*/10 * * * *` schedule (plus `workflow_dispatch` for an on-demand run or; see the ops runbook (local ops records)
test alert):; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
- `GET https://archive.crowntime.cn/` must return HTTP 200 (following; see the ops runbook (local ops records)
  redirects);; see the ops runbook (local ops records)
- `GET https://archive.crowntime.cn/health/ready` must return HTTP 200; see the ops runbook (local ops records)
  (see `app/main.py`'s `_readiness_body` — 503 on a DB outage or; see the ops runbook (local ops records)
  unmigrated schema, by design, never a bare 200).; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
On failure it POSTs the same generic JSON schema as `notify.sh` to the; see the ops runbook (local ops records)
`ALERT_WEBHOOK_URL` **repository secret** (`Settings → Secrets and; see the ops runbook (local ops records)
variables → Actions` — this is a separate value/store from the server's; see the ops runbook (local ops records)
`backend/.env`; point both at the same destination if you want one; see the ops runbook (local ops records)
channel). If that secret is unset, the run still fails (visible in the; see the ops runbook (local ops records)
Actions tab / any configured GitHub notification), it just cannot also; see the ops runbook (local ops records)
push an outbound alert.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
A scheduled GitHub Actions workflow can be delayed under platform load;; see the ops runbook (local ops records)
treat this as a fast, low-maintenance check, not a hard SLA-grade monitor.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
### Sending a test alert (Scope C); see the ops runbook (local ops records)
; see the ops runbook (local ops records)
Actions → "External Uptime Check" → Run workflow → set; see the ops runbook (local ops records)
`send_test_alert` to `true`. This does **not** touch production; it only; see the ops runbook (local ops records)
verifies the webhook path end-to-end.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
## Operator response runbook; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
| Alert `failed_stage` | Meaning | First response |; see the ops runbook (local ops records)
| --- | --- | --- |; see the ops runbook (local ops records)
| `disk-usage-check` (disk/inode/memory/swap/pg_connections) | A resource threshold was breached, or PostgreSQL was unreachable when checked | `ssh` in, `systemctl status wecom-disk-usage-check.service` / `journalctl -u wecom-disk-usage-check.service -n 50` for the specific failing check; see the ops runbook (local ops records)|
| `backup-once` / backup job name | The nightly encrypted backup did not complete | Check `journalctl -u wecom-backup.service`; re-run manually (`scripts/backup_once.sh`) once the underlying cause is fixed; do not let two consecutive nights fail silently |; see the ops runbook (local ops records)
| `wecom-archive-worker.service` | Sync+decrypt failed for one or more tenants | `journalctl -u wecom-archive-worker.service`; check WeCom credential validity and reachability before assuming a code regression |; see the ops runbook (local ops records)
| `wecom-external-contact-reconcile.service` | Daily full reconcile failed | `journalctl -u wecom-external-contact-reconcile.service`; the incremental refresh path (`wecom-external-contact-refresh.service`) backstops this between daily runs |; see the ops runbook (local ops records)
| `wecom-external-contact-refresh.service` | Incremental refresh queue drain failed (GH-104 Follow-up A: a missing sys.path bootstrap previously made every run fail with `ModuleNotFoundError`, silently, with no alert — this OnFailure= wiring closes that blind spot) | `journalctl -u wecom-external-contact-refresh.service`; the daily full reconcile above is the safety net between refresh cycles, not a substitute for fixing the refresh path itself |; see the ops runbook (local ops records)
| `wecom-billing-*` / `wecom-payment-*` `.service` | A commercial-loop batch failed | Treat as high priority — see [payment-billing-runtime.md](https://github.com/zuohaisu/wecom-archive/wiki/Operations-Payment-Billing); do not manually re-trigger checkout, only investigate the batch job |; see the ops runbook (local ops records)
| `external-uptime-check` (`public_endpoint_or_readiness`) | Production is unreachable or not ready from outside the ECS | Check `archive.crowntime.cn` reachability yourself; `ssh` in and check `systemctl status wecom-archive-365.service`, `nginx`, and `postgresql`; this is the alert most likely to indicate a full host/network outage rather than a single job failure |; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
Runbook ownership: whoever owns `docs/operations/*` for this project; see the ops runbook (local ops records)
today (see `git log` on this file for the most recent editor) is the; see the ops runbook (local ops records)
default owner of this runbook until the team names a specific on-call; see the ops runbook (local ops records)
role; this document does not invent a named owner that doesn't exist yet.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
## Non-goals; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
Unchanged from the parent issue: off-host backup/restore (#105), a full; see the ops runbook (local ops records)
metrics/trace platform, SIEM, HA architecture, or a broad logging refactor; see the ops runbook (local ops records)
unrelated to secret leakage.; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
## Outstanding operator action; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
Code/config in this repository is ready but inert until an operator:; see the ops runbook (local ops records)
; see the ops runbook (local ops records)
1. picks an approved alert destination compatible with the generic JSON; see the ops runbook (local ops records)
   payload (or fronts it with a relay);; see the ops runbook (local ops records)
2. sets `ALERT_WEBHOOK_URL` in production `backend/.env` and confirms a; see the ops runbook (local ops records)
   manual test alert arrives;; see the ops runbook (local ops records)
3. adds the same (or a different, equally approved) value as the; see the ops runbook (local ops records)
   `ALERT_WEBHOOK_URL` GitHub Actions repository secret and runs the; see the ops runbook (local ops records)
   workflow's `send_test_alert` input once;; see the ops runbook (local ops records)
4. deploys `main` so `deploy/systemd/wecom-job-failure-alert@.service` and; see the ops runbook (local ops records)
   the seven updated `OnFailure=` unit files land on the server (steps 7 and; see the ops runbook (local ops records)
   10 of `scripts/deploy_server.sh`; no manual install step needed since; see the ops runbook (local ops records)
   all seven are already in `MANAGED_UNITS`).; see the ops runbook (local ops records)