Automated Device Scraper Engine — Profile Configuration Specification

This document is the technical overview and architecture guide for the configuration schema used by the Automated Device Scraper Engine. The engine uses specialized YAML profiles to drive automated authentication, routing, and DOM-based telemetry harvesting across network-connected infrastructure assets.

Contents

• Architecture Overview
• 1. Tool Manifest Profile (tools/*.yaml)
• 2. Page Mapping Matrix (tools/pages/*.yaml)
• 3. Scripts Repository Inventory (tools/scripts/*.yaml)
• Delay Resolution Order
• Script Run Ordering
• Login Retry Behavior
• Per-Run Cache Clearing
• Ignoring SSL Certificate Errors
• Experimental CDP Async Execution
• Debug DOM Snapshots
• Worker Viewport & Browser Console Logging
• Script Failure Isolation & Reporting
• Status Dashboard (Form2)
• Custom Status Rules
• Event Log (Form1)
• Live Config Reload
• Global App Config (app_config.yaml)
• Tests
• License

---

Architecture Overview

The system decouples device access credentials, execution route definitions, and data-harvesting logic into three distinct layers:

Layer	Location	Responsibility
1. Tool Manifest Profiles	tools/*.yaml	Device identity, network location, login mechanics, execution scope
2. Page Mapping Matrices	tools/pages/*.yaml	Sequences interrogation scripts against explicit application routes / URL hashes
3. Scripts Repository	tools/scripts/*.yaml	Reusable JavaScript primitives for DOM parsing and data extraction

A tool manifest pulls in its page files via pageFilePatterns, and each page entry pulls in its scripts by name from the scripts repository — so a single script can be shared across multiple pages and tools.

Parallel-mode per-worker login: when pages run in parallel, each page-worker tab performs its own interactive login before navigating to its target page — sharing the environment's cookies is not sufficient by itself for every device. URL hash fragments (#page:...) are never sent to the server, so a shared cookie jar should in principle behave identically to opening several browser tabs at the same URL; some device apps nonetheless gate access on tab-scoped state (e.g. sessionStorage, which browsers never share across tabs even within the same profile) that's only populated by actually submitting the login form in that tab. This was diagnosed via a debug DOM snapshot showing a rejected worker tab stuck on the login page regardless of session cookies or wait time.

---

1. Tool Manifest Profile (tools/*.yaml)

Governs asset initialization, automated credential injection, and page discovery settings.

Schema

Property	Type	Description
toolName	String	Unique system identifier for the device deployment.
predefinedUser	String	Authentication username injected into interactive login sequences.
predefinedPass	String	Authentication password injected into interactive login sequences.
addr	String	Fully qualified base URL endpoint (e.g. http://192.0.2.10).
pageLogin	String	Relative route path where authentication must be forced (e.g. / or /login.php).
elementName	String	CSS selector targeting the username input field.
elementPass	String	CSS selector targeting the password input field.
elementSubmit	String	CSS selector targeting the login button / form submission node.
loginRetryAttempts	Integer	Max login attempts before giving up. See Login Retry Behavior. Default: 1 (no retry).
loginRetryDelaySeconds	Integer	Pause between login retry attempts. See Login Retry Behavior. Default: 1.
bypassLogin	Boolean	If true, skip the login page/handshake for this tool entirely and go straight to navigating and scripting its pages — for devices/dashboards with no authentication at all. Default: false. In parallel mode this also skips creating the master login worker (nothing to anchor); in sequential mode it skips the login attempt and the post-login settle delay.
forcePageRunMode	String	parallel | sequential. Per-tool override for page run mode — wins over the global "Execute Tool Pages in Parallel" checkbox for this tool only. Omit (or use an unrecognized value) to fall back to the global checkbox; an unrecognized value is logged as a warning.
disableCache	Boolean	Whether to clear this tool's CacheData/<ToolName> folder after every run. See Per-Run Cache Clearing. Omit to fall back to defaultDisableCache in app_config.yaml.
ignoreSslCertError	Boolean	If true, ignore TLS certificate errors (ERR_CERT_AUTHORITY_INVALID, ERR_CERT_DATE_INVALID, ERR_CERT_COMMON_NAME_INVALID, etc.) for this tool instead of blocking navigation — for devices with an untrusted/self-signed/expired certificate on their web UI. Default: false. See Ignoring SSL Certificate Errors.
pageFilePatterns	Array	Wildcard filename patterns used to discover this tool's page files under tools/pages/.

Example — tools/GXP1610_site1_1001.yaml

toolName: GXP1610_site1_1001
predefinedUser: admin
predefinedPass: secret
addr: http://192.0.2.10
pageLogin: /
elementName: '[class="gwt-TextBox"]'
elementPass: '[class="gwt-PasswordTextBox"]'
elementSubmit: '[class="gwt-Button"]'
loginRetryAttempts: 6
loginRetryDelaySeconds: 5
forcePageRunMode: sequential
pageFilePatterns:
  - "gxp1610_*.yaml"

---

2. Page Mapping Matrix (tools/pages/*.yaml)

Maps explicit URL hash routes or paths to the script extraction sequences that should run on them.

Schema

Property	Type	Description
destinationUrl	String	Target URL to navigate to. Can be a relative path (/status/acts) — resolved against the owning tool's addr at load time via ToolConfig.ResolveUrl, the same convention pageLogin/BaseLoginUrl already uses — or a fully-qualified absolute URL/hash route (http://192.0.2.10/#page:status_account), which passes through unchanged. Prefer the relative form for new tools: it avoids repeating the device's scheme+host in every page entry, and an addr change only needs updating in one place.
scripts	Array	Ordered list of scripts to execute inside this page's window context.

Script Invocation (scripts[])

Property	Type	Description
name	String	Key matching an entry in the Scripts Repository.
optionsOverride	Object	Per-usage overrides for the script's own base options.
optionsOverride.DelaySeconds	Integer	Overrides the post-navigation cooldown before this script runs, for this page only. See Delay Resolution Order.
optionsOverride.RunIndex	Integer	Overrides the script's own base options.RunIndex for this page only. See Script Run Ordering.
optionsOverride.FailureLevel	String	Overrides the script's own base options.FailureLevel for this page only. See Script Failure Isolation & Reporting.
optionsOverride.UseExperimentalCdpAsync	Boolean	Overrides the script's own base options.UseExperimentalCdpAsync for this page only. See Experimental CDP Async Execution.

Example — tools/pages/gxp1610_01.yaml

- destinationUrl: "http://192.0.2.10/#page:status_account"
  scripts:
    - name: "ExstractPhoneRegistrationStatus"
    - name: "ExtractPhoneHookStatus"
      optionsOverride:
        DelaySeconds: 3

---

3. Scripts Repository Inventory (tools/scripts/*.yaml)

Reusable, pure-JavaScript data-harvesting primitives that execute inside rendered window environments.

Schema

Property	Type	Description
name	String	Unique registry key matched by page mapping invocations.
returnType	String	DeviceMeta | ProblemList. Tells the UI parser how to process the payload.
options	Object	Base/default options for this script, used whenever a page invocation doesn't supply its own optionsOverride.
options.DelaySeconds	Integer	Default post-navigation cooldown before this script's DOM interrogation runs. See Delay Resolution Order.
options.RunIndex	Integer	Default execution group for this script within a page. See Script Run Ordering. Default: 0.
options.FailureLevel	String	common | important | critical. How this script's execution failure affects the rest of its page's run. See Script Failure Isolation & Reporting. Default: common.
options.UseExperimentalCdpAsync	Boolean	If true, run this script through the experimental CDP-based executor instead of the standard one — required for a script using real async/await/fetch. See Experimental CDP Async Execution. Default: false.
scriptText	Block String	Self-executing JavaScript anonymous function ((function(){...})();). Must return a serialized JSON string.

returnType: DeviceMeta maps key/value metric rows; ProblemList surfaces flat operational errors.

Required Return Formats

returnType: "DeviceMeta" — must return a JSON string matching:

{
  "deviceName": "String (Identifier Context Label)",
  "deviceData": [
    { "name": "Metric Property Name", "value": "Current Observed Value" }
  ]
}

returnType: "ProblemList" — must return a flat array identifying active error incidents:

[
  { "issueName": "Alert Description text", "severity": "disaster-bg | average-bg | warning-bg" }
]

Example — tools/scripts/gxp1610.yaml

- name: ExstractPhoneRegistrationStatus
  returnType: "DeviceMeta"
  options:
    DelaySeconds: 30
  scriptText: |
    (function() {
        let deviceName = document.querySelector('[id="topBanner"] .gwt-HTML')?.innerText || 'Unknown';
        let deviceData = [];
        document.querySelectorAll('[class="table-row"]')?.forEach((row) => {
            const cols = Array.from(row.querySelectorAll('td'));
            if (!cols[1]?.innerText) return;
            deviceData.push({ name: cols[0]?.innerText, value: cols[3]?.innerText });
        });
        return JSON.stringify({ deviceName: deviceName, deviceData: deviceData });
    })();

---

Delay Resolution Order

Each script executes only after a post-navigation cooldown, resolved in this order (later wins):

1. Base default — the script's own options.DelaySeconds in tools/scripts/*.yaml. Applies everywhere that script is used unless overridden below.
2. Page-level override — optionsOverride.DelaySeconds on that script's entry in tools/pages/*.yaml. Wins over the base default for that specific page invocation only.
3. Engine fallback — if neither is set anywhere, the engine falls back to 5 seconds.

Cold WebView2 tabs (a page-worker's very first navigation, as used by parallel page execution) need noticeably longer than a warm, already-bootstrapped session to render a single-page app's DOM — tune DelaySeconds accordingly per script rather than assuming one value fits every page.

Script Run Ordering

Every script on a page has a resolved RunIndex, following the same base/override precedence as DelaySeconds above (options.RunIndex on the script is the default; optionsOverride.RunIndex on that page's invocation wins if set). Default RunIndex is 0.

Within one page's script list:

1. Scripts are grouped by RunIndex.
2. Groups run in ascending index order — every script in a lower-numbered group finishes before the next group starts.
3. Scripts within the same group run concurrently against each other (their ExecuteScriptAsync calls are issued without waiting for one to finish before starting the next) — each still waits out its own resolved DelaySeconds first, so scripts in a group don't necessarily start at the exact same instant, but they don't block on one another either.

Because the default is 0 for every script, scripts with no explicit RunIndex all run in the same group — i.e. in parallel by default. To force strict ordering between specific scripts (e.g. a script that depends on a prior one changing state), give them increasing RunIndex values via options.RunIndex (or optionsOverride.RunIndex for just one page).

Note this is concurrency in issuing the calls, not real multi-threaded execution inside the page — all scripts on a page still run against the same single-threaded JS context in that one browser tab. The benefit is skipping the accumulated wait: N scripts with a 3s DelaySeconds each cost roughly 3×N seconds run one-by-one, but roughly 3 seconds run together in one RunIndex group.

Login Retry Behavior

The login handshake script (StandardLoginHandshake) reports a form-not-in-DOM-yet condition by returning a string starting with ERROR: instead of throwing — this doubles as a readiness probe for single-page apps whose login form hasn't rendered when the first attempt runs.

On that signal, the engine waits loginRetryDelaySeconds and re-runs the login script without re-navigating (the page is still loading, not broken), up to loginRetryAttempts total attempts. loginRetryAttempts: 1 (the default) performs no retry.

Per-Run Cache Clearing

After every worker for a tool has finished (success or failure), the engine — by default — deletes that tool's own CacheData/<ToolName> WebView2 profile folder, and only that tool's folder, never a sibling tool's. This forces a genuine fresh login on the next run instead of silently reusing a still-valid persisted session cookie.

This matters for continuous-run mode: a site whose login issues a persistent (non-session) cookie stays authenticated across cycles, so navigating to the login URL on a later cycle redirects straight past the login form to the dashboard — the login script then correctly reports "form not found" (there's nothing to find), which used to be misread as a login failure and abort the whole tool for that cycle. Clearing the cache after each run means every cycle starts from zero and gets a real SUCCESS login response, not an ambiguous one.

Per-tool override — disableCache (tools/*.yaml):

Value	Behavior
true	Always clear this tool's cache after every run (the behavior above).
false	Never clear it — preserve the WebView2 profile (browser cache, not just cookies) across runs.
unset	Falls back to defaultDisableCache in app_config.yaml.

Some devices are slow/weak enough that a fully cold profile (no cached JS/HTML assets, not just no cookies) can't finish bootstrapping its SPA within the login retry window — clearing the cache on every run made login fail every time for exactly this reason. Set disableCache: false on that tool to keep its cache warm across runs instead. GXP1610_site1_1001.yaml has this set for that reason.

Ignoring SSL Certificate Errors

Some devices (e.g. embedded web UIs like the GXP1610) serve HTTPS with a self-signed, expired, or otherwise untrusted certificate. Without this flag, Chromium/WebView2 blocks navigation entirely with an interstitial (ERR_CERT_AUTHORITY_INVALID, ERR_CERT_DATE_INVALID, ERR_CERT_COMMON_NAME_INVALID, etc.), which the scraper has no way to click through.

Per-tool override — ignoreSslCertError (tools/*.yaml): default false. When true, ParallelScraperManager applies two layers so certificate errors never block navigation:

1. CreateEnvironmentOnUiAsync builds the tool's CoreWebView2Environment with CoreWebView2EnvironmentOptions.AdditionalBrowserArguments = "--ignore-certificate-errors" — a Chromium launch argument that suppresses the interstitial at the browser-process level, before any page-level event would even fire. This is the primary mechanism and applies to every worker sharing that environment (master login + all page workers), since it's set once per tool run.
2. InstantiateBrowserContextOnUiAsync additionally subscribes CoreWebView2.ServerCertificateErrorDetected and sets e.Action = CoreWebView2ServerCertificateErrorAction.AlwaysAllow whenever ignoreSslCertError is set — a second layer in case some certificate error surfaces as an event instead of being caught by the launch argument. Each occurrence logs a WARN line (Ignoring TLS certificate error (<status>) for <url> per ignoreSslCertError=true.) so silently-bypassed certificate problems stay visible in scraper_execution.log.

Since this is applied per-CoreWebView2Environment (i.e. per tool run, not globally), tools without the flag set are unaffected — a misconfigured/compromised certificate on one device doesn't loosen validation for any other tool.

Experimental CDP Async Execution

The problem: CoreWebView2.ExecuteScriptAsync — the standard way this engine runs every script — does not correctly wait for a script's returned Promise. Confirmed with a standalone WebView2 test harness across every syntax variant:

Script shape	ExecuteScriptAsync result
Plain synchronous function	Correct value
async function invoked with no top-level await	{} — a bare, un-awaited Promise serializes to an empty object (Promises have no enumerable own properties)
await (async function(){...})(); (await on the call)	null
(await (async function(){...}))(); (await on the bare function reference before calling it)	null
Bare top-level Promise.resolve(...), no async function at all	{}

Any script using real async/await/fetch and returning its result the normal way silently comes back as {} or null — indistinguishable from a real failure, and this is not fixable by changing where await/parens go. This is what broke ExtractPhoneHookStatus_GRP2614's original fetch-based implementation (see git history / prior investigation) — worked perfectly when the same code was pasted into DevTools Console (which has its own REPL-style promise handling), never worked through ExecuteScriptAsync. The newer CoreWebView2.ExecuteScriptWithResultAsync (adds Succeeded/ResultAsJson/Exception) has the exact same limitation — it gives real exception details for a synchronous throw, but still returns {} for an un-awaited Promise.

Root cause: ExecuteScriptAsync is a thin wrapper around the Chrome DevTools Protocol method Runtime.evaluate, which has an awaitPromise parameter the WebView2 SDK's own wrapper does not set. Calling Runtime.evaluate directly (via CoreWebView2.CallDevToolsProtocolMethodAsync, the same mechanism already used for browser console logging) with awaitPromise: true and returnByValue: true does correctly resolve the promise — confirmed against a real local HTTP endpoint in the same test harness.

The fix — opt-in, not a default. ExperimentalCdpScriptExecutor.cs is a small, hand-written class that mirrors CoreWebView2.ExecuteScriptAsync's shape (Task<string> ExecuteScriptAsync(CoreWebView2, string)) and, deliberately, its result format — a JS string return value comes back double-JSON-encoded exactly like the standard path, verified byte-for-byte identical for string/JSON-string/number/bool/null/object return values in the test harness. This means UnwrapJsonEncodedString, StatusLineDto, and Form1.Manager_DataScraped don't need to know or care which path actually ran a script.

One deliberate improvement over the standard path: an uncaught JS exception throws a real CoreWebView2ScriptExecutionException instead of silently completing with "null" — this lets RunSingleScriptAsync's existing retry/FailureLevel/snapshot logic react to it exactly like it already does for the standard path's genuine (rare) exceptions.

options.UseExperimentalCdpAsync (base on the script in tools/scripts/*.yaml, overridable per-page via optionsOverride.UseExperimentalCdpAsync — same base/override precedence as DelaySeconds/RunIndex/FailureLevel): default false (the standard ExecuteScriptAsync path, unchanged). Set true on a specific script to run it through ExperimentalCdpScriptExecutor instead, enabling normal async function/await fetch(...) syntax. Deliberately per-script, not per-tool or global — opting one script in never affects any other script's execution path.

Script syntax gotcha specific to this path: write the script as a plain (async function() { ... })(); invocation — no await keyword outside the function. ExecuteScriptAsync tolerates (and silently mishandles) a leading await (async function(){...})();/(await (async function(){...}))();, but raw CDP Runtime.evaluate does not parse top-level await at all and fails with ReferenceError: await is not defined — a real, visible exception now (see the exception-throwing improvement above), not another silent null. await is only valid inside the async function body, exactly like ExtractPhoneHookStatus_GRP2614's working version.

Why this stays experimental / opt-in rather than becoming the default: most of what's risky about bypassing the official SDK method is addressable with code — and has been (the format-parity work and the real-exception behavior above are exactly that). The one risk that genuinely isn't fixable this way: the Chrome DevTools Protocol's method/parameter surface — unlike the versioned CoreWebView2 API — carries no stability guarantee across the Chromium versions bundled with future WebView2 Runtime auto-updates (Microsoft's own documented position). A Runtime auto-update on the deployment machine could in principle change or drop Runtime.evaluate's behavior without any change to this application at all. Scripts that don't need real async/await should stay on the standard path; only opt a script in when it genuinely needs fetch/await and a synchronous alternative (e.g. a blocking XMLHttpRequest, xhr.open(method, url, false)) isn't a better fit for that specific case.

Debug DOM Snapshots

When minLogLevel: "DEBUG" in app_config.yaml, the engine dumps the live page DOM (document.documentElement.outerHTML, post-render — not the original server response) to disk so a failed extraction or login can be inspected after the fact instead of re-running the tool with guessed delay values. Taken only on failure, in two places:

• Right after an extraction script fails (an exception from ExecuteScriptAsync — a WebView2 fault, timeout, navigation drop, etc.) — one snapshot per failed attempt, not on every successful run.

• Right after a failed login attempt — one snapshot per failed attempt in the loginRetryAttempts loop, so a login that fails 3 times in a row leaves 3 separate snapshots showing the DOM at each attempt (not just the final one).

• Location: snapshots/ (created next to the executable, alongside tools/ and CacheData/).

• Filename: {ToolName}_{ScriptName}_{yyyyMMdd_HHmmss_fff}.html — {ScriptName} is <scriptName>_attempt{N} for a failed extraction script, or StandardLoginHandshake_attempt{N} for a failed login.

• Log line: scraper_execution.log gets a DEBUG entry — Snapshot for script '<name>' created: <filename> — right after each snapshot is written.

• At any other minLogLevel (INFO and above), no snapshot is taken and no extra WebView2 round-trip happens — this is a debug-only diagnostic, not part of normal extraction.

Worker Viewport & Browser Console Logging

Every worker (master login, sequential, and each parallel page) is a WebView2 control created in ParallelScraperManager.InstantiateBrowserContextOnUiAsync. Two things about how it's set up:

• Viewport size: it used to be left at WinForms' default control size — nowhere near a real browser window. It's now explicitly sized to 1280x800 and positioned far off-screen (not hidden — Visible stays true, since setting it false would make Chromium report document.visibilityState as "hidden" to the page, and some SPAs use that to defer rendering, reintroducing the same class of problem). This is a fix under active investigation for SPAs that do viewport-dependent layout/initialization and can silently fail to render correctly in a degenerate-sized surface, independent of timing or cache.
• Browser console logging: when minLogLevel: "DEBUG", the page's own DevTools Console — console.log/warn/error calls and uncaught JS exceptions — is captured via the Chrome DevTools Protocol (Runtime.enable + the Runtime.consoleAPICalled/Runtime.exceptionThrown events; WebView2 has no simpler high-level "console message" event) and written to scraper_execution.log, tagged [BROWSER CONSOLE:<type>] or [BROWSER EXCEPTION]. This surfaces real JS errors happening during page load/SPA bootstrap that were previously invisible — the same information you'd see in a normal browser's DevTools Console tab.

Script Failure Isolation & Reporting

A single script failing (WebView2 fault, navigation drop, script timeout, etc.) no longer aborts the whole page or tool run by default — it's caught per-script, logged at ERROR level in scraper_execution.log, and reported in Form1's status window (prefixed [ERROR], or [CRITICAL] per below) so it's visible without opening the log file. The same applies to login failures and other fatal per-tool faults.

Exactly what happens next is controlled per-script by FailureLevel (base options.FailureLevel, overridable per page via optionsOverride.FailureLevel — same precedence as Delay Resolution Order):

Level	Behavior on failure
common (default)	Logged and reported; nothing else is affected — the rest of the current RunIndex group and all later groups on this page still run.
important	Retried up to 3 attempts total, waiting the script's own resolved DelaySeconds between attempts. If it's still failing after the last attempt, it's treated like common from there — logged, reported, and the rest of the page's run continues.
critical	Logged and reported (tagged [CRITICAL], not [ERROR]) — no retry, and no further RunIndex groups run for this page (scripts already running in the same group still finish, but the next group is skipped).

Status Dashboard (Form2)

• Row identity: every row carries a RowHash — a SHA-256-derived id (see RowHasher.ComputeHash in StatusLineModels.cs) computed from (Tool, Page, Script), replacing what used to be a 3-field string comparison with a single hash equality check. Re-running a script replaces every row sharing its batch hash instead of appending — the whole batch is swapped as a unit, so a metric that stops appearing (a resolved problem, a vanished extension) is cleared instead of staying behind as dead data. Note this also applies to a script that legitimately returns 0 rows on a bad/transient scrape — its prior rows get cleared too, not just on a good 0-row result.

• Hash column: the last column displays ... for every row; hover over it to see a tooltip with the row's full hash and the Tool/Page/Script that produced it — useful for confirming why a row did or didn't get replaced. It's a per-cell tooltip (only shows over that column), implemented via ListView.HitTest in MouseMove, since ListView only natively supports one tooltip per whole row.

• Sort order: rows are sorted by severity — Disaster → Average (also covers Zabbix's native high-bg severity, treated the same as Average) → Warning → Info → no/unrecognized status — top to bottom. Within the same severity tier, newest results stay on top.

• Copy to clipboard: select one or more rows and press Ctrl+C (or right-click → Copy Selected Rows) to copy them as tab-separated text with a header row — pastes directly into Excel or a .csv file.

• No manual Clear/Acknowledge buttons: removed — row lifecycle (add/update/clear) is now fully automatic per the replace-on-rescrape behavior above, so there was nothing left for them to meaningfully do.

• Sound/speech alerts: when a row's severity is at or worse than the Sound Alert From toolbar dropdown (Off / Info / Warning / Average / Disaster — default Warning, matching Warning/Average/Disaster all alerting by default), the dashboard plays SystemSounds.Exclamation and, if Speak Device/Metric is checked (default on), announces the alert via Windows' built-in TTS (System.Speech.Synthesis.SpeechSynthesizer — a .NET Framework assembly, not a NuGet package, using whatever SAPI voice is already installed on the machine; no internet or extra install needed, safe for an offline target PC):

  • DeviceMeta rows: "Alert. <device>. <metric>." — or, when the status came from a saved custom rule (see Custom Status Rules below), "Alert. <device>. <metric>. <condition>.", e.g. "Alert. Zabbix APC Bunker Input Voltage. 230 is below 250.". The condition phrase (StatusLineItem.CustomRuleDescription) is only present for rule-driven matches — the built-in fallback rules (ping down, battery capacity, etc.) have no saved comparison to describe, so those still speak just the metric name.
  • ActiveProblems rows: "Alert. <device>. <issue text>." — MetricOrIssueName is always the generic "Active Incident" for problems, so the actual issue text (Value) is spoken instead.

  A separate Speak Problem Details checkbox (default checked, not saved to app_config.yaml — always resets to checked when the dashboard reopens) controls only whether the issue text is included for ActiveProblems rows: checked says "Alert. <device>. <issue text>."; unchecked still announces "Alert. <device>.", just without the detail — it doesn't silence problem announcements entirely, that's what Speak Device/Metric is for. The alert sound always plays regardless of either toggle. Speak Device/Metric and Sound Alert From are saved to app_config.yaml (alerts.threshold, alerts.speakDeviceAndMetric) and restored on restart.

  Each (Tool, Device, Metric) alerts once when it first crosses into the qualifying severity — it does not replay on every subsequent scrape while it stays bad, only when it recovers (or disappears from that script's results entirely) and then degrades again. This state is tracked per-dashboard-instance (Form2._activeAlertKeys); closing and reopening the dashboard forgets prior alert history and will re-announce any still-active problems on the next scrape. For ActiveProblems rows, the alert key also includes the problem's id (see ActiveProblemPayload.Id / tools/scripts/zabbix.yaml's data-eventid), since MetricOrIssueName alone is always the generic "Active Incident" — without the id, two different simultaneous problems on the same device would collide onto one alert key and only the first would ever sound.

• Always On Top: a toolbar checkbox that pins the dashboard window above other windows (this.TopMost). Not persisted across restarts — it's a per-session toggle. The Define Custom Rule Threshold... dialog inherits this — it's shown with dlg.TopMost = this.TopMost and ShowDialog(this) — otherwise a non-TopMost modal dialog can render behind a TopMost owner in the OS z-order and become unclickable.

Custom Status Rules

Right-clicking a DeviceMeta row → Define Custom Rule Threshold... opens a dialog (FormulaBuilderDialog in Form2.cs) that saves a per-(Tool, Device, Metric) rule to formulas.json, evaluated by FormulaPersistenceManager.EvaluateMetric every time that metric is re-scraped, overriding the built-in fallback rules (StatusLineDto.DetermineMetaConditionFormula).

• Data Casting Type: Int | Float | Double | String.
• Check Operator: =, !=, >, < for numeric types; =, !=, contains for String; any (matches unconditionally, ignoring both the operator's usual comparison and the data type) for all types.
• Numeric comparisons are culture-invariant. All parsing (EvaluateNumeric, and the Int-cast validation in FormulaBuilderDialog.BtnSave_Click) explicitly uses CultureInfo.InvariantCulture. This was a real, previously-undiscovered bug: scraped values and this app's own regex extraction always use . as the decimal point, but double.TryParse/float.TryParse without an explicit culture default to CultureInfo.CurrentCulture — on a machine whose OS culture uses , as the decimal separator (e.g. uk-UA), that silently made double.TryParse("230.4") return false, so every numeric rule with a fractional value failed on exactly the locale this app is actually deployed on, independent of the Int-truncation logic itself being correct. Covered by NumericComparison_IsCultureInvariant in the test project (see Tests).
• Int comparisons truncate rather than round — 100.9 truncates to 100, not 101 — both the scraped value and the saved threshold.
• Matched rules describe themselves for TTS. EvaluateMetric has an overload returning out string matchDescription — a human-readable phrase for the specific comparison that fired, e.g. "230 is below 250", "93 equals 93", "OK is not Fail", "Battery Fail contains Fail". Per-operator verbs: > → is above, < → is below, != → is not, = → equals, contains → contains. It's null when no rule matched, or when the rule's operator is any (there's no specific value comparison to describe). StatusLineDto.FromDeviceMeta threads this through to StatusLineItem.CustomRuleDescription, which Form2.TriggerAlert appends to the spoken alert (see Sound/speech alerts above). The plain 4-arg EvaluateMetric overload (no out parameter) still exists for callers that don't need the description and delegates to the 5-arg version.

Event Log (Form1)

Form1's StatusBox — every [SYSTEM]/[ERROR]/per-tool log line printed to the main window — is capped at 5000 lines (Form1.MaxStatusLogLines), FIFO: once the cap is hit, the oldest lines are dropped first as new ones arrive, so a long-running continuous session doesn't grow the textbox's text unbounded over hours or days. All appends go through a single AppendStatusLine helper (not StatusBox.AppendText directly) so the cap is enforced in one place. This only trims the on-screen log — scraper_execution.log on disk is unaffected and keeps everything.

Live Config Reload

tools/*.yaml, tools/pages/*.yaml, and tools/scripts/*.yaml no longer need an application restart to pick up an edit. ToolRepository and ScriptsRepository are still Lazy<T> singletons eagerly loaded once at startup (see Program.cs), but each now also exposes a Reload() method that re-reads every file from disk and atomically swaps in a fresh dictionary (guarded by a lock, so a reload can never observe a half-populated repository mid-swap).

Form1.RunSelectedToolsOnceAsync calls ToolRepository.Instance.Reload() and ScriptsRepository.Instance.Reload() at the start of every run — both the single manual "Start Scraper Engine" click and, importantly, every cycle of the continuous run loop. The checkbox selection itself (which tool names are checked) is still captured once when you click Start/Start Loop, but the actual ToolConfig object for each selected name is re-resolved fresh from the just-reloaded repository every single time, not reused from a stale snapshot — so a fix to a destinationUrl, a tweaked script, a changed delay, or an edited credential takes effect on the very next run/cycle without stopping the app. If a selected tool's name no longer resolves after a reload (e.g. its file was deleted or renamed), that tool is skipped for that run with a WARN log line rather than aborting the whole run.

This does not cover adding a brand-new tool file while the app is running — the tool checkbox list in the engineListView is still only built once at Form1 startup from ToolRepository.Instance.GetAllTools(), so a newly-added tool has no checkbox to select until the app is restarted. Editing an existing tool's config (the common case — fixing a URL, adjusting a script, etc.) needs no restart.

Global App Config (app_config.yaml)

A single YAML file at the executable's root (renamed from the old logger_config.yaml, which only held logger settings) holding logger settings, engine-wide defaults, and remembered Form1 UI state. Loaded once by AppConfig.Instance — explicitly initialized as the very first line of Program.Main(), before Logger (which now reads its own settings from AppConfig.Instance.Data instead of parsing its own copy of the file) or anything else touches configuration.

# Supported levels in order of severity: DEBUG, INFO, WARNING, ERROR, CRITICAL
minLogLevel: "DEBUG"
logToFile: true
logFilePath: "scraper_execution.log"

# Global default for a tool's disableCache when tools/*.yaml doesn't set its own value.
defaultDisableCache: true

# Remembered Form1 UI state - saved automatically when changed, restored on startup.
ui:
  parallelTools: true
  parallelPages: false
  intervalSeconds: 60

# Remembered Form2 sound/speech alert settings - saved automatically when changed, restored on startup.
alerts:
  threshold: "Warning"
  speakDeviceAndMetric: true

Property	Type	Description
minLogLevel	String	See Debug DOM Snapshots — must be "DEBUG" for snapshot capture to run at all.
logToFile	Boolean	Whether log lines are also appended to logFilePath, not just System.Diagnostics.Debug.
logFilePath	String	Relative (to the executable) or absolute path for the log file.
defaultDisableCache	Boolean	Global fallback for disableCache on any tool that doesn't set its own. See Per-Run Cache Clearing.
ui.parallelTools	Boolean	Restores/persists the "Execute Selected Tools in Parallel" checkbox.
ui.parallelPages	Boolean	Restores/persists the "Execute Tool Pages in Parallel" checkbox.
ui.intervalSeconds	Integer	Restores/persists the continuous-run interval input.
alerts.threshold	String	Off | Info | Warning | Average | Disaster. Restores/persists Form2's "Sound Alert From" dropdown. See Status Dashboard.
alerts.speakDeviceAndMetric	Boolean	Restores/persists Form2's "Speak Device/Metric" toggle.

The ui and alerts sections are written back to disk (via AppConfig.Instance.Save()) immediately whenever any of those controls changes — not just on exit — so the last-used values survive even an unclean shutdown.

Tests

WebToolsDataMonitor.Tests (modern SDK-style project, net48, MSTest) sits alongside the main project in WebToolsDataMonitor.slnx and references WebToolsDataMonitor.csproj directly via ProjectReference — no code duplication, it exercises the exact production classes.

Currently covers FormulaPersistenceManager.EvaluateMetric/EvaluateNumeric (see Custom Status Rules): every check operator × data type combination, the Int-decimal-truncation regression, negative numbers, null rawValue/ComparisonValue (the null-safety fix), an unparseable value, no matching rule, and the culture-invariance regression (temporarily switches the running thread to uk-UA to prove .-decimal values still parse).

Run via dotnet test from WebToolsDataMonitor.Tests/. Two things worth knowing before adding more:

• FormulaPersistenceManager keeps its saved rules in a shared static list for the process's lifetime, persisted to formulas.json next to the test binary — there's no reset hook. Every test uses its own unique (Tool, Device, Metric) triple (a Guid suffix via a [CallerMemberName] helper) so tests can't see or clobber each other's rules despite sharing that state.
• The assembly has [assembly: DoNotParallelize] for the same reason — parallel test execution isn't inherently unsafe given the unique keys, but there's no need to rely on that when the tests only take milliseconds anyway.

License

See LICENSE. Free to use, copy, modify, and incorporate into other projects — including commercially — but any redistribution (source, binary, fork, or as part of a larger work) must credit ivan-odintsov and link back to github.com/ivan-odintsov/WebToolsDataMonitor.