Skip to content

v0.2.0-beta.4

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 29 Aug 18:18
v0.2.0-beta.4
f7b244b

Automation Sandbox v0.2.0-beta.4

Preview NuGet package artifact release. Rolls up the work merged since v0.2.0-beta.3.

Breaking Changes

  • SimilarityWeights.MinimumLlmConfidence has been removed (#253). It stopped gating LLM acceptance when the independent-agreement quorum landed (v0.2.0-beta.3), but stayed on the public configuration surface as a no-op. Code that sets it (new SimilarityWeights { MinimumLlmConfidence = ... }) no longer compiles; delete the assignment. LLM acceptance is controlled by MinimumConsensusVotes only. Self-reported confidence is still recorded for audit, never compared or thresholded.

Highlights

  • Healing modes (#248): SelfHealingEngine takes an explicit HealingMode - Observe (record only), Review (propose, never apply), AutoHeal (apply and retry), FailClosed (never heal, rethrow immediately). The default is Review, so a heal is never silently applied unless you ask for it.
  • Threshold profiles and per-application calibration (#252): Conservative / Balanced / Aggressive presets for SimilarityWeights, plus a calibrate CLI command that sweeps a repository's own locator history and reports the false-heal / recall trade-off at each threshold. The shipped MinimumConfidence stays 0.50; calibrate against your own UI rather than treating a measured range as portable.
  • Secret and PII redaction on by default (#249, #202, #270): DOM text and TestIntent strings are scrubbed of API keys and secrets (Stripe, GCP service-account keys, generic bearer tokens) before any of it reaches an LLM provider. The TextSanitizer hook is opt-in for custom rules on top.
  • Prompt-injection hardening (#250): LlmHealingPrompt isolates candidate element data from the instruction envelope, so injected directives inside element text cannot steer the model's pick.
  • xUnit and NUnit fixture helpers (#262, #283): SelfHealing.Testing.SelfHealingTestBase drops self-healing into a consumer test suite with no per-test wiring, verified end-to-end under both runners.
  • Playwright end-to-end sample (#251): a runnable consumer showing live web discovery, intent-driven test generation, self-healing, and the safety guardrails together.
  • DORA and engine-reliability metrics workflow (#323, #328): a weekly (and manual) PowerShell report - deployment frequency, lead time for changes, change-failure rate, MTTR - alongside the self-healing gate thresholds read straight from SimilarityWeights.cs. Visibility only; it gates nothing, and it fails loudly rather than publishing a report built on an empty API fetch.
  • Per-package CycloneDX SBOMs (#274): the release workflow generates and attaches one SBOM per package with a pinned tool version.
  • Beta-exit criteria and public API stability policy (#255, #282): documented Tier 1/2/3 API surface and the bar for a 1.0.
  • 10,000-control resolver benchmark (#265, #290) and a custom ILlmHealingProvider implementation guide (#263, #284).

Fixes

  • Batch reconciliation: mixed heuristic / LLM contention on the same candidate now resolves as ambiguous (declining every claimant) instead of arbitrarily by score scale (#268).
  • LocatorRepository freshness is keyed on a content hash, not just mtime/size, so an in-place edit of the same length is picked up (#273). Harness contention detection is keyed on path-qualified element identity, not the AutomationId string, closing the empty-id and duplicate-id blind spots (#272, #292).
  • LLM response parsing no longer discards a valid consensus vote over a malformed optional field: a string-valued confidence, a numeric candidateId (#304), or a non-string reasoning (#321) are all parsed defensively.
  • Resource leaks on error paths closed: ApplicationConnector.Launch no longer orphans the launched process if UIA initialisation throws (#305, and Attach is left to dispose only, never kill); PlaywrightLiveExplorer.DisposeAsync runs _playwright.Dispose() in a finally so a browser-close fault cannot leak the driver (#306).
  • Generated FlaUI tests compile: the file-upload path emits fully-qualified FlaUI.Core.Definitions.ControlType (#303).
  • Web mapping: <form> maps to Group to match its ARIA role (#308); a symbol-only ARIA role no longer throws in PlaywrightLocatorEmitter (#309); ARIA role normalisation is aligned with the UIA ControlType taxonomy (#267).
  • Intent scoring: a 2-character element label no longer earns a spurious exact-match bonus from a bare substring hit (#307).
  • Healing report HTML is written atomically (temp file then rename), matching the JSON path (#310).
  • Codegen: regex strings are escaped for C# and TypeScript UrlContains output (#266).
  • Windows net48 coverage is restored (coverlet 8.0+ ships no .NET Framework collector) and guarded so the Discovery assembly cannot silently drop out again (#289).
  • Dead code removed from TreeCalibrator (orphaned reference-identity mutation helpers, an unused probe kind) and PlaywrightApplicationConnector (#311).
  • Documentation realigned with the code: HealingHistory schema fields, M6 implementation status, versioning tiers, the AGENTS.md project/workflow inventory, stale model defaults and self-contradicting benchmark figures, and English/Turkish parity (#300, #312, #313, #264).
  • Dependency upgrades across the libraries and the test runner (#294).

Known Limitations

  • Calibration: higher confidence thresholds reduce false heals, but the magnitude does not transfer between applications. Calibrate against your own UI history; the shipped MinimumConfidence is 0.50.
  • Accuracy: the opt-in batch ownership guard eliminates observed shared-candidate collisions while preserving correct survivor heals, but it is a collision guard, not an absence detector - uncontested deleted-element false heals are unchanged.
  • LLM providers: covered by mocked-HTTP unit tests; the nightly workflow gates one known ground-truth agreement scenario against independent providers. Release creation does not run a live gate. llm-smoke.yml is a manual single-endpoint diagnostic after GitHub Models' retirement (#44).
  • Platform: desktop automation is Windows-only (FlaUI / UIA3). The heuristic engine and the web path are cross-platform and verified on Linux in CI. Linux desktop discovery via AT-SPI is tracked in #17.

Packages

  • AutomationSandbox.UiModel
  • AutomationSandbox.SelfHealing
  • AutomationSandbox.LlmHealing
  • AutomationSandbox.Discovery
  • AutomationSandbox.WebDiscovery
  • AutomationSandbox.IntentAutomation
  • AutomationSandbox.PlaywrightLiveExploration

This release publishes package files as GitHub Release assets. They are also pushed to nuget.org via Trusted Publishing.

Assets 23
Loading