Skip to content

v0.2.0-beta.2

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 14 Aug 21:07
v0.2.0-beta.2
177759a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Automation Sandbox v0.2.0-beta.2

Preview NuGet package artifact release.

Breaking Changes

Both changes are in the LLM fallback path. Consumers who never configure an LLM provider are unaffected.

  • A single LLM provider's pick is no longer accepted (#10). Acceptance is now by consensus: at least SimilarityWeights.MinimumConsensusVotes (default 2) providers must independently name the same candidate. Self-reported confidence is still recorded but is never compared or thresholded, because those numbers are not calibrated across model architectures. This fails quietly by design - with one provider configured, healing degrades to the heuristic result instead of raising an error. If you relied on single-provider LLM healing, configure a second provider from a different model family, or expect heuristic-only behaviour. MinimumLlmConfidence remains on SimilarityWeights and is still reported, but no longer gates anything.
  • Worst-case provider latency rose from ~15s to ~35s (#11). Providers now retry transient failures, so a slow or failing endpoint occupies its per-attempt timeout more than once. The new TotalTimeout (35s cloud, 70s Ollama) caps the whole operation including backoff. Set maxRetries: 0 to restore single-shot behaviour.

Highlights

  • Consensus acceptance for LLM picks (#10): Providers are polled in parallel and a pick is accepted only on agreement. Invalid candidate ids are discarded before the vote is counted, so one hallucinating provider cannot invalidate everyone else's valid votes. A three-way split and a tie for the lead both count as disagreement and fall back to the heuristic result. HealResult.AgreedProviders records who agreed.
  • Provider resilience (#11): Retry with exponential backoff and jitter on transient failures (429, 500, 502, 503, 504, HttpRequestException); immediate fail-fast on 400/401/403/404. Retry-After is honoured up to a 10s ceiling - a longer value is read as quota exhaustion and fails fast rather than stalling a test run. Dual timeout budgets: per attempt and per operation.
  • Per-provider attempt telemetry (#11): LlmHealingResult.AttemptCount and HealResult.ProviderAttempts record how many attempts each provider needed, so a flaky provider is identifiable rather than merely producing a failed heal.
  • Healing report schema v6: adds AgreedProviders (v5) and ProviderAttempts (v6). Older reports upgrade in place on the next write; the new fields stay null on upgraded entries, which means "not recorded by that build" rather than "there were none".
  • Configurable provider names (#10): every provider takes a name: parameter. OpenAiHealingProvider accepts any OpenAI-compatible endpoint, so several instances (for example Groq and Cerebras) can vote alongside each other and remain distinguishable in the report.
  • Shared HTTP provider base (#48): the four providers now derive from HttpLlmHealingProvider, which owns orchestration, validation and retry wiring. Behaviour-neutral; public constructors are unchanged.

Known Limitations

  • Calibration: The thresholds shipped in this release are initial estimates, not values derived from a real-world benchmark dataset: MinimumEvidenceWeight (0.40), MinimumCandidateMargin (0.05), MinimumSemanticScore (0.01), MinimumConsensusVotes (2), the 10s Retry-After ceiling, and default AssertGenerationMode (Strict). Each is documented with its rationale and is subject to recalibration under #15.
  • Accuracy: Healing accuracy is 100% on the bundled WinForms and WPF demo case studies. The false-positive rate on organic, third-party applications has not been measured (#15).
  • LLM Providers: Provider integrations are covered by mocked-HTTP unit tests, and the Windows CI leg additionally makes a real Gemini call on every run when GEMINI_API_KEY is configured (a non-gating comparison harness). What is not covered is a gating live assertion across providers, so consensus itself has never been exercised against real models; llm-smoke.yml runs one OpenAI-compatible endpoint on manual dispatch only, after GitHub Models began returning HTTP 410 ahead of its retirement (#21, #44). Nightly multi-provider collection is tracked in #47.
  • Platform: Desktop automation is Windows-only (FlaUI / UIA3). The core heuristic engine and web automation path are cross-platform and verified on Linux in CI (#13).

Packages

  • AutomationSandbox.UiModel
  • AutomationSandbox.SelfHealing
  • AutomationSandbox.LlmHealing
  • AutomationSandbox.Discovery
  • AutomationSandbox.WebDiscovery
  • AutomationSandbox.IntentAutomation
  • AutomationSandbox.PlaywrightLiveExploration

This release publishes package files as GitHub Release assets only. It does not push to nuget.org or GitHub Packages.

Assets 16
Loading