Skip to content

test(tests): add provider hardening end-to-end coverage - #39

Merged
patlux merged 3 commits into
mainfrom
test/provider-hardening-e2e
Aug 7, 2026
Merged

test(tests): add provider hardening end-to-end coverage#39
patlux merged 3 commits into
mainfrom
test/provider-hardening-e2e

Conversation

@patlux

@patlux patlux commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • extend real-Pi/mock-upstream coverage for discovery timeout, reasoning payloads, runtime commands, nested tool schemas, image rejection, and overflow compaction/retry
  • add opt-in live Command Code E2E using existing Pi authentication
  • validate a live tool round trip and an authenticated prompt from the packed npm artifact
  • run the complete test suite in CI and document the new behavior

Dependency

Stacked on #38 (which is stacked on #37). Merge in order: #37#38 → this PR.

Verification

  • npm test
  • npm run test:e2e:live
  • npm run format:check
  • npm pack --dry-run
  • git diff --check

Notes

  • The live E2E consumes provider capacity and therefore remains an explicit local command rather than a CI step.
  • OMP process coverage still skips because omp is unavailable on the test host.

@patlux patlux self-assigned this Aug 7, 2026
Base automatically changed from feat/request-fidelity to main August 7, 2026 13:18
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@patlux
patlux merged commit 7d02d34 into main Aug 7, 2026
11 checks passed
@patlux
patlux deleted the test/provider-hardening-e2e branch August 7, 2026 13:20
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Memory benchmark

Compared base db49f5a with PR head ff1e23b on the same GitHub-hosted darwin runner. Lower values are better.

Metric Base PR PR − Base Change
Stable RSS 167.1 MiB ± 1.7 MiB 167.1 MiB ± 1.0 MiB -0.4 MiB ± 1.6 MiB -0.2%
Physical footprint 110.2 MiB ± 0.4 MiB 110.8 MiB ± 0.6 MiB -0.1 MiB ± 0.5 MiB -0.1%
Physical peak 126.2 MiB ± 1.2 MiB 127.4 MiB ± 0.9 MiB -0.4 MiB ± 1.2 MiB -0.3%

Interpretation: No metric shows a clear Base-to-PR difference beyond its measured run-to-run variation.

Extension overhead above pi baseline

Metric Base overhead PR overhead Difference
Stable RSS +4.8 MiB ± 2.4 MiB +6.0 MiB ± 0.5 MiB -0.4 MiB ± 1.6 MiB
Physical footprint +3.2 MiB ± 2.1 MiB +4.0 MiB ± 0.9 MiB -0.1 MiB ± 0.5 MiB
Physical peak +2.3 MiB ± 2.5 MiB +3.9 MiB ± 1.0 MiB -0.4 MiB ± 1.2 MiB

Values are medians of 6 alternating, paired runs. The value after ± is the median absolute deviation (MAD). Each process was sampled 12 times after a 2500 ms warm-up.

Environment: pi 0.82.1, Bun 1.3.11, darwin arm64. RSS comes from ps; physical footprint and peak come from macOS footprint.

This is a comparative signal, not a pass/fail threshold. GitHub-hosted runner noise can affect absolute values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant