fix(performance): skip on proxy errors and enforce PM token guard by ian-flores · Pull Request #209 · posit-dev/vip

ian-flores · 2026-04-21T22:02:50Z

Summary

Two performance-test issues — one a real bug (opaque connection failures behind a proxy), one a symmetry fix that makes Package Manager load tests behave consistently.

fix(performance): login page load time test fails with connection refused behind proxy #177 — test_login_load_times.py only caught httpx.ConnectError and hard-failed with "connection refused" even when the real problem was a proxy/firewall/DNS/closed-port issue between the test runner and the product. Now catches httpx.ProxyError and httpx.ConnectTimeout too, and converts all three into pytest.skip with a clear network-path diagnostic.
fix(performance): duplicate load tests for Package Manager (test_load vs test_user_simulation) #174 — test_load.py skips Package Manager tests when the PM token is missing; test_user_simulation.py silently coerced the missing token to "" and ran unauthenticated, producing 401s that masqueraded as load-capacity results. Added the same token guard to test_user_simulation.simulate_pm. The two tests measure different things (burst concurrency vs. session-simulation traffic) and are complementary — they now skip symmetrically when the token is missing.

Test plan

Ruff: just check passes
Selftests: uv run pytest selftests/ passes (311/311; includes 8 new tests added during review cycles for the skip paths)
Product tests:
- test_login_load_times — with an unreachable URL (https://this-does-not-resolve-abc-xyz-9999.internal.invalid) the test skips with my new diagnostic: "Connect login URL not reachable from test runner (...): [Errno 8] nodename nor servname provided, or not known. Check network path, proxy configuration, DNS resolution, and that the port is open from the runner."
- test_user_simulation — PM step skipped with "Skipped: Package Manager token is not configured" when package_manager.token is unset (matching the pre-existing guard in test_load.py)

Fixes #177
Fixes #174

- test_login_load_times: catch httpx.ProxyError and httpx.ConnectTimeout alongside httpx.ConnectError, and convert them into pytest.skip with a clear network-path diagnostic instead of a misleading fail. - test_user_simulation: add the same Package Manager token guard as test_load.py so both tests skip symmetrically when the token is missing, rather than one running unauthenticated and returning 401s that masquerade as load-capacity results. Fixes #177 Fixes #174

Copilot

Pull request overview

Updates VIP performance tests to behave more robustly in common network/proxy topologies and to avoid running Package Manager load tests without authentication, aligning behavior across performance suites.

Changes:

Convert login page load-time connectivity failures into clear pytest.skip results by catching additional httpx proxy/timeout exceptions.
Add a Package Manager token guard to the Locust-based session simulation so it skips when the token is missing (matching test_load.py behavior).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
`src/vip_tests/performance/test_login_load_times.py`	Improves resiliency/diagnostics by skipping on proxy/connectivity errors instead of hard-failing.
`src/vip_tests/performance/test_user_simulation.py`	Enforces a missing-PM-token skip to prevent unauthenticated runs producing misleading 401-based “load” results.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-04-21T22:08:51Z

Preview Links

Website	Validation Report

https://posit-dev.github.io/vip/pr-preview-site/pr-209/	https://posit-dev.github.io/vip/pr-preview/pr-209/

- Expand comment on ConnectError/ProxyError/ConnectTimeout tuple explaining why httpx.TransportError is intentionally not used (mid-transfer errors should still fail, not skip) and that prerequisites already covers reachability - Update simulate_pm comment to cite exact line in test_load.py for the symmetric token guard, removing ambiguity about whether guards match - Add selftests/test_performance_steps.py covering both new skip paths: all three pre-connect exception types skip with correct messages, and simulate_pm skips on empty token with informative message

- Replace cryptic generator-throw lambda with a named _raise function in _run_measure for readability - Drop redundant cfg.connect.enabled = True (enabled defaults to True; non-empty url is sufficient for is_configured) - Remove unused monkeypatch parameter from TestSimulatePmTokenGuard helpers and test methods (simulate_pm short-circuits before any I/O) - Drop stale line-number cross-reference from simulate_pm comment; reference the guard condition by symbol instead

- Derive _VALID_USERS from PerformanceConfig().load_user_counts[0] so token-guard tests stay decoupled from hardcoded default values - Tighten test_no_url_skips_before_token_check to assert "url" in the skip message rather than just any non-empty string - Drop request=None kwarg from httpx.ConnectTimeout construction (undocumented corner; consistent with ProxyError/ConnectError which omit it) - Replace cross-file comment in simulate_pm with self-contained rationale; cross-file line references rot when the referenced file changes