Skip to content

nexus: add PymtHouse capability integration follow-ups#307

Draft
eliteprox wants to merge 51 commits into
mainfrom
feat/capability-explorer-latest
Draft

nexus: add PymtHouse capability integration follow-ups#307
eliteprox wants to merge 51 commits into
mainfrom
feat/capability-explorer-latest

Conversation

@eliteprox
Copy link
Copy Markdown
Contributor

@eliteprox eliteprox commented May 12, 2026

Summary

  • PymtHouse OIDC device-flow login with explicit user consent (user_code confirmation before approve)
  • Manifest-based capability denylisting for pymthouse billing provider
  • Python-gateway discovery routes (default + plan-scoped) with tiered shuffle; default to pymthouse when configured
  • Dev API key minting, expiry, and billing OAuth hardening

Out of scope (follow-up PRs)

  • Usage & Billing dashboard UI (GET /api/v1/billing/pymthouse/usage + DeveloperView metrics)
  • Leaderboard plan-create / capability picker plugin UI
  • Discovery-service Go client integration (DISCOVERY_SERVICE_URL)
  • Plugin UMD stylesheet lifecycle

Test plan

  • pymthouse-device-initiate, provider-restrictions, python-gateway route unit tests
  • Manual device flow: CLI device code → login → confirm user_code on consent screen → token
  • CI / Vercel preview for stacked diff

Summary by CodeRabbit

  • New Features

    • Added PymtHouse billing provider with OIDC device-flow authentication support
    • Developer API keys now display expiration information
    • Enhanced discovery plan creation with capability filtering and provider selection
    • New Usage & Billing dashboard showing consumption metrics and pipeline-model breakdown
    • Improved orchestrator discovery with tiered capability allowlists
  • Improvements

    • Enhanced plugin stylesheet lifecycle management
    • Refined discovery plan visibility and scoping by billing provider

Review Change Stack

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented May 12, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
naap-platform Ready Ready Preview, Comment May 23, 2026 5:20am

Request Review

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 12, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds PymtHouse billing integration: env/config, device-flow middleware/UI, token and usage APIs, discovery provider scoping with manifest sync and caching, developer key expiry, discovery service client, UI/Docs/tests updates, and plugin CSS lifecycle.

Changes

End-to-end PymtHouse integration and discovery updates

Layer / File(s) Summary
Complete change set
...
All diffs implementing PymtHouse env/client, device flow, billing token/usage routes, provider-scoped discovery and caching, key expiry, UI/Docs/tests, and UMD plugin stylesheet lifecycle.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

  • livepeer/naap#124: Touches the same billing OAuth provider start/callback/result surfaces now extended for PymtHouse.
  • livepeer/naap#213: Modifies Orchestrator Leaderboard backend/ClickHouse paths also refactored here.
  • livepeer/naap#150: Updates developer key routes overlapping with this PR’s provider-specific key expiry/shape.

Suggested reviewers

  • seanhanca
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/capability-explorer-latest

@github-actions github-actions Bot added the has-migration Includes database migration label May 12, 2026
@github-actions
Copy link
Copy Markdown

🗃️ Database Migration Detected

This PR includes changes to Prisma schema files. Please ensure:

  • Migration is backward-compatible or a rollback plan exists
  • Data migration scripts are included if needed
  • Schema changes will be auto-applied to the preview database (Neon preview branch) during the Vercel preview deployment
  • Verify the preview deployment works correctly with the new schema
  • On merge to main, schema changes will auto-promote to production via prisma db push

Preview DB: This PR's Vercel preview deployment uses an isolated Neon database branch. Schema changes are applied automatically via prisma db push during the preview build. The preview branch is reset after each production deploy.

Requesting review from the core team: @livepeer/core

@github-actions github-actions Bot added size/XL Extra large PR (500+ lines) scope/shell Shell app changes scope/packages Shared package changes scope/infra Infrastructure changes plugin/developer-api Developer API plugin labels May 12, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 12, 2026

⚠️ This PR is very large (10695 lines changed). Please split it into smaller, focused PRs if possible.

@eliteprox eliteprox force-pushed the feat/capability-explorer-latest branch from fc19916 to 3a03531 Compare May 14, 2026 18:57
@github-actions github-actions Bot removed the scope/infra Infrastructure changes label May 14, 2026
@eliteprox eliteprox force-pushed the feat/capability-explorer-latest branch from 3a03531 to 47b9fd4 Compare May 14, 2026 19:10
@eliteprox eliteprox changed the base branch from feat/orchestrator-leaderboard to feat/orchestrator-leaderboard-followups May 14, 2026 19:10
@eliteprox eliteprox force-pushed the feat/capability-explorer-latest branch from c75c271 to 8bbc282 Compare May 20, 2026 21:01
@github-actions github-actions Bot added the scope/infra Infrastructure changes label May 20, 2026
@eliteprox eliteprox changed the base branch from feat/orchestrator-leaderboard-followups to main May 20, 2026 21:01
seanhanca and others added 8 commits May 20, 2026 17:03
…support

Wire `upstreamStaticBody` through the connector template pipeline (loader
interface, JSON schema, admin template route, seed script) so endpoints
can send a pre-configured request body to the upstream service.

Add the `clickhouse-query` connector template with four endpoints:
  - /network_prices  — static SQL for orchestrator pricing data
  - /query           — dynamic SELECT-only queries with regex + blacklist
  - /ping            — ClickHouse health check
  - /tables          — list tables via SHOW TABLES

Include a how-to guide with dashboard visualization example and update
the connector catalog.

Made-with: Cursor
- Use ?? null instead of || null for upstreamStaticBody to preserve
  empty strings and match the seed script's nullish coalescing
- Remove hardcoded allowedHosts from clickhouse-query.json so host
  validation derives from upstreamBaseUrl at runtime, avoiding
  conflicts when users override the base URL
- Remove envKey from clickhouse-query.json — a single env var cannot
  map to two separate Basic-auth secrets (username + password);
  users configure credentials via the Settings tab UI instead
- Add JSDoc docstrings to all exported interfaces and functions in
  the connector template loader and route handlers

Made-with: Cursor
…E tests

- Remove NEXT_PUBLIC_APP_URL/NEXTAUTH_URL from committed .env (use .env.local)
- Patch start.sh to sync NEXT_PUBLIC_APP_URL with SHELL_PORT on every start
- Remove hardcoded localhost:3000 from CORS allowlist in next.config.js
- Consolidate 7 inline localhost fallbacks to use shared appUrl from lib/env
- Fix layout.tsx stale localhost:3001 fallback
- Use request.url origin for ClickHouse gateway URL resolution
- Add auth headers + credentials to plugin frontend API calls
- Make Playwright webServer.url respect PLAYWRIGHT_BASE_URL
- Add orchestrator-leaderboard E2E spec (stub + live modes)
- Add resolveClickhouseGatewayQueryUrl unit tests

Made-with: Cursor
…d webhook guide

- Add TabNav component for Leaderboard / Plans tab switching
- Add PlansOverviewPage with summary stats, plan cards, endpoint info
- Add PlanDetailPage with live-editing config (SLA weights, min score,
  topN, sort, filters) and "Apply Changes" to see results update
- Add EndpointGuide component showing API endpoint URL, copy button,
  and 3-step webhook setup guide for signer ORCHESTRATOR_DISCOVERY_URL
- Add usePlans/usePlanDetail hooks for data fetching and state management
- Add frontend API functions: fetchPlans, fetchPlanResults, updatePlan, seedDemoPlans
- Add backend POST /plans/seed route for dev-only demo data (4 plans)

Made-with: Cursor
Migrate orchestrator-leaderboard plugin from hardcoded Tailwind gray/blue
classes to NaaP's CSS-variable-based design tokens. Rewrites tailwind.config.js
with semantic color mappings, globals.css with :root/.dark CSS variables and
.glass-card, and all 7 TSX component files with token-backed utility classes.

Also fixes a bug where updating a plan's configuration (e.g. SLA min score)
did not refresh the orchestrator results — the PUT handler now invalidates the
in-memory planCache entry so the next GET /results forces a fresh evaluation.

Made-with: Cursor
… scoping

- Fix auth redirect loop: always clear httpOnly cookie via logout endpoint
  before redirecting to /login, preventing middleware loop when DB is
  unreachable or session is invalid
- Fix plan visibility: use OR logic in scopeWhere so plans match by either
  teamId or ownerUserId (build-time seed sets ownerUserId only)
- Fix seed button hidden on Vercel: remove isLocalhost gate so all
  authenticated users can seed demo data
- Fix seed API blocked on Vercel: remove NODE_ENV=production check that
  blocked preview deployments
- Fix per-user seeding: use user-scoped billingPlanIds (demo-{userId}-slug)
  so each user gets their own isolated set of demo plans without unique
  constraint collisions
- Set teamId in build-time seed for proper scope matching

Made-with: Cursor
…ct resolver and audit log

Replace the hard-coded ClickHouse-only refresh pipeline with a pluggable
multi-source architecture:

- 4 source adapters: livepeer-subgraph (on-chain registry), clickhouse-query,
  naap-discover, naap-pricing — each behind the SourceAdapter interface
- 3 new Service Gateway connector templates for the new upstream APIs
- Hybrid conflict resolver: source-level membership + field-level metric priority
  with ethAddress↔orchUri cross-reference joining
- Durable audit log (LeaderboardRefreshAudit) recording per-source stats,
  conflicts, dropped rows, and warnings on every refresh
- Admin-configurable source priority/toggle via LeaderboardSource Prisma model
- GET/PUT /sources and GET /audits REST endpoints with admin authz
- Admin UI: tabbed settings with drag-to-reorder Data Sources panel and
  expandable Refresh Audit log
- Comprehensive docs: data-sources.md, updated openapi.yaml, api-reference.md,
  how-to-guide.md, for-ai.md
- 20 new unit tests (resolver + adapters) + E2E test spec
- All 119 existing tests continue to pass

Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced a new script command `seed:discovery-plans` in package.json to facilitate the seeding of discovery plans, enhancing the setup process for development and testing environments.
…st handling

- Updated the orchestrator leaderboard API to support a new `manifestOnly` query parameter, allowing for streamlined responses when only the manifest is requested.
- Replaced the deprecated `syncPymthouseManifestSnapshot` function with `ensurePymthouseManifestFresh` to improve manifest freshness checks.
- Enhanced caching strategies by setting appropriate `Cache-Control` headers for responses, ensuring better performance and data consistency.
- Refactored the `getDashboardPipelineCatalog` function to accept an optional `bypassCache` parameter, allowing for more flexible data retrieval.
- Improved capability filtering logic to respect provider restrictions, ensuring only allowed capabilities are returned based on the billing provider.
- Updated documentation to reflect changes in API behavior and caching mechanisms.
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts (1)

188-199: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

getCachedPlanResults returns the first matching entry, which may not be the freshest.

When multiple composite cache entries exist for the same planId (e.g., after capability changes), this returns the first valid one found during iteration. Map iteration order is insertion order, so this could return a stale entry if a newer one was inserted later. Consider returning the entry with the most recent cachedAt timestamp.

Proposed fix
 export function getCachedPlanResults(planId: string): PlanResults | null {
   const prefix = `${planId}${PLAN_CACHE_KEY_SEP}`;
+  let best: PlanCacheEntry | null = null;
   for (const [key, entry] of planCache) {
     if (!key.startsWith(prefix)) continue;
     if (!isValid(entry)) continue;
-    return {
-      ...entry.results,
-      meta: { ...entry.results.meta, cacheAgeMs: Date.now() - entry.cachedAt },
-    };
+    if (!best || entry.cachedAt > best.cachedAt) {
+      best = entry;
+    }
   }
-  return null;
+  if (!best) return null;
+  return {
+    ...best.results,
+    meta: { ...best.results.meta, cacheAgeMs: Date.now() - best.cachedAt },
+  };
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts` around lines 188 -
199, getCachedPlanResults currently iterates planCache and returns the first
valid entry for a given planId which can be stale due to insertion order; update
getCachedPlanResults to scan all entries whose keys start with
`${planId}${PLAN_CACHE_KEY_SEP}`, filter by isValid(entry), and select the entry
with the largest entry.cachedAt (newest) before returning its results (still
adding cacheAgeMs as Date.now() - cachedAt); use the existing symbols planCache,
PLAN_CACHE_KEY_SEP, isValid, cachedAt, and PlanResults to locate and implement
this change.
plugins/orchestrator-leaderboard/frontend/src/pages/PlanCreatePage.tsx (1)

164-172: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Clamp topN before state update.

Current parsing can still pass invalid values (e.g., negative or >1000) into request payloads. Clamp in code, not only via input attributes.

Suggested fix
             <input
               type="number"
               min={1}
               max={1000}
               value={topN}
-              onChange={(e) => setTopN(Number(e.target.value) || 10)}
+              onChange={(e) => {
+                const n = Number(e.target.value);
+                if (!Number.isFinite(n)) {
+                  setTopN(10);
+                  return;
+                }
+                setTopN(Math.min(1000, Math.max(1, Math.trunc(n))));
+              }}
               className="w-full px-3 py-2 bg-bg-secondary border border-[var(--border-color)] rounded-lg text-text-primary text-sm"
             />
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/orchestrator-leaderboard/frontend/src/pages/PlanCreatePage.tsx`
around lines 164 - 172, The Top N input currently uses onChange={(e) =>
setTopN(Number(e.target.value) || 10)} which can still allow values outside
[1,1000] and mishandle 0; update the onChange handler in PlanCreatePage (the
code that reads topN and calls setTopN) to parse the value, handle NaN by
falling back to a sane default, and clamp it with Math.max(1, Math.min(1000,
parsedValue)) before calling setTopN; also ensure any place that reads topN for
request payloads uses the clamped value (e.g., before submit) so only valid
values are sent.
🧹 Nitpick comments (4)
apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts (2)

220-228: 💤 Low value

startLocalRefreshLoop ignores passed authToken.

The function accepts authToken as a parameter but never forwards it to refreshAllPlans(). Either pass the token or remove the unused parameter.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts` around lines 220 -
228, startLocalRefreshLoop currently accepts authToken but never uses it; update
the setInterval callback to forward the token to refreshAllPlans (e.g.
refreshAllPlans(authToken).catch(...)) so the auth context is preserved, or if
the token is not needed remove the authToken parameter from
startLocalRefreshLoop; locate the function startLocalRefreshLoop and the call to
refreshAllPlans to apply the change.

105-109: 💤 Low value

Unused parameters in evaluateAndCache.

The parameters authToken, requestUrl, and cookieHeader are declared but never used in the function body. If these are placeholders for future functionality, consider documenting that intent or removing them to avoid confusion.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts` around lines 105 -
109, The function evaluateAndCache currently declares unused parameters
authToken, requestUrl, and cookieHeader; either remove these parameters from the
evaluateAndCache signature (and update any call sites) to eliminate dead
parameters, or if they are intentionally reserved for future use, rename them to
_authToken, _requestUrl, and _cookieHeader (or add a short JSDoc/TODO noting
they are intentionally unused) so linters and readers know they are
placeholders; locate the symbol evaluateAndCache to apply the change and ensure
consistency across callers.
bin/db-setup.sh (1)

86-88: 💤 Low value

A && B || C pattern may run C if B fails (SC2015).

While unlikely since log_success is just an echo, the static analysis warning is technically valid. The warning also applies to line 83. For robustness, consider using explicit if-then-else, though this is low risk given the simple logging functions.

Safer alternative
-npx tsx bin/seed-discovery-plans.ts 2>/dev/null && log_success "Default discovery plans seeded" || log_warn "Discovery plan seed had issues (non-critical)"
+if npx tsx bin/seed-discovery-plans.ts 2>/dev/null; then
+  log_success "Default discovery plans seeded"
+else
+  log_warn "Discovery plan seed had issues (non-critical)"
+fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@bin/db-setup.sh` around lines 86 - 88, The current use of the A && B || C
pattern (running npx tsx ... && log_success ... || log_warn ...) can call the
fallback even if the success logging command fails (shellcheck SC2015); update
the seed commands (referencing log_info, log_success, log_warn and the npx tsx
bin/seed-discovery-plans.ts invocation executed from ROOT_DIR) to use an
explicit if-then-else: run the npx command and check its exit status with if
...; then call log_success; else call log_warn; fi (apply the same change to the
similar pattern earlier on line with the other seed command too).
apps/web-next/src/lib/pymthouse-manifest.ts (1)

56-61: 💤 Low value

Environment variable fallbacks include typo variants.

The code falls back from PYMTHOUSE_* to PMTHOUSE_* (missing 'Y'). If this is intentional for backwards compatibility, consider adding a deprecation warning when the typo variant is used. Otherwise, this may silently accept misconfigured environments.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/web-next/src/lib/pymthouse-manifest.ts` around lines 56 - 61, The
fallback uses misspelled env names (PMTHOUSE_*) for publicId, m2mId and
m2mSecret; either remove the typo fallbacks to require the correct PYMTHOUSE_*
vars, or keep them but detect when the PMTHOUSE_* variant is used and emit a
deprecation warning (e.g., console.warn or existing logger) indicating the
correct PYMTHOUSE_* variable to use; update the logic around publicId, m2mId and
m2mSecret to check which source was used and log the deprecation when PMTHOUSE_*
is present.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@apps/web-next/src/app/api/v1/orchestrator-leaderboard/python-gateway/route.ts`:
- Around line 87-89: The call to ensurePymthouseManifestFresh() is placed
outside the existing try/catch so any errors bypass your route's error boundary;
move the conditional block that calls ensurePymthouseManifestFresh() (the if
(billingProvider === 'pymthouse') { await ensurePymthouseManifestFresh(); }
logic) inside the same try that wraps the route processing so failures are
caught by the route's catch and correct status mapping is applied.

In `@plugins/orchestrator-leaderboard/frontend/src/components/StyledCheckbox.tsx`:
- Around line 21-33: The visible checkbox isn’t clickable because the native
<input> is sr-only and not associated with the visible span; wrap the input and
the visible span in a clickable label (or make the input absolutely fill the
container and be invisible but not sr-only) inside the StyledCheckbox component
so clicks on the rendered box toggle the input; keep the input’s checked,
disabled, ariaLabel, onChange handler and the peer class intact (update the
input’s classes to either remove sr-only and use "absolute inset-0 w-full h-full
opacity-0 cursor-pointer" or wrap both elements in a <label> so the span remains
a visual-only element while clicks target the input).

In `@plugins/orchestrator-leaderboard/frontend/src/globals.css`:
- Around line 108-113: The Stylelint rule "declaration-empty-line-before" is
being violated around the .section-label-text and nearby selector blocks; update
the CSS so each selector's first declaration conforms to the project's rule by
either inserting a single blank line before the declaration block or removing an
unexpected blank line as required by the configured rule; specifically adjust
the .section-label-text rule and the other affected selector near line 148 so
the spacing before their first property matches the configured
declaration-empty-line-before behavior (ensure consistent blank-line
presence/absence across those selector blocks).

In `@plugins/orchestrator-leaderboard/frontend/src/pages/LeaderboardPage.tsx`:
- Around line 70-80: The auto-refresh toggle button does not expose its pressed
state to assistive tech; update the button (the element that calls
setAutoRefresh and reads autoRefresh, near Radio) to include
aria-pressed={autoRefresh} and add a stateful aria-label (e.g., "Auto-refresh
on" / "Auto-refresh off" derived from autoRefresh) so screen readers receive the
toggle semantics while preserving the existing onClick behavior and visual
classes.

---

Outside diff comments:
In `@apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts`:
- Around line 188-199: getCachedPlanResults currently iterates planCache and
returns the first valid entry for a given planId which can be stale due to
insertion order; update getCachedPlanResults to scan all entries whose keys
start with `${planId}${PLAN_CACHE_KEY_SEP}`, filter by isValid(entry), and
select the entry with the largest entry.cachedAt (newest) before returning its
results (still adding cacheAgeMs as Date.now() - cachedAt); use the existing
symbols planCache, PLAN_CACHE_KEY_SEP, isValid, cachedAt, and PlanResults to
locate and implement this change.

In `@plugins/orchestrator-leaderboard/frontend/src/pages/PlanCreatePage.tsx`:
- Around line 164-172: The Top N input currently uses onChange={(e) =>
setTopN(Number(e.target.value) || 10)} which can still allow values outside
[1,1000] and mishandle 0; update the onChange handler in PlanCreatePage (the
code that reads topN and calls setTopN) to parse the value, handle NaN by
falling back to a sane default, and clamp it with Math.max(1, Math.min(1000,
parsedValue)) before calling setTopN; also ensure any place that reads topN for
request payloads uses the clamped value (e.g., before submit) so only valid
values are sent.

---

Nitpick comments:
In `@apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts`:
- Around line 220-228: startLocalRefreshLoop currently accepts authToken but
never uses it; update the setInterval callback to forward the token to
refreshAllPlans (e.g. refreshAllPlans(authToken).catch(...)) so the auth context
is preserved, or if the token is not needed remove the authToken parameter from
startLocalRefreshLoop; locate the function startLocalRefreshLoop and the call to
refreshAllPlans to apply the change.
- Around line 105-109: The function evaluateAndCache currently declares unused
parameters authToken, requestUrl, and cookieHeader; either remove these
parameters from the evaluateAndCache signature (and update any call sites) to
eliminate dead parameters, or if they are intentionally reserved for future use,
rename them to _authToken, _requestUrl, and _cookieHeader (or add a short
JSDoc/TODO noting they are intentionally unused) so linters and readers know
they are placeholders; locate the symbol evaluateAndCache to apply the change
and ensure consistency across callers.

In `@apps/web-next/src/lib/pymthouse-manifest.ts`:
- Around line 56-61: The fallback uses misspelled env names (PMTHOUSE_*) for
publicId, m2mId and m2mSecret; either remove the typo fallbacks to require the
correct PYMTHOUSE_* vars, or keep them but detect when the PMTHOUSE_* variant is
used and emit a deprecation warning (e.g., console.warn or existing logger)
indicating the correct PYMTHOUSE_* variable to use; update the logic around
publicId, m2mId and m2mSecret to check which source was used and log the
deprecation when PMTHOUSE_* is present.

In `@bin/db-setup.sh`:
- Around line 86-88: The current use of the A && B || C pattern (running npx tsx
... && log_success ... || log_warn ...) can call the fallback even if the
success logging command fails (shellcheck SC2015); update the seed commands
(referencing log_info, log_success, log_warn and the npx tsx
bin/seed-discovery-plans.ts invocation executed from ROOT_DIR) to use an
explicit if-then-else: run the npx command and check its exit status with if
...; then call log_success; else call log_warn; fi (apply the same change to the
similar pattern earlier on line with the other seed command too).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b2c520cd-f0b0-4c32-8917-7749c8bb6b89

📥 Commits

Reviewing files that changed from the base of the PR and between 9422c0a and b5d5bff.

📒 Files selected for processing (44)
  • apps/web-next/src/app/api/v1/orchestrator-leaderboard/capability-catalog/route.ts
  • apps/web-next/src/app/api/v1/orchestrator-leaderboard/plans/[id]/python-gateway/route.ts
  • apps/web-next/src/app/api/v1/orchestrator-leaderboard/plans/[id]/results/route.ts
  • apps/web-next/src/app/api/v1/orchestrator-leaderboard/plans/seed/route.ts
  • apps/web-next/src/app/api/v1/orchestrator-leaderboard/python-gateway/route.ts
  • apps/web-next/src/components/layout/app-layout.tsx
  • apps/web-next/src/components/layout/top-bar.tsx
  • apps/web-next/src/lib/discovery-service/client.ts
  • apps/web-next/src/lib/facade/index.ts
  • apps/web-next/src/lib/facade/resolvers/pipeline-catalog.ts
  • apps/web-next/src/lib/orchestrator-leaderboard/__tests__/provider-restrictions.test.ts
  • apps/web-next/src/lib/orchestrator-leaderboard/__tests__/types-validation.test.ts
  • apps/web-next/src/lib/orchestrator-leaderboard/refresh.ts
  • apps/web-next/src/lib/orchestrator-leaderboard/types.ts
  • apps/web-next/src/lib/orchestrators-discovery-policy.ts
  • apps/web-next/src/lib/pymthouse-manifest.test.ts
  • apps/web-next/src/lib/pymthouse-manifest.ts
  • apps/web-next/tests/leaderboard-posture.spec.ts
  • apps/workflows/developer-web/src/pages/DeveloperView.tsx
  • bin/db-setup.sh
  • bin/vercel-build.sh
  • docs/pymthouse-integration.md
  • package.json
  • plugins/developer-api/frontend/src/pages/DeveloperView.tsx
  • plugins/orchestrator-leaderboard/docs/how-to-guide.md
  • plugins/orchestrator-leaderboard/docs/openapi.yaml
  • plugins/orchestrator-leaderboard/frontend/src/App.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/AdminSettings.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/CapabilityGroupPicker.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/CapabilityTag.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/CollapsibleTagList.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/EndpointGuide.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/FormLabel.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/SectionLabel.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/StyledCheckbox.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/TabNav.tsx
  • plugins/orchestrator-leaderboard/frontend/src/globals.css
  • plugins/orchestrator-leaderboard/frontend/src/hooks/useCapabilityCatalog.ts
  • plugins/orchestrator-leaderboard/frontend/src/hooks/usePlans.ts
  • plugins/orchestrator-leaderboard/frontend/src/lib/api.ts
  • plugins/orchestrator-leaderboard/frontend/src/pages/LeaderboardPage.tsx
  • plugins/orchestrator-leaderboard/frontend/src/pages/PlanCreatePage.tsx
  • plugins/orchestrator-leaderboard/frontend/src/pages/PlanDetailPage.tsx
  • plugins/orchestrator-leaderboard/frontend/src/pages/PlansOverviewPage.tsx
💤 Files with no reviewable changes (2)
  • apps/web-next/src/app/api/v1/orchestrator-leaderboard/plans/seed/route.ts
  • plugins/orchestrator-leaderboard/docs/openapi.yaml
✅ Files skipped from review due to trivial changes (6)
  • apps/workflows/developer-web/src/pages/DeveloperView.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/TabNav.tsx
  • plugins/orchestrator-leaderboard/frontend/src/components/SectionLabel.tsx
  • bin/vercel-build.sh
  • package.json
  • docs/pymthouse-integration.md

Comment on lines +87 to +89
if (billingProvider === 'pymthouse') {
await ensurePymthouseManifestFresh();
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Move manifest refresh inside the existing error boundary

ensurePymthouseManifestFresh() is outside the try, so any refresh failure escapes your route-level error handling and status mapping. Keep it inside the try to preserve consistent API behavior.

Proposed fix
-  if (billingProvider === 'pymthouse') {
-    await ensurePymthouseManifestFresh();
-  }
-
   try {
+    if (billingProvider === 'pymthouse') {
+      await ensurePymthouseManifestFresh();
+    }
+
     const ordered: string[] = [];
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if (billingProvider === 'pymthouse') {
await ensurePymthouseManifestFresh();
}
try {
if (billingProvider === 'pymthouse') {
await ensurePymthouseManifestFresh();
}
const ordered: string[] = [];
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@apps/web-next/src/app/api/v1/orchestrator-leaderboard/python-gateway/route.ts`
around lines 87 - 89, The call to ensurePymthouseManifestFresh() is placed
outside the existing try/catch so any errors bypass your route's error boundary;
move the conditional block that calls ensurePymthouseManifestFresh() (the if
(billingProvider === 'pymthouse') { await ensurePymthouseManifestFresh(); }
logic) inside the same try that wraps the route processing so failures are
caught by the route's catch and correct status mapping is applied.

Comment on lines +21 to +33
<span className="relative inline-flex items-center justify-center shrink-0 w-3.5 h-3.5">
<input
type="checkbox"
checked={checked}
disabled={disabled}
aria-label={ariaLabel}
onChange={(e) => onChange(e.target.checked)}
className="peer sr-only"
/>
<span
aria-hidden
className="pointer-events-none absolute inset-0 rounded border border-zinc-600 bg-zinc-800 transition-colors peer-checked:bg-[var(--accent-emerald)] peer-checked:border-[var(--accent-emerald)] peer-disabled:opacity-50 peer-focus-visible:ring-2 peer-focus-visible:ring-[var(--accent-emerald)] peer-focus-visible:ring-offset-1 peer-focus-visible:ring-offset-[var(--bg-secondary)]"
/>
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Make the visible checkbox area actually clickable.

On Line 22, the native input is sr-only inside a non-label wrapper, so the visible box isn’t a reliable click target. This can make mouse interaction fail for most of the rendered control.

🔧 Proposed fix
-export const StyledCheckbox: React.FC<StyledCheckboxProps> = ({
+export const StyledCheckbox: React.FC<StyledCheckboxProps> = ({
   checked,
   disabled = false,
   onChange,
   'aria-label': ariaLabel,
 }) => (
-  <span className="relative inline-flex items-center justify-center shrink-0 w-3.5 h-3.5">
+  <label className="relative inline-flex items-center justify-center shrink-0 w-3.5 h-3.5 cursor-pointer">
     <input
       type="checkbox"
       checked={checked}
       disabled={disabled}
       aria-label={ariaLabel}
       onChange={(e) => onChange(e.target.checked)}
-      className="peer sr-only"
+      className="peer absolute inset-0 h-full w-full opacity-0 cursor-pointer"
     />
     <span
       aria-hidden
       className="pointer-events-none absolute inset-0 rounded border border-zinc-600 bg-zinc-800 transition-colors peer-checked:bg-[var(--accent-emerald)] peer-checked:border-[var(--accent-emerald)] peer-disabled:opacity-50 peer-focus-visible:ring-2 peer-focus-visible:ring-[var(--accent-emerald)] peer-focus-visible:ring-offset-1 peer-focus-visible:ring-offset-[var(--bg-secondary)]"
     />
     <Check
       size={9}
       aria-hidden
       className="relative z-[1] text-white opacity-0 transition-opacity peer-checked:opacity-100 pointer-events-none"
     />
-  </span>
+  </label>
 );
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/orchestrator-leaderboard/frontend/src/components/StyledCheckbox.tsx`
around lines 21 - 33, The visible checkbox isn’t clickable because the native
<input> is sr-only and not associated with the visible span; wrap the input and
the visible span in a clickable label (or make the input absolutely fill the
container and be invisible but not sr-only) inside the StyledCheckbox component
so clicks on the rendered box toggle the input; keep the input’s checked,
disabled, ariaLabel, onChange handler and the peer class intact (update the
input’s classes to either remove sr-only and use "absolute inset-0 w-full h-full
opacity-0 cursor-pointer" or wrap both elements in a <label> so the span remains
a visual-only element while clicks target the input).

Comment thread plugins/orchestrator-leaderboard/frontend/src/globals.css
Comment on lines +70 to +80
<button
onClick={() => setAutoRefresh(!autoRefresh)}
className={`flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium transition-all ${
autoRefresh
? 'bg-accent-emerald/15 text-accent-emerald border border-accent-emerald/30'
: 'bg-bg-secondary text-text-secondary border border-[var(--border-color)] hover:border-white/20'
}`}
>
<Radio size={12} className={autoRefresh ? 'animate-pulse' : ''} />
{autoRefresh ? 'Live' : 'Auto-refresh'}
</button>
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Expose toggle state semantics for the auto-refresh control.

This button behaves like a toggle, but it doesn’t currently expose pressed state for assistive tech. Add aria-pressed (and optionally a stateful aria-label).

Suggested fix
         <button
           onClick={() => setAutoRefresh(!autoRefresh)}
+          aria-pressed={autoRefresh}
+          aria-label={autoRefresh ? 'Disable auto-refresh' : 'Enable auto-refresh'}
           className={`flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium transition-all ${
             autoRefresh
               ? 'bg-accent-emerald/15 text-accent-emerald border border-accent-emerald/30'
               : 'bg-bg-secondary text-text-secondary border border-[var(--border-color)] hover:border-white/20'
           }`}
         >
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
<button
onClick={() => setAutoRefresh(!autoRefresh)}
className={`flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium transition-all ${
autoRefresh
? 'bg-accent-emerald/15 text-accent-emerald border border-accent-emerald/30'
: 'bg-bg-secondary text-text-secondary border border-[var(--border-color)] hover:border-white/20'
}`}
>
<Radio size={12} className={autoRefresh ? 'animate-pulse' : ''} />
{autoRefresh ? 'Live' : 'Auto-refresh'}
</button>
<button
onClick={() => setAutoRefresh(!autoRefresh)}
aria-pressed={autoRefresh}
aria-label={autoRefresh ? 'Disable auto-refresh' : 'Enable auto-refresh'}
className={`flex items-center gap-1.5 px-3 py-1.5 rounded-lg text-xs font-medium transition-all ${
autoRefresh
? 'bg-accent-emerald/15 text-accent-emerald border border-accent-emerald/30'
: 'bg-bg-secondary text-text-secondary border border-[var(--border-color)] hover:border-white/20'
}`}
>
<Radio size={12} className={autoRefresh ? 'animate-pulse' : ''} />
{autoRefresh ? 'Live' : 'Auto-refresh'}
</button>
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/orchestrator-leaderboard/frontend/src/pages/LeaderboardPage.tsx`
around lines 70 - 80, The auto-refresh toggle button does not expose its pressed
state to assistive tech; update the button (the element that calls
setAutoRefresh and reads autoRefresh, near Radio) to include
aria-pressed={autoRefresh} and add a stateful aria-label (e.g., "Auto-refresh
on" / "Auto-refresh off" derived from autoRefresh) so screen readers receive the
toggle semantics while preserving the existing onClick behavior and visual
classes.

…nctionality

- Added a modal for displaying test results and errors when running tests against the API endpoint.
- Implemented a new `testPlanResultsEndpoint` function to facilitate testing of the API with the current session token.
- Updated the `EndpointGuide` component to include a button for running tests, showing loading states and handling responses.
- Refactored the `CollapsibleTagList` component to prevent event propagation on toggle button clicks, improving user interaction.
- Renamed the section label in `PlanDetailPage` to "Discovery API" for better clarity.
eliteprox added 2 commits May 23, 2026 00:20
- Deleted the following API routes: `/api/v1/pymthouse/capabilities/catalog`, `/api/v1/pymthouse/network/price`, and `/api/v1/pymthouse/sla/summary`.
- Removed the `pymthouse-plan-builder.ts` file, which contained logic for building capabilities catalog, SLA summary, and network price payloads.
- Updated documentation to reflect the removal of these routes and the new data retrieval methods from the NaaP deployment.
…er-sdk

- Replaced all instances of `@pymthouse/builder-api` with `@pymthouse/builder-sdk` across the codebase.
- Updated the package version in `package.json` and `package-lock.json` to `0.0.8`.
- Modified environment variable documentation to reflect the new SDK usage.
- Ensured all related tests and configurations are aligned with the new package structure.
eliteprox added 2 commits May 23, 2026 00:41
- Added `rimraf` as a dependency for improved build script cleanup.
- Updated `.env.local.example` to correct the environment variable name for `PYMTHOUSE_BASE_URL`.
- Refactored cookie handling functions to support asynchronous operations, ensuring proper signing and parsing of device approval cookies.
- Enhanced rate limiting logic to handle anonymous requests more effectively.
- Improved error handling in API routes for better debugging and user feedback.
- Updated tests to reflect changes in cookie handling and ensure robust functionality.
- Added new development dependencies: `cac`, `check-error`, `deep-eql`, `loupe`, `pathval`, `strip-literal`, `tinypool`, and `tinyspy` with their respective versions and metadata.
- Updated `vite-node` to version `3.2.4` and included its dependencies.
- Regenerated `package-lock.json` to reflect the latest changes in the dependency tree.
…mantics

After the fail-closed change to isPipelineModelInManifest (a missing
manifest now denies by default), two tests started relying on side
effects of the old fail-open behavior:

- orchestrators-discovery-policy "passes when pair is on allowlist":
  the mock Response lacked `headers`, so `res.headers.get('etag')`
  threw and the catch left the snapshot null. Add `headers: new
  Headers()` so the manifest is actually parsed and the allow path is
  exercised.
- orchestrator-leaderboard refresh.test: the mockPlan used
  `billingProviderSlug: 'pymthouse'`, which triggers manifest
  filtering on every evaluation. Without creds/seed the manifest is
  null and every capability is filtered out. Use `null` so the cache
  and ranking logic (what these tests are actually covering) runs
  unaffected by manifest gating. Also add the now-required
  `visibility` field.

Co-authored-by: Cursor <cursoragent@cursor.com>
eliteprox added a commit that referenced this pull request May 23, 2026
Move orchestrator-leaderboard backend, manifest gating, python-gateway
routes, discovery-service client, plugin UI, and docs from #307 into
this PR. Excludes developer-api and billing OIDC/auth changes.

Includes capability catalog, tiered discovery shuffle, default/plan-scoped
python-gateway, PymtHouse manifest sync, and removal of demo seed API.

Co-authored-by: Cursor <cursoragent@cursor.com>
@eliteprox eliteprox changed the title feat: add PymtHouse capability integration follow-ups nexus: add PymtHouse capability integration follow-ups May 23, 2026
@eliteprox eliteprox marked this pull request as draft May 23, 2026 06:46
eliteprox added a commit that referenced this pull request May 26, 2026
Stack remaining #307 work on feat/discovery-plan-billing-provider:

- PymtHouse OIDC device-flow login and device approval routes
- Billing token/usage APIs and usage helpers
- Developer API key expiry and billing OAuth hardening
- developer-api plugin UI (usage, keys, discovery plan tooltip)
- UMD plugin stylesheet lifecycle, dev-api packages, builder-sdk
- Prisma pkceCodeVerifier on BillingProviderOAuthSession

Co-authored-by: Cursor <cursoragent@cursor.com>
eliteprox added a commit that referenced this pull request May 26, 2026
Stack remaining #307 work on feat/discovery-plan-billing-provider:

- PymtHouse OIDC device-flow login and device approval routes
- Billing token/usage APIs and usage helpers
- Developer API key expiry and billing OAuth hardening
- developer-api plugin UI (usage, keys, discovery plan tooltip)
- UMD plugin stylesheet lifecycle, dev-api packages, builder-sdk
- Prisma pkceCodeVerifier on BillingProviderOAuthSession

Co-authored-by: Cursor <cursoragent@cursor.com>
eliteprox added a commit that referenced this pull request May 27, 2026
Stack remaining #307 work on feat/discovery-plan-billing-provider:

- PymtHouse OIDC device-flow login and device approval routes
- Billing token/usage APIs and usage helpers
- Developer API key expiry and billing OAuth hardening
- developer-api plugin UI (usage, keys, discovery plan tooltip)
- UMD plugin stylesheet lifecycle, dev-api packages, builder-sdk
- Prisma pkceCodeVerifier on BillingProviderOAuthSession

Co-authored-by: Cursor <cursoragent@cursor.com>
eliteprox added a commit that referenced this pull request May 27, 2026
Stack remaining #307 work on feat/discovery-plan-billing-provider:

- PymtHouse OIDC device-flow login and device approval routes
- Billing token/usage APIs and usage helpers
- Developer API key expiry and billing OAuth hardening
- developer-api plugin UI (usage, keys, discovery plan tooltip)
- UMD plugin stylesheet lifecycle, dev-api packages, builder-sdk
- Prisma pkceCodeVerifier on BillingProviderOAuthSession

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added needs-rebase Has merge conflicts and removed needs-rebase Has merge conflicts labels May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

has-migration Includes database migration needs-rebase Has merge conflicts plugin/developer-api Developer API plugin scope/infra Infrastructure changes scope/packages Shared package changes scope/shell Shell app changes size/XL Extra large PR (500+ lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants