fix(validate): forward opaque pmth session to remote signer (workaround without pymthouse#255)#427
Closed
seanhanca wants to merge 21 commits into
Closed
fix(validate): forward opaque pmth session to remote signer (workaround without pymthouse#255)#427seanhanca wants to merge 21 commits into
seanhanca wants to merge 21 commits into
Conversation
PR #210 expects composite API keys on the remote signer DMZ. Replace the legacy user-JWT forward with per-user composite key mint at validate time, and forward composite PYMTHOUSE_API_KEY values directly without the signer-session exchange hop. Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses CodeRabbit review on #421: - Regression fix: move getSignerRouting() INSIDE the composite `.pmth_` fast-path. A bare `pmth_` key resolves its DMZ url via exchangeApiKeyForSignerSession() and never needed routing, so running the routing lookup first could spuriously throw "no remote signer DMZ url" for bare-key callers. Extracted resolveDmzUrl() to dedupe url extraction/validation. - Documented the legacy mint-per-call fallback as a tracked follow-up: key reuse/TTL/revoke needs new persistent secret storage (Builder API returns the secret only at creation); prod uses the composite fast-path and never mints here. - Added a regression test asserting the bare-pmth_ path skips getSignerRouting() and still resolves via the exchange's own signerUrl (24/24 pass). Co-authored-by: Cursor <cursoragent@cursor.com>
Document live status after deploying gateway 1bf13cd image that restores Daydream lv2v payments while keeping pymthouse byoc path on the shared node. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Probe script honors BYOC_SIGNER_URL over validate signerSession.url for staging canary without global pymthouse routing flip. Document Run 36 staging signer wiring (production webhook/Kafka/OpenMeter, global routing). Co-authored-by: Cursor <cursoragent@cursor.com>
Document OpenMeter baseline, validate 404 blocker, staging signer auth probe, and orchestrator reserve failure with no billed usage delta. Co-authored-by: Cursor <cursoragent@cursor.com>
Document root cause of prod validate 404 (front door OFF, no livepeer-dev override), sender reserve owner for app_98575870, and fix probe to use Bearer. Co-authored-by: Cursor <cursoragent@cursor.com>
Document flag state confirmation, key registration, canonical naap_ format requirement, and validate smoke passing with composite pmth bearer. Co-authored-by: Cursor <cursoragent@cursor.com>
Validate front door now returns composite bearer; BYOC probe still blocked on unfunded pymthouse app sender reserve (no wallet funding). Co-authored-by: Cursor <cursoragent@cursor.com>
Use ByocJobRequest from byoc.py with payload= and orch_url= so the probe imports correctly against the dual-path gateway branch. Co-authored-by: Cursor <cursoragent@cursor.com>
Run 42 confirms hosted naap_ 502 is prod DMZ rejecting type:byoc (not gateway), Daydream MCP regression safe on dual-path 1bf13cd, and adds blocker 2b to plan. Co-authored-by: Cursor <cursoragent@cursor.com>
Path A blocked on prod DMZ type:byoc rejection; Path B Daydream regression PASS. Co-authored-by: Cursor <cursoragent@cursor.com>
Path A still blocked on prod type:byoc and staging auth; top-up did not produce billed naap gens. Path B Daydream regression PASS. Co-authored-by: Cursor <cursoragent@cursor.com>
Run 44 addendum explains why staging preview returns not a JWT for app.pmth_ bearers while prod appears to accept them, and what John must fix on pymthouse. Co-authored-by: Cursor <cursoragent@cursor.com>
Record the composite bearer webhook fix PR so John can review and deploy. Co-authored-by: Cursor <cursoragent@cursor.com>
John's app_98575870 routing flip is live (validate + routing → test-production signer with type:byoc), but billed naap_ generation remains blocked at webhook 401 not a JWT until pymthouse#255 deploys. Co-authored-by: Cursor <cursoragent@cursor.com>
Stale _validate_session_cache on sdk-staging-1 caused naap_ inference to hit old prod DMZ despite SIGNER_FROM_VALIDATE=1; container restart aligns hosted path with test-production (401 not a JWT). Co-authored-by: Cursor <cursoragent@cursor.com>
Re-smoke confirms test-production routing and SDK cache fix hold; billed naap_ path blocked until composite bearer webhook PR merges. Co-authored-by: Cursor <cursoragent@cursor.com>
Live probes show pymthouse#255 still OPEN with merge conflicts, validate regressed to token-bundle (no composite endpoint), and SDK path fails with IncompleteRead after 401-class webhook auth. Documents minimum one-shot deploy set for John and latent post-#255 blockers. Co-authored-by: Cursor <cursoragent@cursor.com>
Workaround option 1 — no pymthouse#255 needed. resolveSignerEndpoint now returns getSignerRouting DMZ url + the validate-minted pmth_ bearer instead of minting or forwarding composite app.pmth_ API keys. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
scope/shell
|
|
|
Warning Review limit reached
Next review available in: 53 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe PR switches signer endpoint resolution to opaque session tokens, adds a gated BYOC E2E probe, updates signer configuration documentation, and adds comprehensive billed E2E audit, planning, and results documentation. ChangesBilled E2E flow
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@BILLED-E2E-BLOCKER-AUDIT.md`:
- Around line 35-41: Rewrite the audit’s blocker and deployment guidance to
reflect the opaque-session contract: remove composite PYMTHOUSE_API_KEY
configuration and minting/exchange remediation, and document that success
requires the enabled flag, successful getSignerRouting() resolution, and
forwarding the validation-minted opaque pmth_ token as the Bearer token to the
resolved DMZ endpoint. Update expected success signatures and webhook guidance
to require opaque-token support, removing claims that bare opaque tokens or
endpoint-form routing indicate failure.
In `@BILLED-E2E-REMAINING-PLAN.md`:
- Around line 143-156: The runbook still documents composite app_…pmth_… tokens
and lists pymthouse#255 as an active blocker. Update the referenced verification
steps, routing explanation, status tables, and blocker list to describe the
adapter forwarding the bare Bearer pmth_… opaque session minted during
validation, and remove or mark pymthouse#255 as resolved where applicable while
preserving the existing Daydream and NaaP path distinctions.
In `@scripts/byoc-e2e-probe.py`:
- Around line 24-27: Update the response handling in the request helper around
urllib.request.urlopen and urllib.error.HTTPError so the complete decoded
response body is returned without slicing to 500 characters. Preserve the
existing decoding and error replacement behavior, allowing downstream JSON
parsing and signerSession extraction to process the full response.
- Around line 54-78: Update the BYOC probe flow around submit_byoc_job and the
gateway ImportError handler to track whether the probe ran successfully; return
a nonzero failure status when submission raises an exception or the gateway
dependency cannot be imported, while preserving return 0 only for a successful
submission.
In `@USER-E2E-DEMO-RESULTS.md`:
- Around line 3927-3997: The historical Run 45 entry still presents
composite-bearer support as the active blocker; append a current-state
verification entry after the existing records for the new signer-session
contract. Cover the bare opaque pmth_ Authorization header, getSignerRouting()
fallback behavior, and whether direct opaque-session authentication succeeds,
while explicitly preserving the prior runs as historical and identifying any
remaining blocker from the new result.
- Around line 3-9: Redact operator and infrastructure identifiers throughout the
report sections around the visible metadata and the other referenced ranges,
replacing emails, hostnames, project/org/branch/database details, wallet
addresses, and persistent IDs with stable aliases. Preserve the report’s meaning
while moving any raw identifying evidence to a protected artifact rather than
committing it in this document.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 22d4f31f-ce69-4520-98f6-e8badb6b8948
📒 Files selected for processing (7)
BILLED-E2E-BLOCKER-AUDIT.mdBILLED-E2E-REMAINING-PLAN.mdUSER-E2E-DEMO-RESULTS.mdapps/web-next/src/lib/billing/pymthouse-adapter.test.tsapps/web-next/src/lib/billing/pymthouse-adapter.tsapps/web-next/src/lib/pymthouse-signer-exchange-config.tsscripts/byoc-e2e-probe.py
Comment on lines
+143
to
+156
| 1. `POST operator.livepeer.org/api/v1/keys/validate` (Bearer `naap_…`) → 200, `signerSession.url` + composite `Authorization`. | ||
| 2. `POST sdk.daydream.monster/inference` (flux-schnell, flux-dev, nano-banana) → 200 + image. | ||
| 3. `scripts/byoc-e2e-probe.py` with `NAAP_KEY` set → `submit_byoc_job: PASS`. | ||
| 4. OpenMeter: new gens under `byoc/<cap>` with fees > 0. | ||
|
|
||
| **Owner:** `j0sh` (#41 merge); qiang (redeploy); John (DMZ signer image + flags). | ||
|
|
||
| ### Dual-path deploy — Daydream + NaaP on one SDK node (Run 35 audit) | ||
|
|
||
| `sdk.daydream.monster` (`sdk-staging-1`) serves **both** Daydream keys (`pmth_` / `sk_`) and NaaP keys (`naap_`). Signer routing and payment type are **independent** layers: | ||
|
|
||
| | Layer | Daydream key (`pmth_` / `sk_`) | NaaP key (`naap_`) | | ||
| |---|---|---| | ||
| | **Signer selection** (`app.py` `_effective_signer`) | `SIGNER_FROM_VALIDATE=1` is **inert** — `_resolve_validate_session` returns `None` for non-`naap_` tokens → static `SIGNER_URL` = `https://signer.daydream.live` + caller `Authorization` forwarded | Validate → `signerSession.url` = pymthouse DMZ + composite `app.pmth_` bearer | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Update the runbook to describe the new bare opaque-session path.
These sections still expect a composite app_…pmth_… bearer and treat pymthouse#255 as a current blocker. The adapter now forwards Bearer pmth_… minted during validation, so the verification steps, routing explanation, status tables, and blocker list must reflect that contract.
Also applies to: 234-240, 297-301, 335-365, 384-386
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@BILLED-E2E-REMAINING-PLAN.md` around lines 143 - 156, The runbook still
documents composite app_…pmth_… tokens and lists pymthouse#255 as an active
blocker. Update the referenced verification steps, routing explanation, status
tables, and blocker list to describe the adapter forwarding the bare Bearer
pmth_… opaque session minted during validation, and remove or mark pymthouse#255
as resolved where applicable while preserving the existing Daydream and NaaP
path distinctions.
Comment on lines
+24
to
+27
| with urllib.request.urlopen(req, timeout=60) as resp: | ||
| return resp.status, resp.read().decode("utf-8", errors="replace")[:500] | ||
| except urllib.error.HTTPError as e: | ||
| return e.code, e.read().decode("utf-8", errors="replace")[:500] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not truncate the response before parsing it.
A validate response longer than 500 characters becomes invalid JSON at Line 42, causing the probe to crash before extracting signerSession.
Proposed fix
with urllib.request.urlopen(req, timeout=60) as resp:
- return resp.status, resp.read().decode("utf-8", errors="replace")[:500]
+ return resp.status, resp.read().decode("utf-8", errors="replace")
except urllib.error.HTTPError as e:
- return e.code, e.read().decode("utf-8", errors="replace")[:500]
+ return e.code, e.read().decode("utf-8", errors="replace")Also applies to: 42-42
🧰 Tools
🪛 Ruff (0.15.21)
[error] 24-24: Audit URL open for permitted schemes. Allowing use of file: or custom schemes is often unexpected.
(S310)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/byoc-e2e-probe.py` around lines 24 - 27, Update the response handling
in the request helper around urllib.request.urlopen and urllib.error.HTTPError
so the complete decoded response body is returned without slicing to 500
characters. Preserve the existing decoding and error replacement behavior,
allowing downstream JSON parsing and signerSession extraction to process the
full response.
Comment on lines
+54
to
+78
| try: | ||
| sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "livepeer-python-gateway", "src")) | ||
| from livepeer_gateway.byoc import ByocJobRequest, submit_byoc_job | ||
|
|
||
| req = ByocJobRequest(capability=CAP, payload={"prompt": "probe", "width": 512, "height": 512}) | ||
| try: | ||
| submit_byoc_job( | ||
| req, | ||
| orch_url=ORCH, | ||
| signer_url=signer_url, | ||
| signer_headers=headers, | ||
| timeout=120.0, | ||
| ) | ||
| print("submit_byoc_job: PASS") | ||
| except Exception as exc: | ||
| msg = str(exc) | ||
| print(f"submit_byoc_job: FAIL — {msg[:300]}") | ||
| if "invalid job type" in msg: | ||
| print(" → signer PR not deployed yet") | ||
| if "Could not verify job creds" in msg: | ||
| print(" → orch V1 verify PR not deployed yet") | ||
| except ImportError as exc: | ||
| print(f"gateway import skipped: {exc}") | ||
|
|
||
| return 0 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Return a failure status when the BYOC probe fails or cannot run.
Both submission errors and missing gateway dependencies currently fall through to return 0, allowing CI or operators to treat a failed/skipped probe as successful.
Proposed fix
except Exception as exc:
msg = str(exc)
print(f"submit_byoc_job: FAIL — {msg[:300]}")
if "invalid job type" in msg:
print(" → signer PR not deployed yet")
if "Could not verify job creds" in msg:
print(" → orch V1 verify PR not deployed yet")
+ return 1
except ImportError as exc:
print(f"gateway import skipped: {exc}")
+ return 2
return 0📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| try: | |
| sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "livepeer-python-gateway", "src")) | |
| from livepeer_gateway.byoc import ByocJobRequest, submit_byoc_job | |
| req = ByocJobRequest(capability=CAP, payload={"prompt": "probe", "width": 512, "height": 512}) | |
| try: | |
| submit_byoc_job( | |
| req, | |
| orch_url=ORCH, | |
| signer_url=signer_url, | |
| signer_headers=headers, | |
| timeout=120.0, | |
| ) | |
| print("submit_byoc_job: PASS") | |
| except Exception as exc: | |
| msg = str(exc) | |
| print(f"submit_byoc_job: FAIL — {msg[:300]}") | |
| if "invalid job type" in msg: | |
| print(" → signer PR not deployed yet") | |
| if "Could not verify job creds" in msg: | |
| print(" → orch V1 verify PR not deployed yet") | |
| except ImportError as exc: | |
| print(f"gateway import skipped: {exc}") | |
| return 0 | |
| try: | |
| sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "livepeer-python-gateway", "src")) | |
| from livepeer_gateway.byoc import ByocJobRequest, submit_byoc_job | |
| req = ByocJobRequest(capability=CAP, payload={"prompt": "probe", "width": 512, "height": 512}) | |
| try: | |
| submit_byoc_job( | |
| req, | |
| orch_url=ORCH, | |
| signer_url=signer_url, | |
| signer_headers=headers, | |
| timeout=120.0, | |
| ) | |
| print("submit_byoc_job: PASS") | |
| except Exception as exc: | |
| msg = str(exc) | |
| print(f"submit_byoc_job: FAIL — {msg[:300]}") | |
| if "invalid job type" in msg: | |
| print(" → signer PR not deployed yet") | |
| if "Could not verify job creds" in msg: | |
| print(" → orch V1 verify PR not deployed yet") | |
| return 1 | |
| except ImportError as exc: | |
| print(f"gateway import skipped: {exc}") | |
| return 2 | |
| return 0 |
🧰 Tools
🪛 Ruff (0.15.21)
[warning] 68-68: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/byoc-e2e-probe.py` around lines 54 - 78, Update the BYOC probe flow
around submit_byoc_job and the gateway ImportError handler to track whether the
probe ran successfully; return a nonzero failure status when submission raises
an exception or the gateway dependency cannot be imported, while preserving
return 0 only for a successful submission.
Comment on lines
+3
to
+9
| **Operator:** `qiang@livepeer.org` (`system:admin`), NaaP prod `https://operator.livepeer.org`. | ||
| **Method:** Automated the previously-manual prereqs via a Neon-API-obtained prod `DATABASE_URL` | ||
| (authorized) + admin flag-override API. Drove the full billed chain to the point of failure. | ||
| **Secrets:** Neon API key and the prod connection URI were handled as sensitive — never written to | ||
| this file, logs, or the report. Connection was via the Neon project `green-base-78237656` ("naap", | ||
| org `Vercel: Livepeer Foundation`), primary branch `br-patient-cake-aigip99y` (`main`), db `neondb`, | ||
| role `neondb_owner`. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the file and inspect the referenced line ranges.
wc -l USER-E2E-DEMO-RESULTS.md
sed -n '1,40p' USER-E2E-DEMO-RESULTS.md
printf '\n--- lines 700-740 ---\n'
sed -n '700,740p' USER-E2E-DEMO-RESULTS.md
printf '\n--- lines 3368-3392 ---\n'
sed -n '3368,3392p' USER-E2E-DEMO-RESULTS.md
# Also search for the specific identifiers mentioned in the review comment.
printf '\n--- identifier search ---\n'
rg -n "qiang@livepeer.org|DATABASE_URL|Neon|green-base-78237656|br-patient-cake-aigip99y|neondb_owner|wallet|client ID|operator.livepeer.org|Vercel: Livepeer Foundation" USER-E2E-DEMO-RESULTS.md || trueRepository: livepeer/naap
Length of output: 25041
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the exact content around the reported ranges.
for range in "1,20" "719,731" "3376,3384"; do
echo "--- $range ---"
sed -n "${range}p" USER-E2E-DEMO-RESULTS.md
doneRepository: livepeer/naap
Length of output: 3869
Redact operator and infrastructure identifiers from the report (lines 3-9, 719-731, 3376-3384). This committed report exposes an operator email plus Neon project/org/branch/db details, internal hostnames, wallet addresses, and persistent IDs; replace them with stable aliases and keep raw evidence in a protected artifact.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@USER-E2E-DEMO-RESULTS.md` around lines 3 - 9, Redact operator and
infrastructure identifiers throughout the report sections around the visible
metadata and the other referenced ranges, replacing emails, hostnames,
project/org/branch/database details, wallet addresses, and persistent IDs with
stable aliases. Preserve the report’s meaning while moving any raw identifying
evidence to a protected artifact rather than committing it in this document.
Comment on lines
+3927
to
+3997
| ## Blockers (updated) | ||
|
|
||
| 1. **P0 — Prod DMZ `type:byoc`:** unchanged. | ||
| 2. **P1 — Staging preview composite bearer:** must fix before sender-reserve top-up can be validated. | ||
| 3. **P2 — Sender reserve:** John reports funded; re-test after P1 (expect Run 42-style `no sender reserve` → success). | ||
|
|
||
| ## Artifacts | ||
|
|
||
| - `/tmp/run44-validate.json`, `/tmp/run44-om-baseline.json`, `/tmp/run44-om-after.json` | ||
| - `/tmp/run44-probe-prod-schnell.txt`, `/tmp/run44-probe-staging-schnell.txt`, `/tmp/run44-dmz-matrix.txt` | ||
| - `/tmp/run44-sdk-inference-naap.json`, `/tmp/run44-sdk-inference-daydream.json` | ||
|
|
||
| ## Spend | ||
|
|
||
| **≈ $0.006** — two Path B flux-schnell generations (Storyboard MCP + direct SDK Daydream bearer), **$0.00320** each. Path A: **$0** (no billed generation; top-up did not change meter). | ||
|
|
||
| --- | ||
|
|
||
| ## Run 44 addendum — staging 401 `not a JWT` root cause (2026-07-16 ~12:05 PT) | ||
|
|
||
| Independent investigation of the Run 43/44 staging auth regression. **Verdict: not a Railway staging-signer config regression — it is a pymthouse.com webhook verifier gap, unmasked after sender-reserve top-up.** | ||
|
|
||
| ### TL;DR | ||
|
|
||
| | question | answer | | ||
| |---|---| | ||
| | **Why 401 on staging?** | Staging signer (`#3980` + `-byocPerCapPricing`) now reaches `authLivePayment` → `POST pymthouse.com/webhooks/remote-signer`. The webhook verifier chain accepts **bare `pmth_*` opaque sessions** and **OIDC JWTs** only — **not** NaaP's composite `app_XXX.pmth_YYY` API-key bearer. OIDC fallback returns `"not a JWT"` → go-livepeer surfaces **HTTP 401**. | | ||
| | **Why prod "accepts" composite?** | **Misleading.** Prod DMZ lacks `#3980` → `type:byoc` fails at **line-710 type gate** (`invalid job type`) **before** webhook. `type:lv2v` control fails at **numTickets > 100** **before** webhook. Prod never exercises webhook auth in matrix probes. | | ||
| | **Was Run 42 auth OK?** | **No — auth was never reached.** Run 42 staging failed at **`no sender reserve`** during `genPayment` (**before** `authLivePayment` at line 809). Composite bearer was **not** validated in Run 42; the top-up in Run 44 advanced the failure point from reserve → webhook. | | ||
| | **What auth does staging expect?** | **Composite `app.pmth_` bearer** (same as prod validate output). Staging **expects** it; pymthouse **webhook** does not verify it yet. JWT-only docs in `builder-api.md` are stale post-PR #210 / NaaP #421. | | ||
| | **NAAP validate** | **PASS** — still mints composite `app_98575870….pmth_…` + prod DMZ URL (reconfirmed `/tmp/run44-validate.json`). | | ||
|
|
||
| ### Auth path comparison — prod vs staging preview | ||
|
|
||
| Both Railway services use the **same signer-dmz entrypoint** pattern (`docker/signer-dmz/entrypoint.sh`): | ||
|
|
||
| | config | prod `pymthouse-production` | staging `pymthouse-signer-test-preview` | | ||
| |---|---|---| | ||
| | **Image / binary** | Older go-livepeer (**no `#3980` `type:byoc`**) | Newer go-livepeer (**`#3980` ON** + `-byocPerCapPricing`) | | ||
| | **`REMOTE_SIGNER_WEBHOOK_URL`** | `https://pymthouse.com/webhooks/remote-signer` (expected) | Same (expected — `railway-signer-env.sh` default) | | ||
| | **`WEBHOOK_SECRET`** | Shared with Vercel pymthouse | Same shared secret (expected) | | ||
| | **OIDC / JWKS** | `NEXTAUTH_URL=https://pymthouse.com`, issuer `…/api/v1/oidc` | Same production pymthouse backend (John clone) | | ||
| | **Webhook host** | **Vercel pymthouse.com** (not Railway) | **Same Vercel webhook** — fix is **app-side**, not staging Railway env | | ||
|
|
||
| **Only meaningful runtime difference:** staging go-livepeer accepts `type:byoc` and proceeds far enough to call the identity webhook; prod rejects `type:byoc` earlier. | ||
|
|
||
| ### Webhook verifier chain (root cause) | ||
|
|
||
| `pymthouse/src/app/webhooks/remote-signer/route.ts` builds: | ||
|
|
||
| ```text | ||
| createFirstMatchEndUserVerifier([ | ||
| opaqueSessionVerifier, // matches token.startsWith("pmth_") → sessions table | ||
| legacyOidcVerifier, // matches 3-part JWT → else throws "not a JWT" | ||
| ]) | ||
| ``` | ||
|
|
||
| NaaP composite bearer format: `app_98575870d7ae33589a3f0660.pmth_<hex>` (PR #421 / pymthouse PR #210). | ||
|
|
||
| - **Fails opaque verifier** — does not start with `pmth_` (starts with `app_`). | ||
| - **Fails OIDC verifier** — not a 3-part JWT → `@pymthouse/clearinghouse-identity-webhook` throws `WebhookError("not a JWT", { status: 401 })`. | ||
| - **Missing verifier** — no `resolveActiveAppApiKey()` path (`app-api-keys.ts` already implements DB lookup for `pmth_*` + `clientId`). | ||
|
|
||
| `/sign-orchestrator-info` returns **HTTP 200** for composite bearer on **both** hosts — **does not call the webhook** (`SignOrchestratorInfo` has no auth). Do not use it as composite-auth proof. | ||
|
|
||
| ### Live minimal probes (2026-07-16, $0 spend) | ||
|
|
||
| | probe | prod DMZ | staging preview | | ||
| |---|---|---| | ||
| | `GET /healthz` | **200** | **200** | | ||
| | `POST /generate-live-payment` empty body (no bearer) | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Add a post-PR verification entry for the new signer-session contract.
The latest runs still report Authorization: Bearer app_….pmth_… and treat pymthouse#255 composite-bearer webhook support as the active blocker. This PR instead forwards the opaque pmth_… session directly and removes the composite/JWT fallback paths. Keep these runs as historical records, but append an updated current-state result covering the bare pmth_… header, getSignerRouting() fallback behavior, and whether the direct opaque session is accepted; otherwise the ledger will direct operators toward an obsolete blocker.
Also applies to: 4063-4106, 4114-4257
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@USER-E2E-DEMO-RESULTS.md` around lines 3927 - 3997, The historical Run 45
entry still presents composite-bearer support as the active blocker; append a
current-state verification entry after the existing records for the new
signer-session contract. Cover the bare opaque pmth_ Authorization header,
getSignerRouting() fallback behavior, and whether direct opaque-session
authentication succeeds, while explicitly preserving the prior runs as
historical and identifying any remaining blocker from the new result.
Capture builder-sdk 0.6.0 format change, live validate/inference re-probe, and revised action plan now that #424 is on main. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Superseded by merged #424 (builder-sdk 0.6.0, clearinghouse). Do not merge. |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Workaround for the staging/prod remote-signer 401 blocker (pymthouse#255) without waiting for composite
app_XXX.pmth_YYYbearer support to land upstream.When
PER_KEY_REMOTE_SIGNERis enabled, BPP validate already mints an opaquepmth_…signer session. This change makesresolveSignerEndpointforward that session directly to the remote signer DMZ instead of minting or exchanging a composite API key:getSignerRouting()to resolve the direct-DMZ signer API URL (patterns.directDmz.signerApiUrl, with fallbacks).{ url, headers: { Authorization: Bearer <pmth_…> } }using the validate-minted opaque token.The prod remote-signer identity webhook already verifies bare opaque
pmth_…sessions. Compositeapp_XXX.pmth_YYYkeys are what require pymthouse#255 — this path avoids them entirely.Removes the composite-key mint (
createPymthouseApiKey), api-key signer-session exchange, and user-JWT mint branches fromresolveSignerEndpoint(~340 lines deleted).Requires:
PER_KEY_REMOTE_SIGNERfeature flag ON (front door already gates endpoint-form responses on this flag).Test plan
apps/web-next/src/lib/billing/pymthouse-adapter.test.ts— opaque session forwarded with DMZ url, per-instance client routing, fallbacks, missing-DMZ throw, dashboard proxy rejection, noexternalUserIdrequiredPER_KEY_REMOTE_SIGNER=1signerEndpointwithAuthorization: Bearer pmth_…(not composite key)Made with Cursor
Summary by CodeRabbit
Bug Fixes
New Features
Documentation