M016: Enterprise Scale

[TerrifiedBug]

Summary

Enterprise-scale features for VectorFlow — enabling corporate platform teams to manage hundreds of pipelines across multi-environment fleets of 100+ nodes.

7 phases, 31 requirements, 19 plans executed:

• Phase 1: Fleet Performance — Alert eval batched in poll loop (not per-heartbeat), SSE connection limit (1000 default), lazy catalog singleton
• Phase 2: Fleet Organization — Node groups with label enforcement + auto-enrollment, nested pipeline groups (3-level), bulk tag add/remove
• Phase 3: Fleet Health Dashboard — Group-level health cards with expand/collapse drill-down, filter by group/label/compliance, 30s polling
• Phase 4: Outbound Webhooks — WebhookEndpoint + WebhookDelivery models, Standard-Webhooks HMAC-SHA256 signing, retry with dead-letter separation, management UI
• Phase 5: Cross-Env Promotion (UI) — PromotionRequest model, secret pre-flight validation, 5-step promotion wizard with substitution diff, approval workflow, promotion history
• Phase 6: OpenAPI Specification — 31-operation OpenAPI 3.1 spec (16 REST v1 + 15 tRPC), generation script, serving endpoint, API reference docs
• Phase 7: Cross-Env Promotion (GitOps) — @octokit/rest PR creation, merge-triggered auto-deploy, setup wizard with connection validation

Stats: 69 files changed, ~11,000 lines added, 935 tests passing (up from 792)

Test plan

• pnpm test — all 935 tests pass
• pnpm lint — no new errors
• pnpm build — production build succeeds
• Fleet health dashboard renders at /fleet/health with node group cards
• Pipeline promotion wizard opens from "Promote to..." dropdown
• Webhook management accessible at /settings/webhooks
• OpenAPI spec accessible at /api/v1/openapi.json
• GitOps promotion mode appears in environment GitSync settings

[greptile-apps]

Greptile Summary

This PR delivers 7 phases of enterprise-scale features: fleet node groups with label enforcement, a fleet health dashboard, outbound webhooks (Standard-Webhooks signed), cross-environment pipeline promotion (UI + GitOps PR flow), an OpenAPI 3.1 spec, and several performance improvements. The scope is large (~11k lines) but well-structured — each feature has its own router, service, and tests. Three issues need attention before merge.

• P1 — msgId mismatch in outbound webhook delivery (outbound-webhook.ts): dispatchWithTracking generates a msgId for the WebhookDelivery DB record, while deliverOutboundWebhook independently generates its own msgId for the webhook-id HTTP header and HMAC signing string. These are always different UUIDs, which breaks Standard-Webhooks deduplication for receivers and makes VectorFlow's delivery history UI uncorrelatable with received webhook IDs.
• P1 — Cross-team node data exposure in nodesInGroup (node-group.ts): The withTeamAccess("VIEWER") check is scoped to input.environmentId, but the node group is fetched by groupId alone (no environment scope). Nodes are then queried against group.environmentId, which can point to a different team's environment if a crafted groupId is passed.
• P1 — testDelivery mutation missing withAudit (webhook-endpoint.ts): Every mutation must carry withAudit per the project security policy; testDelivery is the only new mutation without it.
• P1 (verify) — withTeamAccess may not resolve from requestId (promotion.ts): approve and reject pass { requestId } to withTeamAccess("EDITOR"). The middleware resolves team context from known field names (id, teamId, environmentId, pipelineId); confirm it handles requestId, or rename the field to align with the convention.

Confidence Score: 3/5

Not safe to merge as-is — cross-team node data exposure is a live security bug, and the msgId mismatch breaks the Standard-Webhooks contract on day one.

Three confirmed P1 issues: a cross-team data exposure vulnerability in the node-group router, a correctness bug in outbound webhook ID tracking that will manifest on every first delivery, and a missing audit trail on the test-delivery mutation. Additionally, withTeamAccess resolution from requestId needs explicit verification. The rest of the code is well-written and structurally sound.

src/server/routers/node-group.ts (cross-team exposure), src/server/services/outbound-webhook.ts (msgId mismatch), src/server/routers/webhook-endpoint.ts (missing audit), src/server/routers/promotion.ts (withTeamAccess field name)

Important Files Changed

Filename	Overview
src/server/services/outbound-webhook.ts	New outbound webhook delivery service — has a msgId mismatch bug where the DB record ID and the webhook-id HTTP header are always different UUIDs
src/server/routers/node-group.ts	New node group router — nodesInGroup fetches nodes using group.environmentId without verifying the group belongs to input.environmentId, enabling cross-team node data exposure
src/server/routers/webhook-endpoint.ts	New webhook endpoint management router — solid auth/ownership checks throughout, but testDelivery mutation is missing withAudit middleware
src/server/routers/promotion.ts	New promotion router — good atomic race-condition guards and self-review block; approve/reject use requestId as input which may not be resolvable by withTeamAccess
src/server/services/gitops-promotion.ts	New GitOps PR-creation service — correctly decrypts token, parses GitHub URLs, creates unique branch names, and embeds promotion ID in PR body for webhook correlation
src/app/api/webhooks/git/route.ts	Extended git webhook handler — adds PR merge trigger for GitOps promotions with HMAC validation, idempotency guard, and atomic status transition; overall sound
src/server/services/retry-service.ts	Extended retry service to handle outbound webhook retries — correctly excludes dead_letter records, uses atomic claim pattern, and handles disabled endpoints
src/app/api/sse/route.ts	Adds per-instance SSE connection limit (default 1000) with 503 + Retry-After response — clean and correct
prisma/schema.prisma	Adds NodeGroup, PromotionRequest, WebhookEndpoint, WebhookDelivery models with correct cascade deletes, indexes, and enum additions
src/app/api/agent/enroll/route.ts	Adds auto-enrollment label template application from matching NodeGroups — correctly non-fatal, well-scoped to the new node's environment

Sequence Diagram

sequenceDiagram
    participant UI as Browser
    participant tRPC as tRPC Router
    participant Octokit as GitHub API
    participant GHWebhook as /api/webhooks/git
    participant PromSvc as promotion-service
    participant OutWH as outbound-webhook

    Note over UI,OutWH: Phase 5/7 – Pipeline Promotion (GitOps path)

    UI->>tRPC: promotion.initiate(pipelineId, targetEnvId)
    tRPC->>PromSvc: preflightSecrets()
    PromSvc-->>tRPC: {canProceed, missing[]}
    tRPC->>Octokit: createPromotionPR(encryptedToken, yaml)
    Octokit-->>tRPC: {prNumber, prUrl}
    tRPC-->>UI: {status: AWAITING_PR_MERGE, prUrl}

    Note over Octokit,GHWebhook: Developer merges PR on GitHub
    Octokit->>GHWebhook: POST pull_request (merged)
    GHWebhook->>GHWebhook: HMAC verify + parse requestId from PR body
    GHWebhook->>GHWebhook: atomic updateMany(AWAITING_PR_MERGE→DEPLOYING)
    GHWebhook->>PromSvc: executePromotion(requestId, promotedById)
    PromSvc-->>GHWebhook: {pipelineId, pipelineName}
    GHWebhook->>OutWH: fireOutboundWebhooks(promotion_completed)
    GHWebhook-->>Octokit: {deployed: true}

    Note over UI,OutWH: Phase 4 – Outbound Webhooks (event delivery)
    PromSvc->>OutWH: fireOutboundWebhooks(metric, teamId, payload)
    OutWH->>OutWH: dispatchWithTracking() — creates WebhookDelivery record
    OutWH->>OutWH: deliverOutboundWebhook() — HMAC sign + POST
    Note right of OutWH: ⚠ msgId in DB ≠ webhook-id header
    OutWH-->>PromSvc: success / retryable / dead_letter

Loading

Comments Outside Diff (4)

1. src/server/services/outbound-webhook.ts, line 1476-1494 (link)

   msgId stored in DB doesn't match webhook-id HTTP header

   dispatchWithTracking generates a msgId at line 1481 and stores it in the WebhookDelivery record. But deliverOutboundWebhook (called on line 1494) internally generates its own fresh msgId (via crypto.randomUUID()) at runtime, which is what actually goes into the webhook-id header and the HMAC signing string.

   These two UUIDs are always different. This breaks the Standard-Webhooks contract: the webhook-id that receivers use for idempotency/deduplication has no relationship to the msgId stored in WebhookDelivery. Anyone inspecting delivery history in the management UI cannot correlate it with the webhook-id they received.

   The fix is to generate msgId once in dispatchWithTracking and pass it into deliverOutboundWebhook as a parameter, so both the DB record and the HTTP header share the same ID.

   // In dispatchWithTracking — pass msgId to the delivery function
   const msgId = crypto.randomUUID();
   const delivery = await prisma.webhookDelivery.create({ data: { ..., msgId } });
   const result = await deliverOutboundWebhook(endpoint, payload, msgId);

   // In deliverOutboundWebhook — accept an optional msgId instead of always creating one
   export async function deliverOutboundWebhook(
     endpoint: EndpointLike,
     payload: OutboundPayload,
     msgId = crypto.randomUUID(), // default for one-off/test calls
   ): Promise<OutboundResult> {

   Rule Used: ## Security & Cryptography Review Rules

   When revi... (source)

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/server/services/outbound-webhook.ts
   Line: 1476-1494
   
   Comment:
   **`msgId` stored in DB doesn't match `webhook-id` HTTP header**
   
   `dispatchWithTracking` generates a `msgId` at line 1481 and stores it in the `WebhookDelivery` record. But `deliverOutboundWebhook` (called on line 1494) internally generates its own fresh `msgId` (via `crypto.randomUUID()`) at runtime, which is what actually goes into the `webhook-id` header and the HMAC signing string.
   
   These two UUIDs are always different. This breaks the Standard-Webhooks contract: the `webhook-id` that receivers use for idempotency/deduplication has no relationship to the `msgId` stored in `WebhookDelivery`. Anyone inspecting delivery history in the management UI cannot correlate it with the `webhook-id` they received.
   
   The fix is to generate `msgId` once in `dispatchWithTracking` and pass it into `deliverOutboundWebhook` as a parameter, so both the DB record and the HTTP header share the same ID.
   
   ```typescript
   // In dispatchWithTracking — pass msgId to the delivery function
   const msgId = crypto.randomUUID();
   const delivery = await prisma.webhookDelivery.create({ data: { ..., msgId } });
   const result = await deliverOutboundWebhook(endpoint, payload, msgId);
   ```
   
   ```typescript
   // In deliverOutboundWebhook — accept an optional msgId instead of always creating one
   export async function deliverOutboundWebhook(
     endpoint: EndpointLike,
     payload: OutboundPayload,
     msgId = crypto.randomUUID(), // default for one-off/test calls
   ): Promise<OutboundResult> {
   ```
   
   **Rule Used:** ## Security & Cryptography Review Rules
   
   When revi... ([source](https://app.greptile.com/review/custom-context?memory=7cb20c56-ca6a-40aa-8660-7fa75e6e3db2))
   
   How can I resolve this? If you propose a fix, please make it concise.

2. src/server/routers/node-group.ts, line 426-461 (link)

   Cross-team node data exposure via unchecked groupId

   withTeamAccess("VIEWER") resolves the team check from input.environmentId. However, the group is fetched without verifying it belongs to that same environment:

   const group = await prisma.nodeGroup.findUnique({
     where: { id: groupId },  // no environmentId check
   });
   // ...
   const allNodes = await prisma.vectorNode.findMany({
     where: { environmentId: group.environmentId },  // uses the group's env, not input.environmentId
   });

   An attacker can pass their own valid environmentId (to pass the VIEWER check) alongside a groupId belonging to a different team's environment. The team auth check passes, but the node query returns data scoped to the other team's group.environmentId.

   Fix: scope the group lookup to input.environmentId:

   const group = await prisma.nodeGroup.findFirst({
     where: { id: groupId, environmentId: input.environmentId },
   });

   Rule Used: ## Security & Cryptography Review Rules

   When revi... (source)

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/server/routers/node-group.ts
   Line: 426-461
   
   Comment:
   **Cross-team node data exposure via unchecked `groupId`**
   
   `withTeamAccess("VIEWER")` resolves the team check from `input.environmentId`. However, the group is fetched without verifying it belongs to that same environment:
   
   ```typescript
   const group = await prisma.nodeGroup.findUnique({
     where: { id: groupId },  // no environmentId check
   });
   // ...
   const allNodes = await prisma.vectorNode.findMany({
     where: { environmentId: group.environmentId },  // uses the group's env, not input.environmentId
   });
   ```
   
   An attacker can pass their own valid `environmentId` (to pass the VIEWER check) alongside a `groupId` belonging to a different team's environment. The team auth check passes, but the node query returns data scoped to the other team's `group.environmentId`.
   
   Fix: scope the group lookup to `input.environmentId`:
   
   ```typescript
   const group = await prisma.nodeGroup.findFirst({
     where: { id: groupId, environmentId: input.environmentId },
   });
   ```
   
   **Rule Used:** ## Security & Cryptography Review Rules
   
   When revi... ([source](https://app.greptile.com/review/custom-context?memory=7cb20c56-ca6a-40aa-8660-7fa75e6e3db2))
   
   How can I resolve this? If you propose a fix, please make it concise.

3. src/server/routers/webhook-endpoint.ts, line 1123-1152 (link)

   testDelivery mutation is missing withAudit

   Every mutation must use withAudit(action, entityType) per the project's security policy. testDelivery is a mutation that triggers an outbound HTTP request and creates/updates a WebhookDelivery record, but it has no audit middleware:

   testDelivery: protectedProcedure
     .input(z.object({ id: z.string(), teamId: z.string() }))
     .use(withTeamAccess("ADMIN"))
     // ← missing: .use(withAudit("webhookEndpoint.testDelivery", "WebhookEndpoint"))
     .mutation(async ({ input }) => {

   Add .use(withAudit("webhookEndpoint.testDelivery", "WebhookEndpoint")) after withTeamAccess.

   Rule Used: ## Security & Cryptography Review Rules

   When revi... (source)

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/server/routers/webhook-endpoint.ts
   Line: 1123-1152
   
   Comment:
   **`testDelivery` mutation is missing `withAudit`**
   
   Every mutation must use `withAudit(action, entityType)` per the project's security policy. `testDelivery` is a mutation that triggers an outbound HTTP request and creates/updates a `WebhookDelivery` record, but it has no audit middleware:
   
   ```typescript
   testDelivery: protectedProcedure
     .input(z.object({ id: z.string(), teamId: z.string() }))
     .use(withTeamAccess("ADMIN"))
     // ← missing: .use(withAudit("webhookEndpoint.testDelivery", "WebhookEndpoint"))
     .mutation(async ({ input }) => {
   ```
   
   Add `.use(withAudit("webhookEndpoint.testDelivery", "WebhookEndpoint"))` after `withTeamAccess`.
   
   **Rule Used:** ## Security & Cryptography Review Rules
   
   When revi... ([source](https://app.greptile.com/review/custom-context?memory=7cb20c56-ca6a-40aa-8660-7fa75e6e3db2))
   
   How can I resolve this? If you propose a fix, please make it concise.

4. src/server/routers/promotion.ts, line 784-788 (link)

   approve and reject use EDITOR role — withTeamAccess may not resolve from requestId

   Both approve and reject declare .use(withTeamAccess("EDITOR")) with requestId as the sole input field. The architecture docs list the fields withTeamAccess recognises for team resolution: teamId, environmentId, pipelineId, and id. The field here is named requestId, not id.

   If withTeamAccess only matches on the exact field name id, the middleware won't be able to resolve a team context from requestId, which could mean the authorization check silently passes regardless of the caller's actual team membership.

   Check the withTeamAccess implementation to confirm it handles requestId — or rename the input field to id for consistency with the convention, or explicitly pass a teamId alongside it.

   Rule Used: ## Code Style & Conventions

   TypeScript Conven... (source)

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/server/routers/promotion.ts
   Line: 784-788
   
   Comment:
   **`approve` and `reject` use `EDITOR` role — `withTeamAccess` may not resolve from `requestId`**
   
   Both `approve` and `reject` declare `.use(withTeamAccess("EDITOR"))` with `requestId` as the sole input field. The architecture docs list the fields `withTeamAccess` recognises for team resolution: `teamId`, `environmentId`, `pipelineId`, and `id`. The field here is named `requestId`, not `id`.
   
   If `withTeamAccess` only matches on the exact field name `id`, the middleware won't be able to resolve a team context from `requestId`, which could mean the authorization check silently passes regardless of the caller's actual team membership.
   
   Check the `withTeamAccess` implementation to confirm it handles `requestId` — or rename the input field to `id` for consistency with the convention, or explicitly pass a `teamId` alongside it.
   
   **Rule Used:** ## Code Style & Conventions
   
   ### TypeScript Conven... ([source](https://app.greptile.com/review/custom-context?memory=6ae51394-d0b6-4686-bc4c-1ad840c2e310))
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: src/server/services/outbound-webhook.ts
Line: 1476-1494

Comment:
**`msgId` stored in DB doesn't match `webhook-id` HTTP header**

`dispatchWithTracking` generates a `msgId` at line 1481 and stores it in the `WebhookDelivery` record. But `deliverOutboundWebhook` (called on line 1494) internally generates its own fresh `msgId` (via `crypto.randomUUID()`) at runtime, which is what actually goes into the `webhook-id` header and the HMAC signing string.

These two UUIDs are always different. This breaks the Standard-Webhooks contract: the `webhook-id` that receivers use for idempotency/deduplication has no relationship to the `msgId` stored in `WebhookDelivery`. Anyone inspecting delivery history in the management UI cannot correlate it with the `webhook-id` they received.

The fix is to generate `msgId` once in `dispatchWithTracking` and pass it into `deliverOutboundWebhook` as a parameter, so both the DB record and the HTTP header share the same ID.

```typescript
// In dispatchWithTracking — pass msgId to the delivery function
const msgId = crypto.randomUUID();
const delivery = await prisma.webhookDelivery.create({ data: { ..., msgId } });
const result = await deliverOutboundWebhook(endpoint, payload, msgId);
```

```typescript
// In deliverOutboundWebhook — accept an optional msgId instead of always creating one
export async function deliverOutboundWebhook(
  endpoint: EndpointLike,
  payload: OutboundPayload,
  msgId = crypto.randomUUID(), // default for one-off/test calls
): Promise<OutboundResult> {
```

**Rule Used:** ## Security & Cryptography Review Rules

When revi... ([source](https://app.greptile.com/review/custom-context?memory=7cb20c56-ca6a-40aa-8660-7fa75e6e3db2))

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/server/routers/node-group.ts
Line: 426-461

Comment:
**Cross-team node data exposure via unchecked `groupId`**

`withTeamAccess("VIEWER")` resolves the team check from `input.environmentId`. However, the group is fetched without verifying it belongs to that same environment:

```typescript
const group = await prisma.nodeGroup.findUnique({
  where: { id: groupId },  // no environmentId check
});
// ...
const allNodes = await prisma.vectorNode.findMany({
  where: { environmentId: group.environmentId },  // uses the group's env, not input.environmentId
});
```

An attacker can pass their own valid `environmentId` (to pass the VIEWER check) alongside a `groupId` belonging to a different team's environment. The team auth check passes, but the node query returns data scoped to the other team's `group.environmentId`.

Fix: scope the group lookup to `input.environmentId`:

```typescript
const group = await prisma.nodeGroup.findFirst({
  where: { id: groupId, environmentId: input.environmentId },
});
```

**Rule Used:** ## Security & Cryptography Review Rules

When revi... ([source](https://app.greptile.com/review/custom-context?memory=7cb20c56-ca6a-40aa-8660-7fa75e6e3db2))

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/server/routers/webhook-endpoint.ts
Line: 1123-1152

Comment:
**`testDelivery` mutation is missing `withAudit`**

Every mutation must use `withAudit(action, entityType)` per the project's security policy. `testDelivery` is a mutation that triggers an outbound HTTP request and creates/updates a `WebhookDelivery` record, but it has no audit middleware:

```typescript
testDelivery: protectedProcedure
  .input(z.object({ id: z.string(), teamId: z.string() }))
  .use(withTeamAccess("ADMIN"))
  // ← missing: .use(withAudit("webhookEndpoint.testDelivery", "WebhookEndpoint"))
  .mutation(async ({ input }) => {
```

Add `.use(withAudit("webhookEndpoint.testDelivery", "WebhookEndpoint"))` after `withTeamAccess`.

**Rule Used:** ## Security & Cryptography Review Rules

When revi... ([source](https://app.greptile.com/review/custom-context?memory=7cb20c56-ca6a-40aa-8660-7fa75e6e3db2))

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: src/server/routers/promotion.ts
Line: 784-788

Comment:
**`approve` and `reject` use `EDITOR` role — `withTeamAccess` may not resolve from `requestId`**

Both `approve` and `reject` declare `.use(withTeamAccess("EDITOR"))` with `requestId` as the sole input field. The architecture docs list the fields `withTeamAccess` recognises for team resolution: `teamId`, `environmentId`, `pipelineId`, and `id`. The field here is named `requestId`, not `id`.

If `withTeamAccess` only matches on the exact field name `id`, the middleware won't be able to resolve a team context from `requestId`, which could mean the authorization check silently passes regardless of the caller's actual team membership.

Check the `withTeamAccess` implementation to confirm it handles `requestId` — or rename the input field to `id` for consistency with the convention, or explicitly pass a `teamId` alongside it.

**Rule Used:** ## Code Style & Conventions

### TypeScript Conven... ([source](https://app.greptile.com/review/custom-context?memory=6ae51394-d0b6-4686-bc4c-1ad840c2e310))

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "chore: remove .planning files from track..." | Re-trigger Greptile}