feat: Add Bulk Ingest API Endpoint for Post Groups with Posts #24
Conversation
…on and database integration
WalkthroughAdds a Next.js POST API route at Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant C as Client
participant R as Next.js Route\nPOST /api/post-groups/ingest
participant V as Zod Validator
participant DB as Prisma / Database
C->>R: POST body (groups) + x-ingest-secret
R->>R: Check process.env.INGEST_SECRET
alt Missing/invalid secret
R-->>C: 401 Unauthorized
else Secret valid
R->>R: Parse JSON
alt Parse error
R-->>C: 400 Bad Request
else Parsed
R->>V: Validate groups/posts schema
alt Validation fail
R-->>C: 400 Validation details
else Valid
loop For each group
R->>DB: create PostGroup (title, summaries, totalposts)
alt posts exist
R->>DB: createMany(posts mapped, dates normalized)
end
R->>DB: fetch created PostGroup with posts
R-->>R: append { success: true, postGroup }
end
R-->>C: 200 Results array
end
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (7)
src/app/api/post-groups/ingest/route.ts (7)
1-1: Harden secret check (timing‑safe compare) and treat missing secret as 500.Prevents subtle timing leaks and surfaces server misconfiguration distinctly from client auth failures.
+import { timingSafeEqual } from "node:crypto"; @@ -const INGEST_SECRET = process.env.INGEST_SECRET; +// (read per request below to avoid stale values across reloads) @@ - if (!INGEST_SECRET || authHeader !== INGEST_SECRET) { - return NextResponse.json({ error: "Unauthorized" }, { status: 401 }); - } + const secret = process.env.INGEST_SECRET; + const provided = authHeader ?? ""; + if (!secret) { + return NextResponse.json({ error: "Server misconfigured: INGEST_SECRET not set" }, { status: 500 }); + } + const a = Buffer.from(provided); + const b = Buffer.from(secret); + const ok = a.length === b.length && timingSafeEqual(a, b); + if (!ok) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 }); + }Also applies to: 6-6, 9-13
24-33: Tighten validation (min/url/Date coercion), default posts to [], and drop redundant array check.Avoids empty titles, validates URLs, ensures valid Dates before DB, and removes the manual non‑empty array guard in favor of schema min(1).
@@ - const postSchema = z.object({ - content: z.string(), + const postSchema = z.object({ + content: z.string().trim().min(1), sentiment: z.enum(["BULLISH", "NEUTRAL", "BEARISH"]), source: z.enum(["REDDIT", "TWITTER", "YOUTUBE", "TELEGRAM", "FARCASTER"]), - categories: z.array(z.string()), - subcategories: z.array(z.string()), - link: z.string().optional(), - createdAt: z.string().datetime().optional(), - updatedAt: z.string().datetime().optional(), + categories: z.array(z.string()).default([]), + subcategories: z.array(z.string()).default([]), + link: z.string().url().optional(), + createdAt: z.coerce.date().optional(), + updatedAt: z.coerce.date().optional(), }); @@ - const postGroupSchema = z.object({ - title: z.string(), + const postGroupSchema = z.object({ + title: z.string().trim().min(1), bullishSummary: z.string().optional(), bearishSummary: z.string().optional(), neutralSummary: z.string().optional(), - posts: z.array(postSchema).optional(), + posts: z.array(postSchema).default([]), }); @@ - const postGroupsSchema = z.array(postGroupSchema); + const postGroupsSchema = z.array(postGroupSchema).min(1); @@ - // Accept an array of post groups - if (!Array.isArray(data) || data.length === 0) { - return NextResponse.json({ error: "Request body must be a non-empty array of post groups" }, { status: 400 }); - } + // (Redundant after schema .min(1)) @@ - data: group.posts.map((post: { + data: group.posts.map((post: { content: string; sentiment: "BULLISH" | "NEUTRAL" | "BEARISH"; source: "REDDIT" | "TWITTER" | "YOUTUBE" | "TELEGRAM" | "FARCASTER"; categories: string[]; subcategories: string[]; link?: string; - createdAt?: string; - updatedAt?: string; + createdAt?: Date; + updatedAt?: Date; }) => ({ @@ - createdAt: post.createdAt ? new Date(post.createdAt) : undefined, - updatedAt: post.updatedAt ? new Date(post.updatedAt) : undefined, + createdAt: post.createdAt, + updatedAt: post.updatedAt, })),Also applies to: 35-43, 51-55, 78-87, 95-96
4-6: Pin runtime to Node.js to avoid accidental Edge deployment.Prisma requires Node APIs; this prevents accidental Edge runtime selection.
+ +export const runtime = "nodejs";
24-33: Optional: derive enums from Prisma to keep Zod in lockstep with DB.Prevents drift if enum members change in the schema.
// import { Sentiment, Source } from "@prisma/client"; // const postSchema = z.object({ // ... // sentiment: z.nativeEnum(Sentiment), // source: z.nativeEnum(Source), // ... // });
76-99: Optional: chunk large createMany batches to avoid exceeding parameter limits.For very large groups, chunk posts (e.g., 500–1000) to keep queries safe.
// helper async function createPostsInBatches(tx: typeof prisma, posts: any[], postGroupId: string, batch = 500) { for (let i = 0; i < posts.length; i += batch) { const slice = posts.slice(i, i + batch).map(p => ({ ...p, postGroupId })); await tx.post.createMany({ data: slice }); } }
8-13: Operational: add rate limiting and idempotency for external callers.Reduce abuse and duplicate ingestion with:
- Rate limit keyed by IP or service token.
- Optional Idempotency-Key header + unique constraint (e.g., on link) with skipDuplicates.
68-68: Prisma field 'totalposts' (lowercase) matches the code; consider renaming to camelCase.schema.prisma — PostGroup (line 55):
totalposts Int @default(0)
src/app/api/post-groups/ingest/route.ts (line 68):totalposts: Array.isArray(group.posts) ? group.posts.length : 0No runtime error expected; to rename to
totalPostsrun a Prisma migration and update all usages.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/app/api/post-groups/ingest/route.ts(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
src/app/api/post-groups/ingest/route.ts (1)
src/lib/prisma.ts (1)
prisma(13-14)
🔇 Additional comments (1)
src/app/api/post-groups/ingest/route.ts (1)
8-14: LGTM: route skeleton and early auth gate are clean.Good use of App Router POST handler, early return on auth failure, and JSON parse guard.
Description:
/api/post-groups/ingestthat accepts an array of post groups, each with nested posts.INGEST_SECRET) to prevent unauthorized access.Summary by CodeRabbit