feat(api): implement bulk data insertion endpoint with proper validat…#111
feat(api): implement bulk data insertion endpoint with proper validat…#111ayash911 wants to merge 2 commits intogeturbackend:mainfrom
Conversation
…ion and multi-status handling
📝 WalkthroughWalkthroughA new bulk insert endpoint ( Changes
Sequence DiagramsequenceDiagram
actor Client
participant API as Public API
participant Validation as Validator
participant DB as MongoDB
participant Project as Project Model
participant Webhooks as Webhook Dispatcher
Client->>API: POST /bulk (array of items)
API->>Validation: Validate each item schema
Validation-->>API: Per-item validation results
alt Any items fail validation
API->>API: Filter valid items only
end
API->>Project: Check database limit
alt Limit exceeded
API-->>Client: HTTP 403
else Within limit
API->>DB: insertMany(validItems, {ordered: false})
alt Partial/full success
DB-->>API: Insertion results + errors
API->>API: Map bulk errors to original indices
API->>Project: Update databaseUsed
Project-->>API: Acknowledgment
loop For each inserted item
API->>Webhooks: Dispatch webhook
end
API-->>Client: HTTP 207 (partial success)
else All items fail
API-->>Client: HTTP 400
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@apps/public-api/src/controllers/data.controller.js`:
- Around line 160-165: The quota check and the later increment for
project.databaseUsed have a race condition under concurrent bulk writes: instead
of relying on an in-memory check against project.databaseUsed and only
incrementing after insert (variables/functions: project.resources.db.isExternal,
batchDocSize, project.databaseUsed, project.databaseLimit, and the later
increment block), make the reservation atomic by moving the quota enforcement
into the backing store (e.g., perform an atomic conditional update like a DB
transaction or a findOneAndUpdate with $inc and a conditional that ensures
(databaseUsed + batchDocSize) <= databaseLimit), or wrap the insert and usage
increment in a single DB transaction/row lock; if atomic DB updates/transactions
aren’t available, introduce a server-side locking mechanism to serialize quota
checks and increments around this code path.
- Around line 179-195: The controller currently exposes raw Mongo errors and
rethrows raw Error objects; replace those exposures with AppError usage: where
the code references writeErr.errmsg, err.message, and the rethrow (inside the
if/else handling around insertedData, validIndicesMap, errors), map/translate DB
error details to sanitized, user-friendly messages and push new AppError
instances (or convert them via AppError.wrap/construct) into your error flow
instead of including raw errmsg or message, and instead of "throw err" throw a
new AppError with a safe message and appropriate status/code; update handling
around variables insertedData, writeErr, err, validIndicesMap, and errors to
ensure no Mongo internals are returned to the client.
- Around line 119-125: The endpoint currently returns mixed top-level response
shapes (e.g., {error}, {insertedCount}, {errors}, {insertedData}) when
validating incomingDataArray and after processing; change all responses in this
controller that use res.status(...).json(...) to follow the API envelope {
success: boolean, data: {...}, message: string }. For validation failures (e.g.,
when !Array.isArray(incomingDataArray) or incomingDataArray.length === 0 and
other error paths), return res.status(400).json({ success: false, data: {},
message: "Payload must be an array of objects." }) (or the appropriate message).
For success/bulk results (places that currently send
insertedCount/insertedData/errors at top-level), return res.status(200).json({
success: true, data: { insertedCount, insertedData, errors }, message: "Bulk
insert completed" }) and for partial failures set success to false but still
embed details under data. Update every occurrence that references
incomingDataArray validation and the bulk processing response helpers so the
shape is consistent (search for incomingDataArray, insertedCount, insertedData,
errors in this file to apply changes).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e4b43a15-1133-4532-8690-91da03ab40a3
📒 Files selected for processing (3)
apps/public-api/src/__tests__/data.controller.bulk.test.jsapps/public-api/src/controllers/data.controller.jsapps/public-api/src/routes/data.js
|
Will revew this by today. |
|
@coderabbitai Ok to merge?? |
yash-pouranik
added
enhancement
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 23133 🏁 Script executed: Length of output: 812
❌ Open Issues (all Major severity)
✅ What's Good
I'd recommend addressing those 3 issues — particularly the race condition ( |
|
@ayash911 please address those 3 issues and commit. |
|
@ayash911 bro please fix this, THen will merge na |
2 participants
Pull Request Description
Fixes #101
[NSoC'26]
This PR introduces a dedicated bulk operations endpoint (
POST /api/data/:collectionName/bulk) to support high-throughput applications and data migrations. It reduces HTTP round-trips by allowing 100+ document inserts in a single request and correctly enforces project database size quotas mid-flight. Mongoose'sinsertManywithordered: falseensures valid records are preserved even if some records fail due to constraints, responding accurately with a 207 Multi-Status object that identifies failing records by index.Type of Change
Testing & Validation
Backend Verification:
npm testin thebackend/directory and all tests passed.Frontend Verification:
npm run lintin thefrontend/directory.Screenshots / Recordings (Optional)
Checklist
Built with love for urBackend.
Summary by CodeRabbit
New Features
Tests