fix: dedup uniqueKey entries in Node enqueueMany

[stromanni]

Fixes #520.

Problem

enqueueMany sent every job straight to Storage::enqueue_batch, a raw
multi-row INSERT with no dedup check. The partial unique index
idx_jobs_unique_key (unique_key IS NOT NULL AND status IN (pending, running))
then rejected the insert, so a colliding uniqueKey did not silently duplicate —
it failed the whole batch with a raw storage error:

storage error: UNIQUE constraint failed: jobs.unique_key

Single enqueue has always deduped, and Java's batch path did too, so this was
also a cross-SDK behavioural gap.

Fix

New shared helper taskito_core::storage::enqueue_batch_dedup. It splits a
batch by keyed-ness, preserving input order: entries with a unique_key go
through enqueue_unique_batch (one transaction, returning the existing active
job on a collision), entries without one keep the plain enqueue_batch fast
path — so a batch that doesn't dedupe costs exactly what it did before.

Java carried its own copy of this split, doing one enqueue_unique transaction
per keyed job; it now delegates to the core helper and gets the single-transaction
keyed path for free.

Tests

• test_enqueue_batch_dedup in the backend-agnostic storage suite: mixed batch
  covering a collision with an already-active job, a key repeated inside the
  batch, and keyless rows.
• Two Node tests in test/core/batch.test.ts: dedup with input-order ids, and
  job.enqueued emitted once per entry (matching single enqueue).

Docs

Five pages claimed enqueueMany applies no uniqueKey dedup. The two Java
pages carried the same claim and were already wrong before this change.

Note for a follow-up

Python has the same defect. Queue.enqueue_many resolves a per-row unique key
and then hands the batch to plain enqueue_batch, so a duplicate raises
RuntimeError: storage error: UNIQUE constraint failed: jobs.unique_key.
Issue #520 lists Python as already correct; it is not. The Python docs describe
the current behaviour accurately, so this is a deliberate-looking gap rather
than a silent one — left out of this PR, but the new core helper reduces the fix
to routing PyQueue::enqueue_batch through it.

Summary by CodeRabbit

• New Features

  • enqueueMany now deduplicates entries with matching uniqueKey values, including duplicates already pending or running.
  • Returned job IDs preserve the original input order, with duplicates resolving to the existing job.
  • Jobs without a uniqueKey continue using efficient bulk insertion.
  • Enqueue events remain emitted for each input entry.

• Documentation

  • Updated Node and Java API references and reliability guides to describe batch deduplication and idempotent retry behavior.

[coderabbitai]

Warning

Review limit reached

@stromanni, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 38 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 57d630ed-eacb-4669-996f-0891bf5143be

📥 Commits

Reviewing files that changed from the base of the PR and between d454d3a and 8ce120a.

📒 Files selected for processing (8)

• crates/taskito-core/src/storage/mod.rs
• crates/taskito-core/src/storage/sqlite/tests.rs
• crates/taskito-core/tests/rust/storage_tests.rs
• crates/taskito-node/src/queue/mod.rs
• docs/content/docs/java/guides/core/enqueue-options.mdx
• docs/content/docs/node/guides/core/enqueue-options.mdx
• docs/content/docs/shared/guides/reliability/idempotency.mdx
• sdks/node/src/queue.ts

📝 Walkthrough

Walkthrough

Batch enqueue now deduplicates jobs with uniqueKey, preserves input-order IDs, retains bulk insertion for keyless jobs, and applies the shared behavior across core storage, Java, and Node integrations. Tests and documentation cover collisions, intra-batch duplicates, events, and queue counts.

Changes

Batch deduplication

Layer / File(s)	Summary
Core batch deduplication helper crates/taskito-core/src/storage/mod.rs, crates/taskito-core/tests/rust/storage_tests.rs	Adds shared keyed/unkeyed batch routing, result reassembly, backend size validation, and storage contract coverage.
SDK batch enqueue integration crates/taskito-java/src/backend.rs, crates/taskito-node/src/queue/mod.rs, sdks/node/test/core/batch.test.ts	Routes Java and Node batch enqueue through the shared helper and tests returned IDs, pending counts, and emitted events.
Batch deduplication documentation docs/content/docs/{java,node,shared}/..., sdks/node/src/queue.ts	Documents unique-key deduplication, input-order results, and the keyless bulk path across API references and examples.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant NodeQueue
  participant enqueue_batch_dedup
  participant StorageBackend
  NodeQueue->>enqueue_batch_dedup: enqueueMany batch
  enqueue_batch_dedup->>StorageBackend: enqueue_unique_batch keyed entries
  enqueue_batch_dedup->>StorageBackend: enqueue_batch keyless entries
  StorageBackend-->>enqueue_batch_dedup: return Job results
  enqueue_batch_dedup-->>NodeQueue: return IDs in input order

Loading

Suggested reviewers: pratyush618

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding uniqueKey deduplication to Node enqueueMany.
Linked Issues check	✅ Passed	The PR fixes the reported gap by making Node enqueueMany deduplicate uniqueKey entries and preserve input-order IDs.
Out of Scope Changes check	✅ Passed	The changes stay within the deduplication work and its supporting tests, Java delegation, and documentation updates.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/taskito-core/src/storage/mod.rs`:
- Around line 141-146: Update the documentation comment above
Storage::enqueue_unique_batch to limit the SDK routing claim to the Java and
Node SDKs. Keep the descriptions of keyed and unkeyed entries and the partial
unique index behavior unchanged, while explicitly excluding Python from the
claim.
- Around line 163-172: Make enqueueMany’s keyed and plain partition handling
atomic: replace the separate enqueue_unique_batch and enqueue_batch commits in
the mixed-batch path with one backend transaction or transactional mixed-batch
primitive that rolls back both partitions if either insert fails. Preserve
scatter’s created-job mapping, and add a regression test proving a plain-route
failure leaves keyed jobs unpersisted.

In `@docs/content/docs/java/api-reference/queue/index.mdx`:
- Line 53: Update the enqueueMany API reference entry to describe mixed batches
as batched enqueue without promising a single storage call. Retain the
guarantees about returned IDs preserving input order and uniqueKey entries being
deduplicated.

In `@sdks/node/src/queue.ts`:
- Around line 553-555: Update the documentation for the bulk enqueue method near
the typed args/options description to say it returns job IDs in input order,
removing the inaccurate “new” qualifier while preserving the existing uniqueKey
deduplication behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d2bf1fac-90ed-449e-8fb6-fa84dce91b4f

📥 Commits

Reviewing files that changed from the base of the PR and between 35ec678 and d454d3a.

📒 Files selected for processing (12)

• crates/taskito-core/src/storage/mod.rs
• crates/taskito-core/tests/rust/storage_tests.rs
• crates/taskito-java/src/backend.rs
• crates/taskito-node/src/queue/mod.rs
• docs/content/docs/java/api-reference/queue/index.mdx
• docs/content/docs/java/more/examples/bulk-emails.mdx
• docs/content/docs/node/api-reference/queue/index.mdx
• docs/content/docs/node/guides/core/enqueue-options.mdx
• docs/content/docs/node/more/examples/bulk-emails.mdx
• docs/content/docs/shared/guides/reliability/idempotency.mdx
• sdks/node/src/queue.ts
• sdks/node/test/core/batch.test.ts

[stromanni]

CI was red on Rust Tests (Redis): test_enqueue_batch_dedup_is_atomic failed with left: 1, right: 0.

My fault, and it exposed a real overclaim in the atomicity fix. Redis's enqueue_unique_batch is a documented per-row loop over enqueue_unique — each op commits on its own, so a mixed batch is only all-or-nothing on the Diesel backends, not Redis. The doc comment claimed it unconditionally.

Fixed in 8ce120a:

• Doc comment now scopes the guarantee: one transaction on SQLite/Postgres, per-row on Redis where a mid-batch failure leaves earlier rows enqueued.
• The shared cross-backend test drops the rollback assertion and keeps what every backend does guarantee — dependency validation across a mixed batch, including the keyless rows the raw enqueue_batch path inserted unchecked. Renamed test_enqueue_batch_dedup_validates_deps.
• The rollback assertion moved to test_enqueue_batch_dedup_is_atomic in the SQLite unit tests, matching how the suite already handles backend-specific behaviour (see the Lifo dispatch-order test's note about Redis being a FIFO fallback).
• Dropped the all-or-nothing wording from the user-facing idempotency page; it now says one storage call, no backend claim.

Verified against a local redis:7-alpine container, which reproduced the CI failure before the change and passes after: cargo test --workspace --features redis green, plus default and --features postgres, clippy and fmt clean.