Customized job ID cannot be deduplication

[nightire]

The documentation says that when adding a job to the queue, if the jobId is specified, it can be used for deduplication.
But the problem is that I did this, but I didn't see the deduplication effect at all.

My requirement is to restore a batch of jobs when the service starts; but if these jobss already exist in SQLite, they should be skipped.

I prepared a custom jobId for each job , but jobs with the same ID were inserted into the database repeatedly. So I don't understand how exactly this deduplication works.

In addition, I also saw the design of uniqueKey, dedupe.replace and dedupe.extend in the commit history.
But the problem is that I found that they are not in the public interface and there is no way to access them. Is there any plan to release them as public interface?

[egeominotti]

Hey! Thanks for the detailed report - this was super helpful! 🙏

You're absolutely right, this was a bug. The jobId option was throwing an error on duplicates instead of returning the existing job like BullMQ does.

✅ Fixed!

I've just pushed a fix to main. Now jobId works exactly like BullMQ with idempotent behavior:

const queue = new Queue('my-queue', { embedded: true });

// First call: creates the job
const job1 = await queue.add('process', { orderId: 123 }, { jobId: 'order-123' });

// Second call with same jobId: returns existing job (no error, no duplicate!)
const job2 = await queue.add('process', { orderId: 456 }, { jobId: 'order-123' });

console.log(job1.id === job2.id); // true ✅

Your use case (restore jobs on startup)

This now works exactly as you'd expect:

async function restoreJobsOnStartup(savedJobs: SavedJob[]) {
  for (const job of savedJobs) {
    // Safe: existing jobs are returned, not duplicated
    await queue.add(job.name, job.data, { jobId: job.customId });
  }
}

No try/catch needed, no duplicates, just works! 🎉

---

About uniqueKey and dedup options

Good eye! 👀 These are fully implemented internally but not yet exposed in the public JobOptions API. They provide more advanced deduplication strategies:

Option	Behavior
uniqueKey + dedup.ttl	Key expires after TTL (allows re-insert after expiry)
dedup.extend	Extends TTL on duplicate, returns existing job
dedup.replace	Removes old job, creates new one with same key

For most use cases (including yours), jobId should now be sufficient. I'll consider exposing uniqueKey/dedup in a future release if there's more demand for it - let me know if you need these!

---

📦 npm update coming soon!

I'll push the update to npm shortly. In the meantime, you can:

1. Use directly from GitHub:

   bun add github:egeominotti/bunqueue

2. Or wait for the npm release (coming in the next few hours)

Thanks again for catching this! Let me know if you run into any other issues. 🚀

[egeominotti]

@nightire Can you confirm that it works with the latest version? So I can close the issue. Thanks.

[nightire]

@nightire Can you confirm that it works with the latest version? So I can close the issue. Thanks.

I don't know, because I can not use github:egeominotti/bunqueue in my project. The github repo doesn't contains the build files, so it throws like error: Cannot find module 'bunqueue/client'....

[nightire]

@egeominotti I built the library locally and then went to test it, but found that deduplication still didn't work.
Now whenever it starts, multiple tasks with the same custom_id are still created.

[egeominotti]

@nightire Plase, try to download the latest version from npm. Correct version is 2.0.0

https://www.npmjs.com/package/bunqueue

[nightire]

@egeominotti I reinstalled 2.0 and tested it. Unfortunately, deduplication still not works.

I did a simple debug where I executed the method queue.getDeduplicationJobId before I added the job to the queue. If I understand correctly, when a duplicate task is detected, the method returns the custom_id of the task. But in my testing, this method always returned null. Thus, jobs with the same custom_id will still be added.

  console.debug("DEDUPLICATION ID: ", await queue.getDeduplicationJobId(`rollover-${batchId}`))

  await queue.add(
    "rollover",
    { batchId },
    { delay, jobId: `rollover-${batchId}` },
  )

[egeominotti]

@nightire Thank you for the detailed debugging - you helped me find a real bug!

The Bug

The customIdMap (which tracks jobId for deduplication) was not being populated when recovering jobs from SQLite on server restart.

What was happening:

1. Server starts → jobs loaded from SQLite into memory
2. customIdMap was NOT populated during recovery (BUG)
3. getDeduplicationJobId() returned null
4. queue.add() created duplicates

The Fix

v2.0.1 is now available on npm:

bun add bunqueue@2.0.1

Important: SQLite Persistence Required

For deduplication to work across restarts, you must have SQLite persistence enabled. Make sure you have one of these configured:

Option 1: Environment variable

DATA_PATH=./data/queue.db

Option 2: Connection option

const queue = new Queue('my-queue', {
  embedded: true,
  connection: { path: './data/queue.db' }
});

Without persistence, jobs exist only in memory and are lost on restart - so there's nothing to deduplicate against.

Your Code Should Now Work

// Idempotent - returns existing job if it exists, no duplicates
await queue.add(
  "rollover",
  { batchId },
  { delay, jobId: `rollover-${batchId}` }
);

// Or check explicitly first
const existingId = await queue.getDeduplicationJobId(`rollover-${batchId}`);
console.debug("DEDUPLICATION ID:", existingId); // Now returns job ID if exists!

Verified Working

I wrote a test that simulates your exact use case (restart with delayed jobs):

1. Adding job with jobId...
   Created job: 019c1b91-6261-7000-b64f-0aca0dddd5aa

2. Checking getDeduplicationJobId...
   getDeduplicationJobId: 019c1b91-6261-7000-b64f-0aca0dddd5aa ✅

3. Adding duplicate job...
   Same job? ✅ YES

4. Simulating restart...
   [Storage] Flushed write buffer on close

5. After restart - checking getDeduplicationJobId...
   getDeduplicationJobId: 019c1b91-6261-7000-b64f-0aca0dddd5aa ✅
   Found existing job? ✅ YES

6. Adding job with same jobId after restart...
   Same as original? ✅ YES

7. Final job count: 1 ✅

✅ TEST PASSED!

Please update to v2.0.1 and confirm you have SQLite persistence configured. Let me know if it works!

[nightire]

@egeominotti Good job, now I can confirm it works as I expected, thank you!