Imports with multiple queue workers: chunk jobs spin on the WithoutOverlapping lock and can be lost

[wout-o]

With two or more queue workers, import chunk jobs busy-wait on each other. Importer::getJobMiddleware() uses WithoutOverlapping without releaseAfter(), so a worker that cannot take the import lock releases the chunk with a zero-second delay and pops it again about 40 ms later. Every release counts as an attempt, --tries is ignored while the 24-hour retryUntil is in the future, and a release throws nothing, so $maxExceptions never applies.

On the database driver the attempts column eventually overflows (SQLSTATE[22003] ... Out of range value for column 'attempts', after 255 waits on a tinyint migration, 65535 on the current stub). Laravel then moves the chunk to failed_jobs, and because the batch allows failures, the finally callback still sends the "import completed" notification with that chunk's rows missing. We hit this in production the day we went from one worker to two: 3 chunks, one lost after about ten seconds, success notification sent. It is the same mechanism as #16930, and the docs say the import system "will only process one job at a time from each import".

I can see a few ways to address this, each with its own cost, and would value your opinion on them.

1. Chain the chunks inside the batch: Bus::batch([$importJobs->all()]). Chunk N+1 is only pushed after chunk N released the lock, so the lock is never contended and every chunk costs exactly one attempt, whatever the worker count. Costs: a chained job carries the payloads of all later chunks, so the first message holds the whole CSV, total queue writes grow by about (chunks + 1) / 2, and SQS's 256 KB message limit makes it unusable there. The sync driver would have to keep the flat batch, since it runs a chain as nested calls (1000 chunks peaked at 4.5 GB in my measurement). A permanently failed chunk also stops the chain: a visible stall instead of a silent success notification.

2. ->releaseAfter() on the two WithoutOverlapping instances. The remedy the Laravel docs name. Keeps the flat batch and small messages, but every wait still costs an attempt, so a tail chunk of a long import still walks towards the column cap, and idle workers keep polling the waiting chunks.

3. Something else you would rather see

Which direction would you prefer? Let me know if you want my help in setting up a PR to implement the approach you choose.

[danharrin]

Thanks for the detailed write-up. We reproduced the attempts overflow with two workers and a tinyint attempts column.

We’ve opened #20955 to add a five-second release delay to both overlap locks. This reduces rapid retries while preserving the existing batch structure and the ability to enable parallel processing by removing the middleware.

This is a mitigation rather than a complete fix: lock releases still consume attempts. For long imports using a database queue, we recommend widening the attempts column to an unsigned integer. SQS users also need to account for these deliveries in the source queue’s redrive policy.

One distinction from our reproduction: the completion notification was a warning, not a success, and counted the unprocessed rows as failures. However, those rows had no failure-CSV records because they never reached row processing.

We’re not taking the chaining approach because of its payload growth and different behavior after a chunk permanently fails.