Queue Abstractions on Effect Cluster

[afonsomatos]

What is the problem this feature would solve?

Effect Cluster already has the primitives necessary to build a distributed queue system with:

• Multi Level Concurrency
• Prioritisation

The only viable option for production in the NodeJS ecosystem is BullMQ. I want to use Effect!

So that I can replace this with our beautiful Effect:

import { Queue } from 'bullmq';

const myQueue = new Queue('Paint');

await myQueue.add('wall', { color: 'pink' }, { priority: 10 });
await myQueue.add('wall', { color: 'brown' }, { priority: 5 });
await myQueue.add('wall', { color: 'blue' }, { priority: 7 });

import { WorkerPro } from '@taskforcesh/bullmq-pro';

const worker = new WorkerPro('myQueue', processFn, {
  group: {
    concurrency: 3, // Limit to max 3 parallel jobs per group
  },
  concurrency: 100,
  connection,
});

So, just like we have ClusterCron, ClusterQueue is needed.

What is the feature you are proposing to solve the problem?

Primitives to create queues based on effect/cluster.

What alternatives have you considered?

BullMQ which is horrible.
Doing it myself which is painful, as I'm not as skilled as Tim Smart.

[luizhcastro]

+1