Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch job queuing? #123

Closed
vincentjames501 opened this issue May 30, 2017 · 1 comment
Closed

Batch job queuing? #123

vincentjames501 opened this issue May 30, 2017 · 1 comment

Comments

@vincentjames501
Copy link

It would be nice to add to the Client interface the ability to batch queue jobs. In our case we need to batch queue around 40-100k jobs and it would be nice to do it in a single pipeline w/ Jedis.

void enqueue(String queue, List<Job> jobs); // Or Job...
@wbijen
Copy link

wbijen commented Jan 19, 2018

This works!
Here is a link to my repo: https://github.com/wbijen/jesque

public static void doBulkEnqueue(Jedis jedis, String namespace, String queue, List<String> jobJsonList, int batchSize) {
        jedis.sadd(JesqueUtils.createKey(namespace, new String[]{"queues"}), new String[]{queue});
        Pipeline pipeline = jedis.pipelined();
        String key = JesqueUtils.createKey(namespace, new String[]{"queue", queue});
        int i = 0;

        for (String jobJson : jobJsonList) {
            String jobJson = it.next();
            jedis.rpush(key, new String[]{jobJson});
            ++i;
            if (i == batchSize) {
                pipeline.sync();
                pipeline = jedis.pipelined();
                i = 0;
            }
        }
        if (i > 0) {
            pipeline.sync();
        }
    }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants