Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job.getDependencies very bad performance #1323

Closed
jmaver-plume opened this issue Jul 15, 2022 · 1 comment
Closed

Job.getDependencies very bad performance #1323

jmaver-plume opened this issue Jul 15, 2022 · 1 comment
Labels

Comments

@jmaver-plume
Copy link

jmaver-plume commented Jul 15, 2022

Problem:
Job.getDependencies performance is really bad to the point of unusable.

For my personal project at 40000 Child jobs generated using FlowProducer it takes more than 5 minutes on a strong personal computer.

The root cause is this [code snippet]. Destructuring is very very slow. (

bullmq/src/classes/job.ts

Lines 672 to 677 in 907ae1d

const transformedProcessed = Object.entries(processed).reduce(
(accumulator, [key, value]) => {
return { ...accumulator, [key]: JSON.parse(value) };
},
{},
);
)

An easy fix is to mutate accumulator object instead of doing destructuring.

const transformedProcessed = Object.entries(processed).reduce(
  (accumulator, [key, value]) => {
    accumulator[key] = JSON.parse(value);
    return accumulator;
  },
  {},
);

Steps to reproduce:

  1. Create Flow using FlowProducer with large amount of children (e.g. 40000).
  2. Process all those rows.
  3. Use job.getDependencies using the parent job of Flow Producer.
@github-actions
Copy link

🎉 This issue has been resolved in version 1.86.8 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant