Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: [SUP-1093] Replaced manual pMap with actual pMap #4604

Closed
wants to merge 1 commit into from

Conversation

dotkas
Copy link
Contributor

@dotkas dotkas commented May 18, 2023

What does this PR do?

A while back we introduced some pseudo-parallelization (to the extend Node allows) of promises when sending requests to our backend. The solution sped up things a bit, but the Promise-chunk would wait until the slowest promise had finished before starting a new batch.

pMap accounts for this problem. The performance increase will not be that significant for smaller chunks of requests, but for bigger ones the lack of holes in the chain of unbroken ought to speed things up a bit.

This change (like the one before) comes on the back of another customer support ticket, where scanning a large project takes over an hour.

The bottlenecks are numerous, but one of the biggest, as far as I have been able to identify, reveals itself around the requests to vulndb.

Performance comparisons

Comparing the performance running locally on an M1 Macbook Pro, against Snyk CLI v. 1.1159.0.

time snyk test --yarn-workspaces --detection-depth=3

Solution Projects Runtime
Previous 72 32m
This one 72 25m

Still quite a way to go to get this down to a few minutes, but I guess it's a start.

@github-actions
Copy link
Contributor

Warnings
⚠️

You've modified files in src/ directory, but haven't updated anything in test folder. Is there something that could be tested?

⚠️

"fix: replaced manual pMap with actual pMap, so not to wait for all promises to finish before moving on" is too long. Keep the first line of your commit message under 72 characters.

Generated by 🚫 dangerJS against d9e9107

@dotkas dotkas closed this May 18, 2023
@dotkas dotkas deleted the dotkas/SUP-1093/performance-improvements branch May 18, 2023 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant