Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: concurrent resource fetching per provider #1065

Merged
merged 7 commits into from
Oct 16, 2023
Merged

feat: concurrent resource fetching per provider #1065

merged 7 commits into from
Oct 16, 2023

Conversation

titanventura
Copy link
Contributor

Problem

Template to fetch resources concurrently using worker pools. Implements concurrency for aws fetcher.
Fixes #1014 partially

Changes Made

  • Added worker pool, make use of the worker pool in AWS

Notes

Perf test needed

Checklist

Reviewers

@Azanul @AvineshTripathi @mlabouardy @jakepage91

@titanventura titanventura changed the title Concurrent resource fetching per provider feat: concurrent resource fetching per provider Oct 10, 2023
@AvineshTripathi AvineshTripathi mentioned this pull request Oct 10, 2023
Closed
mlabouardy
mlabouardy reviewed Oct 15, 2023
View reviewed changes
Copy link
Collaborator

@mlabouardy mlabouardy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any update on the performance benchmark @Azanul ?

@mlabouardy mlabouardy added the aws label Oct 15, 2023
@mlabouardy mlabouardy added this to the v3.1.2 milestone Oct 15, 2023
@Azanul
refac(fetchResources): encapsulate
ce53d68 
Signed-off-by: Azanul <azanulhaque@gmail.com>
@Azanul
fear(fetchResources): benchmark test
1d46d1d 
Signed-off-by: Azanul <azanulhaque@gmail.com>
@Azanul
Copy link
Collaborator

Azanul commented Oct 15, 2023

Before: f2326e9
Before Profile CPU.pb.gz
Before Profile Space.pb.gz

Screenshot 2023-10-15 at 8 31 26 PM

After: 1d46d1d
After Profile CPU.pb.gz
After Profile Space.pb.gz

Screenshot 2023-10-15 at 9 32 16 PM

Tested on M2 Pro with no CPU limit and 10GB RAM limit

@mlabouardy mlabouardy self-requested a review October 16, 2023 11:24
@mlabouardy mlabouardy merged commit ee52e42 into tailwarden:develop Oct 16, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mlabouardy mlabouardy mlabouardy approved these changes

@Azanul Azanul Azanul approved these changes

@ShubhamPalriwala ShubhamPalriwala Awaiting requested review from ShubhamPalriwala

@AvineshTripathi AvineshTripathi Awaiting requested review from AvineshTripathi AvineshTripathi is a code owner

@greghub greghub Awaiting requested review from greghub greghub is a code owner

@Traxmaxx Traxmaxx Awaiting requested review from Traxmaxx

Assignees
No one assigned
Labels
aws
Projects
None yet
Milestone
v3.1.2
Development

Successfully merging this pull request may close these issues.

Fetch resources concurrently using goroutines
3 participants
@titanventura @Azanul @mlabouardy