Replies: 5 comments
-
Hey @ShahBinoy 👋, We also offer priority support for our sponsors. |
Beta Was this translation helpful? Give feedback.
-
The way I've handled this in the past is to use different queues for the tasks and then different workers (with different prefetch settings configured) for each queue. |
Beta Was this translation helpful? Give feedback.
-
That sounds like a good approach @clokep , my only concern is, because the monkey_patch is happening globally, it would be limiting my worker to 1 concurrency and that would limit the number of concurrent boto3 calls that task invokes. |
Beta Was this translation helpful? Give feedback.
-
I'm not sure what monkey patch you're talking about? In my case I was running two completely separate processes for the workers, they had different configurations (I think passed on the command line, but it might have been different settings files). |
Beta Was this translation helpful? Give feedback.
-
Sorry, i might have been mixing concerns. First part of problem is using gevent/eventlet to use "green threads". Here we have a potential to use a large pool of green threads to fetch concurrent S3 objects. But this concurrency value, which is controlled at command line parameter I want to be able to use high gevent pool but reserve only one task at a time. |
Beta Was this translation helpful? Give feedback.
-
Use case Scenario: Process Millions of files from S3
Approach:
** Split small chunks (Task)
** Each chunk, read n(1000s) of files and process it (Long running Task)
My issue is with the
int
math of pre_fetch_count = concurrency * prefetch limit where I have to only one option, if my concurrency increases, so will my prefetch limit.Is there a way to directly affect the number of Prefetch count after the math ? or is there a workaround ?
I want to use Gevent/Eventlet for fetching S3 objects but do not want to pre-fetch more than 1 task at the time.
Checklist
for similar or identical bug reports.
for existing proposed fixes.
to find out if the bug was already fixed in the master branch.
(If there are none, check this box anyway).
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Description
Suggestions
Beta Was this translation helpful? Give feedback.
All reactions