Replies: 1 comment
-
Nobody with any valuable input? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I've a question concerning a service I am working on for a while now. First off all the current situation:
Basically am running a python script via cron which boots up a docker container to run the script every minute. The script basically performs everything:
This works OK and handles some exceptions too but doesn't feel ideal. Sometimes there is a delay which causes tasks to run longer than 1min and cause a backup.
I am trying to increase scalability, make it more robust and also extensable because I'd like to eventually expand upon this with other resources which I would need to periodically check as well. I have always been advised that Celery + Redis/RabbitMQ is a good choice to handle something like this. My experience with those things is exactly 0. So I have been reading up quite a bit on it and going through tutorials, but I still am left behind with some questions on whether I am thinking about it in the right way.
For example how far does one need to cut up tasks? Should I consider my script as one task, or should I start cutting it up in smaller pieces like so?
Or should I perhaps consider one RSS link as one single task or as the "main task" which on its own is cut up in subtasks? But then I imagine I am losing my async in httpx which would increase processing time.
I'm running this on a small Linode VPS but looking at other options like root server from netcup with 2/4 dedicate cores and 8/16GB ram.
Certain aspects are also more time-sensitive than others. For example sending a tweet is considered to be of higher priority while updating the DB can happen in the background. I imagine, given the ability to assign a task to different queues with different priorities, this could be one way to tackle it.
I understand this could be personal preference but given my lack of experience with this particular thing I prefer to ask input from those who do have experience.
Simultaneously I'd also like to move away from pure scripting to more a OOP approach although that will probably be the next step after moving to Celery.
Your input is greatly appreciated!
Beta Was this translation helpful? Give feedback.
All reactions