-
-
Notifications
You must be signed in to change notification settings - Fork 466
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
eventlet concurrency issues #478
Comments
Ignoring the disparaging comments about Eventlet, your problem is probably that you're using librabbitmq rather than pyamqp. The former is a C-library which means it is not automatically monkeypatched into the Eventlet scheduler. If you have it installed kombu will prefer it unless you specify Eventlet and the other greenthreading libraries work just fine and can outperform native Python threads if you work within their constraints. Re: making the |
Restarting the whole universe to add a new planet seems logical. Installed packages in Docker
Config:
Well you should have a look around in Eventlet code, I sed what it does. |
I can't reproduce this. Working The |
Provided example project as a comment in your example I wanted to do, but was forced to use another approach. |
The code you've added to the gist isn't valid for several reasons. The reason that threads aren't working for you is that you've re-implemented the runner but omitted the eventlet monkey patch. From the docs:
Since you've not using |
Well if you would be so kind and point me where this so called 'monkey patching' magic happens in the runner? Your runner starts first service and waits when it receives rpc call and repeats for next service, till all of them are 'loaded.' |
It's here. To understand what it's doing you should read eventlet docs, specifically the bit on patching. The Nameko service running machinery is spread across three locations:
There is configuration-level support for using a custom |
Hey!
Good library, but you need to throw out eventlet and replace it with Python threads. Eventlet is somewhat useless library, sorry author, but implementing 'threading' without using low level system calls, won’t make it concurrent. You’re adding extra layer of complexity on top of Python processes main thread execution. Wrapping functions inside a object and calling them after one has finished, will halt when client uses external library which won’t return pointer back to hub. Library would be useful if you had done all the lifting in C or equivalent and made python bindings to it, like threading library is doing, yes there are convenience methods missing, but this should be place for your library. Today’s concurrency comes from low level timers; when triggered, current CPU state is saved to registers and depending on priority, next state is loaded and executed. Finally, we have discovered real bottleneck - extra complexity at low level which improved may change many things.
When spawning a new container, code will halt at
start()
andprint
command won’t be called at all ifkombu.mixins.run
waits for consumer atrestart_limit.can_consume(_tokens)
kombu.mixins.run
Here is log showing what happens. Thread ID is same for spawned kombu thread which locks up execution.
Problem relies in
messages.py
linekombu expects a thread which has its own process life cycle.
Some notes.
ServiceRunner currenty supports static service array, but in real world everything changes (summer to winter etc), runner should support adding and removing services at runtime.
ServiceContainer(config={}) argument should be optional.
Python was called scripting language, as you write instructions for virtual machine - nothing has changed. You should learn python and move to harder more complex things like C > asm and make improvements at lower level. As a hobby extend and help to improve your preferred language knowing how it runs (to eventlet author).
The text was updated successfully, but these errors were encountered: