-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Differences between quamash and asyncio event loop + processEvents #66
Comments
Primarily performance, but since you're in python YMMV a lot with regards to how much that matters. With quamash you're only running one event loop, and plugging the python functions into it. with this you're running two event loops essentially. With quamash you get to use Qts event loop for socket handling which is potentially faster. (there's a re-implementation of the base event loop in C that's faster, I assume using Qts file-descriptor/socket handling is similar). There's also potentially compatibility concerns if some parts of your code do things the Qt way and some do it using builtin python functions. Then again if everything is done the "python way" in your code instead of the "Qt way" then you probably wouldn't have any problems. I suppose performance would be a stronger motivator if I could like show charts showing what was faster. |
So I looked at the performances impact of Quamash compared to asyncio. To do this, I used aiohttp benchmarks ( https://github.com/KeepSafe/aiohttp/tree/master/benchmark ) and added a test using Quamash event loops. The results do not look good at all. Maybe the overhead caused by PyQt5 is causing this much performance problem ?
|
Yeah that's not great. It'd be good to get profile information out, to see why it's running slowly. |
ok, so most time is spent here: https://github.com/harvimt/quamash/blob/master/quamash/__init__.py#L187 and here: https://github.com/harvimt/quamash/blob/master/quamash/__init__.py#L332-L342 somewhat unsurprising. possible improvements:
|
I did most of the easy fix without much change :
If we compare the two results, it seems that the overhead is mostly caused by the Why do |
Ok so I did a fast implementation of the option ( https://github.com/Insoleet/quamash/tree/perfs ) :
This implementation breaks It looks a bit better but that still not as good as standard asyncio :
Any idea maybe ? |
Qt's IO subsystem just isn't that fast? Context switches between Qt & Python are so slow it doesn't matter? I think that's about as performant as we can get without using numba or cython. The question about handles, is I think that code changed quite a bit in terms of fixing bugs. I wanted to just return the handle, but I needed to wrap the handle in another handle. I made _SimpleTimer a subclass of asyncio.Handle at one point, but couldn't have it be a subclass of both QObject and Handle. I think the PEP only requires that the returned object support a |
Ok I understand. I think 30% in terms of performances gains is still really interesting. I'd love to fix the errors with subprocesses stuff. I need some help because I don't understand some things. For example, why do this test ends even if there is a |
ugh that's the low-level test. I only wrote the low-level tests because the this code is what stops the loop when run_forever is called: On Sat, Sep 10, 2016 at 10:28 AM, Insoleet notifications@github.com wrote:
|
@Insoleet please provide an example code where you don't need to kill an app, but it quits when closing the main window. Thanks |
So, after reading again my code, I found out that the mistake was rather basic. I needed to implement a method called Well, I suggest my pull request which is still a gain of 33% speed. @kamikaze : see #65 (comment) |
@harvimt I also have the same question as indicated in this issue (I admit I am not so great with
Can you elaborate on what you mean by "the differences between the python way and Qt way" (possibly w/ a code snippet)? Is this referring to how to send events back to the Qt event loop (and define callbacks) versus "how to do the same w/ python's |
no. basically I mean if there's a Qt Library for something IO-related (like say QNetworkRequest) vs a python-native module that doesn't depend on Qt (like say aiohttp), you should use the python-native one. |
that would extend to python built-in functions as well, like local file access, and direct network sockets. |
A major difference I didn't see mentioned here is quamash only working for the main application event loop, i.e. QApplication's main/UI thread. Meanwhile the "let's put an event loop in an event loop so you can loop while you loop"-approach actually works for arbitrary QThreads. Sample code: class ...(QObject):
@pyqtSlot()
def run(self):
# Connected to QThread.started; thus will be called in the new thread.
asyncio.set_event_loop(asyncio.new_event_loop())
self.evloop = asyncio.get_event_loop()
self.evloop.run_until_complete(self.run_event_loop())
async def run_event_loop(self):
"""
Runs the event loop. Will only complete if the thread receives an interruption request
(QThread.requestInterruption()).
Since we are using asyncio only for coordination, the loop will block for Qt events.
"""
thread = self.thread()
while not thread.isInterruptionRequested():
await asyncio.sleep(0)
thread.eventDispatcher().processEvents(QEventLoop.AllEvents|QEventLoop.WaitForMoreEvents) The sequence to get such a thread to join is thus as follows: thread.requestInteruption() # Set interruption request flag
thread.eventDispatcher().wakeup() # Causes processEvents() to return even if no real events arrive
thread.quit() # Causes event loop to quit (the one running in QThread.run, not run_event_loop())
thread.join() # Thread will exit now and can be joined. I am using this in the context of QNetworkAccessManager, where I have the ability to retry requests and such things, but I also wanted to avoid the signal/callback-oriented approach, since that tends to get messy fast. Thus, the actual events driving this and completing futures is the QNetworkReply status slots explicitly setting the results on futures previously created. The event loop of asyncio is only around to resume coroutines as the futures they're waiting on enter done status. |
The process_events coroutine at the start of this thread has the slight drawback that it doesn't handle DefferedDelete events, as mentioned in the Qt docs. I noticed this when using QtCharts, with the charts not updating properly. It can be remedied by using async def process_events():
qloop = qt.QEventLoop()
timer = qt.QTimer()
timer.timeout.connect(qloop.quit)
while True:
timer.start(0)
qloop.exec_()
timer.stop()
await asyncio.sleep(0.001) |
The same applies to QAbstractEventDispatcher::processEvents as well. Meanwhile, I wrote Asynker which for my use case completely removes the asyncio dependency (and thus its concurrent.futures and multiprocessing dependencies and some other modules, all of which add up to 1+ MB of bloat after compression if you are building a standalone app); it includes some examples showing how to mix Qt and async; they should apply, for the most part, to asyncio as well. |
It turns out that the loop in my previous post has the drawback of not firing QApplication events, such as aboutToQuit and lastWindowClosed. What works very well for me as an alternative to quamash is def run():
def onTimer():
loop.call_soon(loop.stop)
loop.run_forever()
loop = asyncio.get_event_loop()
timer = qt.QTimer()
timer.start(10)
timer.timeout.connect(onTimer)
qt.QApplication.instance().exec_() Since this is in essence a braindead polling loop there is a compromise between idle efficiency and latency. The reason for not using quamash is that on slow CPUs (cheap Celeron notebooks) it appears that there exists a point were it generates more events than can be handled. When the application ventures beyond this event horizon it freezes with 100% CPU utilization. According to the docs it should be possible to use QAbstractEventDispatcher for a more efficient event loop integration but it's completely unclear to me how. There's a new gevent project that uses this approach. |
Asynker is interesting... it's gives me most of what I wanted out of
quamash... though not subprocess support. (unless I put the subprocess in a
thread?)
QAbstractEventDispatcher would allow you use a different event loop besides
the Qt Event Loop (possibly say the one built into python's asyncio module,
or curio, or whatever) and mostly bypass Qt's built in event loop, looks
like you can do other things as well, but you can say reimplement
registerSocketNotifier if you want.
This is the reverse implementation of Quamash which is a asyncio event loop
API wrapped around the Qt Event Loop.
Though the reverse approach is interesting (if it was done in python it
would mean all events would flow through the interpreter before being
processed).
…On Sun, Sep 9, 2018 at 5:11 AM Ewald de Wit ***@***.***> wrote:
It turns out that the loop in my previous post has the drawback of not
firing QApplication events, such as aboutToQuit and lastWindowClosed. What
works very well for me as an alternative to quamash is
def run():
def onTimer():
loop.call_soon(loop.stop)
loop.run_forever()
loop = asyncio.get_event_loop()
timer = qt.QTimer()
timer.start(10)
timer.timeout.connect(onTimer)
qt.QApplication.instance().exec_()
Since this is in essence a braindead polling loop there is a compromise
between idle efficiency and latency.
The reason for not using quamash is that on slow CPUs (cheap Celeron
notebooks) it appears that there exists a point were it generates more
events than can be handled. When the application ventures beyond this event
horizon it freezes with 100% CPU utilization.
According to the docs it should be possible to use
QAbstractEventDispatcher for a more efficient event loop integration but
it's completely unclear to me how. There's a new gevent project
<https://gitlab.esrf.fr/bliss/qgevent> that uses this approach.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#66 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAHRQpvqzTIueXo3jhwjZedymC-Iyc5vks5uZQVigaJpZM4J0jBs>
.
|
Subprocess handling should be fairly straightforward using QProcess (which is already asynchronous, so only a convenience adapter would be needed[1])... the asyncio.subprocess_exec API looks quite complicated to me. [1] Care should be taken to do error handling properly. I'm thinking along the lines of e.g. def start(self) -> Future:
def started(): future.set_result(None)
def errored(err): future.set_exception(SomeError)
self.process.started.connect(started)
self.process.errored.connect(errored)
future = Future()
future.add_done_callback(disconnect_both_signals)
return future Then proc = Process(['ls', '-l', '/']
await proc.start()
output = ''
while proc.running():
output += await proc.read_stdout()
exit_code = await proc.finish() # <-- would require a bit more finesse than shown above for start() It occurs to me here that there is a term needed to describe concurrency safety aspects of asynchronous programs that I don't know / haven't heard of. With threads, we have thread safety as the notion that we can interact with some object from multiple threads concurrently. A similar notion likely applies to complex asynchronous objects like the ———— Edit I wrote a first implementation of this idea. See https://github.com/enkore/asynker/blob/master/examples/subprocess.py |
Hello,
I was wondering what would be the implication of using the asyncio event loops for a Qt application instead of quamash Event loop.
We can use asyncio with Qt with the following simple code. The app must be killed to be stopped but you get the point :
So what does Quamash brings in terms of compatibility and stability with the Qt system compared to asyncio event loop ? Or is it only about a proof of concept ?
The first thing I can see is that asyncio + modal dialogs does not working easiliy. But I wonder if there are more things to know that I'm not aware of ?
Thanks
The text was updated successfully, but these errors were encountered: