Mouse responsivness drops with large plotting loads

[rafaqz]

I'll use a MakieDraw.jl GeometryCanvas as an easy example:

using Pkg
Pkg.add(url="https://github.com/rafaqz/MakieDraw.jl")
using GLMakie, MakieDraw
c = GeometryCanvas{Point2}()
# Click and drag is fast! 

img = rand(3000, 3000)
heatmap!(c.axis, img; colormap=:magma)
# Now its really slow and laggy

The MakieDraw.jl code is and event handler like this:

on(events(fig).mousebutton, priority = 100) do event
...

https://github.com/rafaqz/MakieDraw.jl/blob/8cb95a4fa018f3159f66e3af2233bdc9de2d6016/src/geometry_canvas.jl#L266

Also notice MakieDraw is printing to the REPL (to debug this) when the click even is triggered. But once there is the heatmap plotted there is a big delay before the on handler actually receives the click - so the delay is not from plotting the clicked points, but lag in mouse input events arriving to the handler.

Also interesting is that WGLMakie.jl is faster than GLMakie.jl. It's still slower once there is a heatmap, but the event is registered in maybe half the time.

There is also basically zero CPU use happening, so I'm assuming it's related to rendering.

[rafaqz]

I guess it's because of this:

Makie.jl/GLMakie/src/events.jl

Line 226 in 641023c

on(updater, screen.render_tick)

Could the delay not be constant? this means the render time is paid twice for mouse events - first in the handler receiving the event and then in viewing the update it causes.

[ffreyer]

I don't notice delay on mouseclicks, though it seems that mouseposition updates can consumed by something so the normal axis drags don't work.

On render_tick - you can check how often these happen with something like

map(_ -> println("tick"), c.fig.scene.current_screens[1].render_tick)

Since #2336 / #2397 rendering only happens when necessary (if you use the default renderloop) so without visual changes you shouldn't be slowed down by rendering. And if you were having render_tick on some async timer wouldn't really help you. It would just move the update to the sleep/yield part of the renderloop. Maybe you would get something out of using callbacks, but I'm not sure if GLFW keeps events in strict order for those. I also don't really see the point in having more updates then rendering opportunities. That'd just be extra work that never gets displayed, wouldn't it?

[rafaqz]

Maybe, but there just is a delay on mouseclicks events because the update rate is tied to rendering. Noticing it will depend on your hardware. Dragging is more generally noticeable because you can see the gap with the cursor.

What is unnecessary is to pay the rendering delay twice on mouseclicks. The CPU is so underutilised in this example that giving it more work is not a problem at all.

A 0.1 second delay in mouse click response is a real UX problem compared to 0.05 seconds.

[ffreyer]

Maybe, but there just is a delay on mouseclicks events because the update rate is tied to rendering. Noticing it will depend on your hardware. Dragging is more generally noticeable because you can see the gap with the cursor.

Even if we polled every half frame you'd see the same gap since the polling happens immediately before rendering right now. This would be more an issue of event processing or rendering being too slow, or maybe vsync.

What is unnecessary is to pay the rendering delay twice on mouseclicks. The CPU is so underutilised in this example that giving it more work is not a problem at all.

I don't see why we would be paying twice. render_tick is triggered once a frame, triggering the updater to get the current mouse position (which as far as I know is actually current, not from the last frame or the last GLFW.PollEvents) which then causes all the observable updates to happen. After that you get the same for all the callbacks via GLFW.PollEvents and then the frame is rendered.

A 0.1 second delay in mouse click response is a real UX problem compared to 0.05 seconds.

Yea that's not nice. Maybe it would be useful to do some timings in

Makie.jl/GLMakie/src/screen.jl

Lines 860 to 881 in 4bebd86

	function on_demand_renderloop(screen::Screen)
	while isopen(screen) && !screen.stop_renderloop
	t = time_ns()
	time_per_frame = 1.0 / screen.config.framerate
	pollevents(screen) # GLFW poll
	if !screen.config.pause_renderloop && requires_update(screen)
	render_frame(screen)
	GLFW.SwapBuffers(to_native(screen))
	end
	t_elapsed = (time_ns() - t) / 1e9
	diff = time_per_frame - t_elapsed
	if diff > 0.001 # can't sleep less than 0.001
	sleep(diff)
	else # if we don't sleep, we still need to yield explicitely to other tasks
	yield()
	end
	end
	cause = screen.stop_renderloop ? "stopped renderloop" : "closing window"
	@debug("Leaving renderloop, cause: $(cause)")
	end

to figure out where you're actually spending time?

[rafaqz]

Ok I guess I don't understand the GPU/CPU interaction here and how the updating happens, except that there is very little CPU activity. I'll try your script when I get time.

It may be better to just work on other optimisations so this is less likely to happen

E.g. subsampling the heatmap removes this problem #973