Makie as a framework for an interactive 3D event display

[tamasgal]

Good day! I am very new to Makie and worked myself through the docs and some examples but I am still not sure if it's the right tool, so allow me to ask some basic questions, I am sure you can toss me into the right direction.

A little bit of motivation: I am about to rewrite a very old project of mine in Julia. It's a 3D event display of neutrino interactions in the KM3NeT detectors written in Python using (Free)GLUT and PyOpenGL (a wild mix of legacy and modern pipelines with shaders). Back then, I only needed a camera and some input detection for keyboard and mouse for zooming, panning, mode selection etc. (the menu was just a text overlay showing the supported key presses), so I went with the bare frameworks and wrote the OpenGL code by hand. It's fairly simple but of course Python has its limits.

This is how it looks like:

Now I revived the project and since we already have a Julia framework, I'd like to do the 3D display in Julia as well.

I am wondering if Makie.jl is the right framework for such an application? I need to keep track of potentially thousands of objects which change colour, shape and position based on time and I am not sure if this is easily doable in Makie. The good thing is: I can more or less get away with just lines and spheres (maybe some transparent sphere shells and cones for wavefront propagation). I also searched for similar applications and found Opticsim.jl (https://github.com/microsoft/OpticSim.jl/blob/main/src/Vis/Visualization.jl). It looked like it's what I am searching for, saw some comments that 2D stuff is done in Luxor because it's faster, but in general I thought Makie is a good candidate but then I was really scared about the TTFP times. The examples take several(!) minutes to start on my macOS M1 and not all of them start at all (that's another issue).
So Maybe it's better to start from scratch and implement my own camera and input processing because Makie is more suited for static display and arrangements? However, taking benefit of the already existing event handling and scene creation of Makie would definitely make my life easier ;)

Sorry for the vague questions and for not spending much more time with trying things by myself but I sometimes I prefer asking experts before I waste time ;) Let me boil everything down to a few concrete ones:

• I need to manage an animated scene with many objects (maybe later I'd like to add some figures like histograms or projections as subviews): do I understand correctly that LScene is the right place to start? (https://docs.makie.org/v0.18.1/examples/blocks/lscene/)
• How do I enter some kind of an event loop in this LScene (given that LScene is the right design pattern)? In the docs I only found movie recording capabilities, which work nicely but I'd like to interact with them (play/pause, slow-down, pan/zoom/rotate, ...)
• Should I do this in Makie (I don't really need figures/axes, but they could come handy as extra widgets), or from scratch or do you know a better framework for this purpose?

Of course I'd like to contribute some basic examples for the Makie docs once I figure out the right way.

[ffreyer]

I am wondering if Makie.jl is the right framework for such an application? I need to keep track of potentially thousands of objects which change colour, shape and position based on time and I am not sure if this is easily doable in Makie.

That should be fine, especially if you can get away with a meshscatter. In that case you'd be restricted to stretching of a constant mesh along x/y/z direction, but you end up with just one render object. Something along the lines of:

fig = Figure(backgroundcolor=:black)
ax = LScene(fig[1, 1], show_axis = false)

pos = 10 .* rand(Point3f, 1000)
colors = rand(RGBf, 1000)
scales = 0.2 .* [Vec3f(0.5) .+ v for v in rand(Vec3f, 1000)]
meshscatter!(ax, pos, color = colors, markersize = scales)

fig

Makie is a good candidate but then I was really scared about the TTFP times.

Makie takes a while going from not-imported to displayed-plot. After that making a new plot is fast and runtime changes are also fast (as long as you don't trigger compilation of stuff you didn't use before). For a long running application like this I don't see TTFP as much of an issue. This also something that has been getting better with new Makie and Julia versions. The long load times you're seeing are probably a combination of package compile time (once per version), import time and jit compilation.

So Maybe it's better to start from scratch and implement my own camera and input processing because Makie is more suited for static display and arrangements?

Only CairoMakie is static. GLMakie and WGLMakie both have a renderloop and interactivity.

I need to manage an animated scene with many objects (maybe later I'd like to add some figures like histograms or projections as subviews): do I understand correctly that LScene is the right place to start? (https://docs.makie.org/v0.18.1/examples/blocks/lscene/)

If you don't want an axis yes. See above

How do I enter some kind of an event loop in this LScene (given that LScene is the right design pattern)? In the docs I only found movie recording capabilities, which work nicely but I'd like to interact with them (play/pause, slow-down, pan/zoom/rotate, ...)

Makie is designed around Observables which propagate updates. For example you could react to changes of events(fig).mouseposition to move some markers around, which will be reflected in the plot window. You can check the events section and observables section in the docs for this. Updating things each frame isn't something Makie exposes, but you can still do in GLMakie by reacting to screen.render_tick where screen = display(fig) or screen = fig.scene.current_screens[1] if the window is already open.

[jkrumbiegel]

Also LScene is really just a plain Scene plus some boilerplate so you can stick it into a Figure with its layout, but for a pure 3D viewer without other layout parts you can just use a Scene directly which has the least amount of overhead.

[tamasgal]

OK thanks!

[tamasgal]

Thanks @ffreyer I start to understand things much better. The TTFP for simple things is definitely fine but for OpticSim I really need to wait up to 5min until I can see something, which is what worried me. But I also realised that it's pinned to older Makie versions etc., so that's probably fine 😉

You can check the events section and observables section in the docs for this.

Yes I saw that already, I am currently experimenting with it!

Updating things each frame isn't something Makie exposes, but you can still do in GLMakie by reacting to screen.render_tick where screen = display(fig) or screen = fig.scene.current_screens[1] if the window is already open.

This is probably more problematic in my case. Let me try to explain: in case of the original application, I simply have a t variable which is updated each frame by a certain amount (e.g. 1 ns per frame). Everything what's displayed on the canvas is based on this single variable (and the camera position), meaning that I can exactly calculate each particles travel path, each optical or acoustic module's state (signalling a hit etc.).
The dummy code for such a display loop is something like:

t = t + tick
for part in hardware_components
    draw(part, t)
end
for particle in particles
    draw(particle, t)
end
...
...

I am now figuring out how screen.render_tick can be utilised. Maybe my approach is simply not the one which fits Makie's architecture. I just don't want to end up with something which works against Makie's system and quickly hits a dead-end due to the bad design 🙈

[ffreyer]

Your options are to run your own timing code (via Timer, a loop with sleep or a self triggering observable) or to have render_tick trigger a time update. Which one you wants depends on what you're doing.

If you want to/need to control time steps yourself (for solving a differential equation for example) you should make your own timer. For example

dt = 0.01
last_time = time()
while running && isopen(screen)
    # update observables with dt step
    delta_time = time() - last_time
    if delta_time < dt
        sleep(dt - delta_time)
    else
        yield()
    end
    last_time = time()
end

If you only care about positions etc when it's time to display them (e.g. you have a solved path $r(t)$ and just need to plug in times), render_tick would be the right way to go. For example:

time_obs = Observable((time(), 0.0))
on(_ -> time_obs[] = (time(), time() - time_obs[][2]), screen.render_tick)
on(time_obs) do (timestamp, delta)
    # update observables with delta/timestamp
end

[SimonDanisch]

I'd start like this:

using GLMakie, GLFW

scene = Scene(backgroundcolor=:black)
cam3d!(scene) # leave out if you implement your own camera

pos = 10 .* rand(Point3f, 1000)
colors = rand(RGBf, 1000)
scales = 0.2 .* [Vec3f(0.5) .+ v for v in rand(Vec3f, 1000)]
meshplot = meshscatter!(scene, pos, color = colors, markersize = scales)
# center camera to boundingbox of plots 

center!(scene)

screen = display(GLMakie.Screen(start_renderloop=false), scene)


# Best look up how renderloop is implemented here, since it's not very well documented
# https://github.com/MakieOrg/Makie.jl/blob/master/GLMakie/src/screen.jl#L656

while isopen(screen)
    # do your updates, put your camera(scene) updates here as well if implementing your own camera
    meshplot.colors = rand(RGBf, 1000)
    meshplot.markersize = 0.2 .* [Vec3f(0.5) .+ v for v in rand(Vec3f, 1000)]
    meshplot[1] = 10 .* rand(Point3f, 1000)
    GLMakie.pollevents(screen)
    GLMakie.render_frame(screen)
    GLFW.SwapBuffers(GLMakie.to_native(screen))
end
GLMakie.destroy!(screen)

Not using Figure + LScene, should give you quite a bit faster TTFP and from what I see you really don't need it :)

You can also look at https://github.com/MakieOrg/Makie.jl/blob/master/src/camera/camera3d.jl, to see how to implement the camera...

In essence, it boils down to updating the matrices in camera(scene)...

There shouldn't be anything you can't do, Makie gets quite low level if you want to (although the low level apis aren't as well documented...)..
For the low level aspects of Scene, there is https://docs.makie.org/stable/tutorials/scenes/ though, which should help you ;)

[SimonDanisch]

There's also good chance that you don't need to re-implement your own camera, since the current 3d camera is pretty customizable thanks to @ffreyer: https://docs.makie.org/v0.18.1/documentation/cameras/index.html#3d_camera

[tamasgal]

Many thanks Simon and Frederic, I think the fog begins to lift 😉 I'll now explore Simon's sketch with the isopen(screen) main loop and try to implement a few basic 3D objects with a simple animation.

[SimonDanisch]

Let us know how it goes, looks like a fun project :)
If something is slow, please open an issue, there shouldn't be any reason besides a bug or wrong usage, to not get the full power of the GPU with GLMakie ;)

[SimonDanisch]

Did you take a look at https://docs.makie.org/stable/tutorials/scenes/ ?
It explains exactly that ;)

using GLMakie
scene = Scene()
cam3d!(scene)
meshscatter!(scene, rand(Point3f, 10))
center!(scene)
pix = Makie.campixel(scene)
text!(pix, Point2f(10, 500), text = "hiiii\nhow are you\nwell well well")
scene

[tamasgal]

Ah damn, I overlooked that. Yes, that's probably, thanks and sorry 🙈

[tamasgal]

Alright, it's taking shape. Many thanks, I think I've got the basic ingredients and the first neutrino is flying happily through space and time ;)

[jkrumbiegel]

Simon what is the difference between doing the campixel thing and specifying space = :screen or whatever the option for pixel space was?

[SimonDanisch]

@jkrumbiegel good point, almost forgot we have that now... I guess none, besides having your own scene, that can be toggled etc...

[tamasgal]

So after a longer break I revived this tiny project and so far the performance seems fine without using custom shaders (I am happy with the ball-shaders from geometry toolbox 😉 )

I still have a few bigger obstacles but let's start with the most annoying one: I can't figure out how to use the cam3d!() properly so that it looks and rotates around a specific target.
For example, here is a scene, the black sphere is (0, 0, 0) and the red one is where I want to "look at" and "rotate around".

I am using this code to create and place the camera (I read through https://docs.makie.org/stable/documentation/cameras/), but as seen in the screenshot, it points slightly above the black sphere (0, 0, 0) and rotates somewhere around the z-axis near the black sphere, between the red and black sphere. It's obviously doing some automatic guessing where I want to look at based on the objects in the scene.

cam = cam3d!(scene, rotation_centre = :lookat)
update_cam!(scene, cameracontrols(scene), Vec3f(1000), det_center)
# also tried with directly passing it:
# update_cam!(scene, cam, Vec3f(1000), det_center)

If I add more objects, it even sends the camera to a very far distance.

What am I doing wrong?

[ffreyer]

Kind of related - #2746 adds left_alt + click to center on plot objects. So for meshscatter you could click on one of the scattered spheres with the pr to center on it. (For other plot types it's usually an interpolated position that gets centered)

[tamasgal]

OK thanks. Now I am mostly fighting with the keyboard control. I think that only "E" is working, which slightly tilts the view but there is no way back ;)

Anyways, since I need to implement quite a few keyboard commands, I'll probably try to use a simple camera, or maybe there is a way to override the existing keys... I'll try.

[ffreyer]

The pr also improves general camera handling, specifically around zoom. I think I also made some adjustments to how rotation, translation and zoom speeds scale.

Regardless of that though you don't have to use the keyboard controls. They're just an addition/alternative to the mouse controls. You should be able to change every keyboard control by passing the correct kwarg to cam3d! (or other constructors that end up using Camera3D).
Following the naming scheme from this:

• W and S are dolly
• R and F are pedestal
• A and D are truck
• J and L are pan
• I and K are tilt
• Q and E are roll
• B and N is fov based zooming
• U and O is a mix of dolly and zoom (moving camera without changing the focal point)
• left mouse is pan and tilt
• right mouse is pedestal and truck
• scroll is fov zoom

[tamasgal]

Nice!

Yes I noticed, by defining something like

    on(events(scene).keyboardbutton, priority = 20) do event
        if event.key == Makie.Keyboard.r
            frame_idx = 0
            return Consume()
        end
        if event.key == Makie.Keyboard.left
            frame_idx -= 100
            return Consume()
        end
        if event.key == Makie.Keyboard.right
            frame_idx += 100
            return Consume()
        end
    end

overrides the default bindings :)

[ffreyer]

You can also do cam3d!(..., up_key = false, ...) or cam.attributes.up_key[] = false to disable them

[tamasgal]

Sorry for the spam, but very happy with the results so far, thanks for all the input! 🙂

That's a muon neutrino CC event in one of the KM3NeT neutrino detectors (it's interactive and nicely animated):