Keyboard controlled 3d camera

[ffreyer]

My goal is to create a more controllable 3d camera. If I can figure out how to transfer everything maybe this can be a replacement for the current 3d camera.

TODO

• translations (x/y/z or truck/pedestal/dolly) via keyboard
• rotations (around y/x/z axis or pan/tilt/roll) via keyboad
• zoom via keyboard
• switch for rotation center (lookat vs eyeposition as center)
• pan/tilt via left mouse drag
• truck/pedestal via right mouse drag
• zoom via mousewheel
• fix issue with far reseting (fixes update_cam! doesn't respect the value of Camera3D.far #801, see also Make mouse scroll change FOV (zoom) instead of moving camera "lookat" #413)
• add perspective switch
• use fov zoom in perspective camera (fixes Zooming without entering shape #602, Camera weirdness #941)
• add cad version (Idk what exactly this is supposed to do so it might be wrong)
• add translations/rotations snapping to axis directions (i.e. x + drag -> only translate in x direction / rotate around x-axis)
• fix issue with camera partially resetting to cam3d! on plot! calls
• cleanup camera correctly
• cleanup restricted rotations with key interaction
• maybe make rotations around the up axis loop-like This only feels right when fixing an axis
• adjust default keys (a, d to translation)
• scale translations with zoom level
• update docs

Related:
https://discourse.julialang.org/t/precise-control-of-navigation-in-3d-scene-in-makie
#941
I'm surprised I'm not finding more issues related to this

[hacklint]

Hi - If I would like to test this branch on my Mac, how do I go about it? I think I did pkg add Makie way back, not JuliaPlots. I guess I kind of need to do pkg add JuliaPlots#camera but with a twist since the pr isn’t merged so not in that repo yet... or?

[ffreyer]

I do ]dev Package and work out the rest with git/vscode. If this branch was in the main repo I think you could do something like ]add Makie#camera but it's on my fork. Maybe You can do ]add https://github.com/ffreyer/Makie.jl.git#camera?

[ffreyer]

I don't think fov zoom + shift lookat are compatible...

[hacklint]

Tried ]add as above, no complaints.
Did rm on my previously added "real Makie" at the same time.
Assume I need to explicitly ask for the keyboard_cam, tried that with specifying camera = Makie.keyboard_cam! as below
I did note the "automatic" added in the source but not sure how that is supposed to work.
However, trying to run the red/blue triangles snippet with new camera failed for me:
`julia> fig, ax, p = mesh(
rand(Point3f0, 30),
[3i-j for i in 1:10, j in 0:2],
color = :white,
ambient = Vec3f0(1,0,0),
diffuse = Vec3f0(-100,0,100),camera=Makie.keyboard_cam!
)
ERROR: StackOverflowError:
Stacktrace:
[1] deepcopy(obs::Observable{Any})
@ Makie ~/.julia/packages/Makie/Khk3y/src/dictlike.jl:72
[2] theme(x::Scene, key::Symbol)
@ Makie ~/.julia/packages/Makie/Khk3y/src/scenes.jl:362
[3] default_theme(scene::Scene)
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:8
[4] (::Makie.var"#174#175")(scene::Scene)
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:217
[5] default_theme(scene::Scene, #unused#::Type{Scatter{Tuple{Vector{Point{3, Float32}}}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/recipes.jl:137
[6] (::Makie.var"#214#216"{Scene, DataType})()
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:522
[7] merged_get!(defaults::Makie.var"#214#216"{Scene, DataType}, key::Symbol, scene::Scene, input::Attributes)
@ Makie ~/.julia/packages/Makie/Khk3y/src/utilities/utilities.jl:221
[8] Scatter{Tuple{Vector{Vec{3, Float32}}}}(scene::Scene, attributes::Attributes, input::Tuple{Observable{Vector{Vec{3, Float32}}}}, args::Observable{Tuple{Vector{Point{3, Float32}}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:521
[9] plot!(scene::Scene, P::Type{Scatter{Tuple{Vector{Vec{3, Float32}}}}}, attributes::Attributes, input::Tuple{Observable{Vector{Vec{3, Float32}}}}, args::Observable{Tuple{Vector{Point{3, Float32}}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:774
[10] plot!(scene::Scene, P::Type{Scatter{ArgType} where ArgType}, attributes::Attributes, args::Observable{Vector{Vec{3, Float32}}}; kw_attributes::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:698
[11] plot!(scene::Scene, P::Type{Scatter{ArgType} where ArgType}, attributes::Attributes, args::Observable{Vector{Vec{3, Float32}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:667
[12] #plot!#218
@ ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:642 [inlined]
[13] scatter!(::Scene, ::Vararg{Any, N} where N; attributes::Base.Iterators.Pairs{Symbol, Any, NTuple{5, Symbol}, NamedTuple{(:marker, :markersize, :markerspace, :color, :visible), Tuple{Char, Observable{Float32}, UnionAll, Symbol, Observable{Any}}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/recipes.jl:19
[14] keyboard_cam!(scene::Scene; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/camera/keyboard_camera3d.jl:125
[15] keyboard_cam!
@ ~/.julia/packages/Makie/Khk3y/src/camera/keyboard_camera3d.jl:19 [inlined]
[16] apply_camera!(scene::Scene, cam_func::typeof(Makie.keyboard_cam!))
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:831
[17] setup_camera!(scene::Scene)
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:850
[18] plot!(scene::Scene, P::Type{Scatter{Tuple{Vector{Vec{3, Float32}}}}}, attributes::Attributes, input::Tuple{Observable{Vector{Vec{3, Float32}}}}, args::Observable{Tuple{Vector{Point{3, Float32}}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:805
--- the last 9 lines are repeated 1675 more times ---
[15094] plot!(::Scene, ::Type{Mesh{ArgType} where ArgType}, ::Attributes, ::Vector{Point{3, Float32}}, ::Vararg{Any, N} where N; kw_attributes::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:show_axis,), Tuple{Bool}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/interfaces.jl:698
[15095] plot(::Type{Mesh{ArgType} where ArgType}, ::Vector{Point{3, Float32}}, ::Vararg{Any, N} where N; axis::NamedTuple{(), Tuple{}}, figure::NamedTuple{(), Tuple{}}, kw_attributes::Base.Iterators.Pairs{Symbol, Any, NTuple{4, Symbol}, NamedTuple{(:color, :ambient, :diffuse, :camera), Tuple{Symbol, Vec{3, Float32}, Vec{3, Float32}, typeof(Makie.keyboard_cam!)}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/figureplotting.jl:28
[15096] mesh(::Vector{Point{3, Float32}}, ::Vararg{Any, N} where N; attributes::Base.Iterators.Pairs{Symbol, Any, NTuple{4, Symbol}, NamedTuple{(:color, :ambient, :diffuse, :camera), Tuple{Symbol, Vec{3, Float32}, Vec{3, Float32}, typeof(Makie.keyboard_cam!)}}})
@ Makie ~/.julia/packages/Makie/Khk3y/src/recipes.jl:15

(@v1.6) pkg> st
Status ~/.julia/environments/v1.6/Project.toml
[537997a7] AbstractPlotting v0.18.2
[5ae59095] Colors v0.12.8
[5789e2e9] FileIO v1.9.1
[5c1252a2] GeometryBasics v0.3.12
[682c06a0] JSON v0.21.1
[570499db] LasIO v0.3.7
[ee78f7c6] Makie v0.13.11 https://github.com/ffreyer/Makie.jl.git#camera
[c3e4b0f8] Pluto v0.14.7

julia> `

[ffreyer]

I'm currently doing Makie.keyboard_cam!(ax.scene). Good to know that passing it doesn't work yet

[hacklint]

Maybe I'm a bit too early trying to test this?
Latest try, with just trying to plot with intention to change camera afterwards as you suggested:
_ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _ | |
| | || | | | (| | | Version 1.6.1 (2021-04-23)
/ |_'|||_'_| | Official https://julialang.org/ release
|__/ |

julia> using AbstractPlotting

julia> using Makie

julia> fig, ax, p = Makie.mesh(rand(Point3f0, 30),
[3i-j for i in 1:10, j in 0:2],
color = :white,
ambient = Vec3f0(1,0,0),
diffuse = Vec3f0(-100,0,100))
Error showing value of type Makie.FigureAxisPlot:
ERROR: No backend available (GLMakie, CairoMakie, WGLMakie)!
Maybe you imported GLMakie but it didn't build correctly.
In that case, try ]build GLMakie and watch out for any warnings.
If that's not the case, make sure to explicitely import any of the mentioned backends.

Stacktrace:
[1] error(s::String)
@ Base ./error.jl:33
[2] backend_display(#unused#::Missing, #unused#::Makie.Scene)
@ Makie ~/.julia/packages/Makie/Khk3y/src/display.jl:43
[3] display
@ ~/.julia/packages/Makie/Khk3y/src/display.jl:62 [inlined]
[4] display
@ ~/.julia/packages/Makie/Khk3y/src/display.jl:52 [inlined]
[5] display(fap::Makie.FigureAxisPlot)
@ Makie ~/.julia/packages/Makie/Khk3y/src/display.jl:51
[6] #invokelatest#2
@ ./essentials.jl:708 [inlined]
[7] invokelatest
@ ./essentials.jl:706 [inlined]
[8] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
@ REPL /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:247
[9] (::REPL.var"#40#41"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
@ REPL /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:231
[10] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
@ REPL /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:462
[11] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
@ REPL /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:229
[12] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
@ REPL /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798
[13] #invokelatest#2
@ ./essentials.jl:708 [inlined]
[14] invokelatest
@ ./essentials.jl:706 [inlined]
[15] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
@ REPL.LineEdit /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2441
[16] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
@ REPL /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1126
[17] (::REPL.var"#44#49"{REPL.LineEditREPL, REPL.REPLBackendRef})()
@ REPL ./task.jl:411

julia> `

[ffreyer]

Makie replaces AbstractPlotting. With this branch added and the latest GLMakie it should work.

using GLMakie
fig, ax, p = meshscatter(rand(Point3f0, 100), markersize=0.01)
cam = Makie.keyboard_cam!(ax.scene)
fig

[ffreyer]

This should work now too. I had a plot in the camera constructor to show where lookat is, which causes the camera picking code to figure again, causing an infinite loop.

fig, ax, p = mesh(
    rand(Point3f0, 30),
    [3i-j for i in 1:10, j in 0:2],
    color = :white,
    ambient = Vec3f0(1,0,0),
    diffuse = Vec3f0(-100,0,100),
    camera=Makie.keyboard_cam!
)

[hacklint]

Sweet!
Added GLMakie from pkg and got the mesh example above running - nice!
Seems I ran into one little issue with zoom_step though, but this feels really promising!

`julia> using GLMakie
[ Info: Precompiling GLMakie [e9467ef8-e4e7-5192-8a1a-b1aee30e663a]

julia> fig, ax, p = mesh(
rand(Point3f0, 30),
[3i-j for i in 1:10, j in 0:2],
color = :white,
ambient = Vec3f0(1,0,0),
diffuse = Vec3f0(-100,0,100),camera=Makie.keyboard_cam!
)

julia> Error in callback:
UndefVarError: zoom_step not defined
Stacktrace:
[1] on_pulse(scene::Scene, cam::Makie.KeyCamera3D, timestep::Float32)
@ Makie ~/.julia/packages/Makie/VP7ek/src/camera/keyboard_camera3d.jl:279
[2] (::Makie.var"#329#336"{Scene, Makie.KeyCamera3D, Attributes})(prev_time::Float64)
@ Makie ~/.julia/packages/Makie/VP7ek/src/camera/keyboard_camera3d.jl:86
[3] #invokelatest#2
@ ./essentials.jl:708 [inlined]
[4] invokelatest
@ ./essentials.jl:706 [inlined]
[5] notify
@ ~/.julia/packages/Observables/OFj0u/src/Observables.jl:88 [inlined]
[6] setindex!
@ ~/.julia/packages/Observables/OFj0u/src/Observables.jl:248 [inlined]
[7] (::Makie.var"#331#338"{Scene, NTuple{14, Symbol}, Makie.KeyCamera3D, Attributes})(event::Makie.KeyEvent)
@ Makie ~/.julia/packages/Makie/VP7ek/src/camera/keyboard_camera3d.jl:106
[8] #invokelatest#2
@ ./essentials.jl:708 [inlined]
[9] invokelatest
@ ./essentials.jl:706 [inlined]
[10] notify(observable::Makie.PriorityObservable{Makie.KeyEvent})
@ Makie ~/.julia/packages/Makie/VP7ek/src/interaction/PriorityObservable.jl:72
[11] setindex!
@ ~/.julia/packages/Makie/VP7ek/src/interaction/PriorityObservable.jl:66 [inlined]
[12] (::GLMakie.var"#keyoardbuttons#68"{Makie.PriorityObservable{Makie.KeyEvent}})(window::GLFW.Window, button::GLFW.Key, scancode::Int32, action::GLFW.Action, mods::Int32)
@ GLMakie ~/.julia/packages/GLMakie/Q90mX/src/events.jl:103
[13] _KeyCallbackWrapper(window::GLFW.Window, key::GLFW.Key, scancode::Int32, action::GLFW.Action, mods::Int32)
@ GLFW ~/.julia/packages/GLFW/BWxfF/src/callback.jl:43
[14] PollEvents
@ ~/.julia/packages/GLFW/BWxfF/src/glfw3.jl:620 [inlined]
[15] pollevents
@ ~/.julia/packages/GLMakie/Q90mX/src/screen.jl:548 [inlined]
[16] fps_renderloop(screen::GLMakie.Screen, framerate::Float64)
@ GLMakie ~/.julia/packages/GLMakie/Q90mX/src/rendering.jl:21
[17] renderloop(screen::GLMakie.Screen; framerate::Float64)
@ GLMakie ~/.julia/packages/GLMakie/Q90mX/src/rendering.jl:48
[18] renderloop(screen::GLMakie.Screen)
@ GLMakie ~/.julia/packages/GLMakie/Q90mX/src/rendering.jl:41
[19] (::GLMakie.var"#50#52"{GLMakie.Screen})()
@ GLMakie ./task.jl:411
julia> `

[ffreyer]

@SimonDanisch What do you think of having this replace Camera3D?

The switch to fov zoom stops you from entering shapes, but I don't think you can get shift_lookat to work correctly with it. I have some approximate solution here now, which works reaosnably well I think. The cad version is probably also a bit different. Other than that this should do everything Camera3D does and more.

[ffreyer]

Closing this in favor of #1062