CSM Events and Drawing Testing

[v-rob]

Edit: This draft has since lost a sense of direction, and not everything pertains to the UI. See this post:

As a heads up: this draft PR is sort of turning into a big playground for messing around with drawing and textures and whatnot. It will definitely be split into many PRs, not all directly related to UI stuff. So don't be surprised if you see some weird commits with stuff that only seems superficially related to UI. (Most will be related, just some won't be). I'm still going to leave it open so everyone can see what's going on, but in no way will this ever be a real PR all together in one.

Accordingly, I'm removing the TODO list as the one I have is too long to keep updated here. To look at what's going on, look at client_lua_api.txt which has many things that have been added and others that will be added.

Old direction (and will be the direction for some of the UI PRs this will be split up into):

This draft PR adds a CSM API for drawing to the screen and receiving events from input devices like keyboard and mouse. This is the essential prerequisite to a Lua-side GUI which is what I intend to replace formspecs with (see #6527 (comment) and onwards and https://forum.minetest.net/viewtopic.php?f=7&p=382529#p382529). The main intent is for GUI and HUD replacement, not CSM, and so it may be isolated in some other part of builtin so that it can be loaded regardless of whether CSM is enabled or not. According to my extensive testing (I didn't profile in milliseconds, but watched the framerate closely with a variety of tests), it is performant enough on both LuaJIT and plain Lua because it is practically no slower than equivalent formspecs.

To do

This PR is a playground Work in Progress. It will be split up into many PRs and closed after I've gotten enough finished.

Things I would like help with:

• How to draw a line? driver->draw2DLine does not clip to a rect and I'm not certain if there's a good way to set the thickness. Same goes for drawing other shapes, like circles or irregular filled polygons.
• How can I get touchscreen events working on non-Android systems? Would be a very nice feature

I don't want to explain every single design choice here because that would take forever, but don't hesitate to ask questions about anything.

I'm quite open to suggestions and criticism.

[sorcerykid]

This all looks very promising! I'm very excited about the prospects of this new framework. Based on the description, I suspect it may address some of the feature requests that I had planned to submit awhile back:

Popup Formspecs
Popup formspecs would be ideally suited for displaying an ephemeral, non-modal GUI on the screen where the permanence of a HUD is not required. A simple use-case would be implementing Windows 10 style notifications in Minetest Game so that users could be alerted of certain game-related events on-the-fly. The following non-interactive formspec elements would be supported: background[], background9[], image[], item_image[], animated_image[], hypertext[], box[], label[], vertlabel[].

Formspec Page Transitions
Minetest currently lacks support for animating the opening, closing, and updating of formspecs, resulting in a GUI that at times is far too abrupt. The introduction of page transitions would allow for a much more fluid presentation of graphical content in-game between state changes including but not limited to fade, dissolve, wipe_left, slide_up, square_in, circle_out.

Full HUD Replacement
I believe that Minetest's HUD, in its current form, is far too archaic and obtuse to remain viable in the long term, at least for serious game development. Consider, for example, the sheer complexity of updating a basic graphical leaderboard on the side of the screen. The entire API needs to be completely re-invented from the ground up if Minetest is to remain at all relevant in comparison to the myriad of other engines which offer a more feature-rich HUD.

NB: I wrote these proposals in the context of the current formspec API. So naturally, they would need to be adapted for this new framework. But I at least wanted to present what I had brainstormed thus far for your consideration. I think all three are absolutely worthy candidates for a next generation, CSM-based approach to the Minetest GUI.

[v-rob]

@paramat
This actually isn't designed with SSCSM or CSM in mind at all. The idea is to make the GUI and HUD Lua-based instead of C++ based. (I've discussed it in IRC a few times, I guess you haven't seen them). My main reason is that I think that it will make development of them faster, less error-prone and laborious, and, if SSCSM ever happens sometime in the future, it will be completely extensible by Lua. So yes, at present, it's a part of CSM, but in principle, that's not what it's primarily for.

When the Lua-side GUI and HUD are created, then they will be loaded separately from the rest of CSM, so basically the API you see above and maybe one or two other things will be loaded. So, it will still be Lua, but not CSM.

@sorcerykid
I have a lot of ideas for the new GUI/HUD. The ones you have here, and quite a few more. Layered GUIs are definitely in mind, and I've already resolved to replace the HUD based on the exact same code, so a gui.Label or whatever could just as easily be used on the HUD with all the same benefits. Transistions and animations are a feature I've long wanted, but that will need a bit more drawing code than I've made at this point, and really someone with a little more expertise. Sometime, after I gather my thoughts together, I'll write a post somewhere describing my thoughts.

[paramat]

v-rob, so are you saying that if SSCSM never happens this can be easily altered to be a formspec replacement?

Why not do this the other way around, code this to be independent of CSM and alter this if SSCSM is added?
Surely until SSCSM is implemented this is more useful being independent of CSM because it could actually be used? It seems useless as long as it is only usable by Client-Provided CSM which requires manual distribution of CSM mods to users.

I have seen you discussing this in IRC, i read all minetest-dev logs =)

[v-rob]

v-rob, so are you saying that if SSCSM never happens this can be easily altered to be a formspec replacement?

A little more like the other way around: it was designed for the formspec replacement in the first place with SSCSM in mind as a future extension. Currently this is in CSM because that was easiest to do (this is my first time working with C++ <-> Lua code), but it's going to have to be moved anyway, so yes, I suppose I can separate it from CSM in this PR. If there are no objections, I'll put that in the checklist.

[paramat]

Well, wait for input from other core devs first, as i am not an expert in CSM and GUI =)
I just needed some explanation of the overall approach of this in terms of CSM.

Also, i am unsure what the best approach is. I just had a thought that, until SSCSM happens, or if SSCSM never happens, this is actually still useable by having a CSMod built-in to the client? Is this the intention?

Because i am unsure i have no objections or requests currently, i will hide my earlier long comment as it was partially invalid and got sidetracked into another issue.

[Desour]

• Formspec and HUD implementation in lua would also need more functionality, like inventory interaction, a register_on_screen_resize, a register_on_show_formspec, preventing input events from being used by minetest (ie. mouse movement to camera movement) and more.
• Splitting this PR into drawing and event sounds like a good idea imo.
• Afaik there were plans to implement ScreenDrawer in lua by caching and sending everything to C++ at once. This PR doesn't seem to include this yet. Does it not increase performance or is performance good enough or what is the reason for this?

[v-rob]

Afaik there were plans to implement ScreenDrawer in lua by caching and sending everything to C++ at once. This PR doesn't seem to include this yet. Does it not increase performance or is performance good enough or what is the reason for this?

After more testing, I found that caching and calling functions separately really has no difference, neither in Lua nor LuaJIT. So, this is a lot nicer and more versatile as it's more interactive. For instance, testing the color of a pixel is possible in this, but not in the batch call.

Formspec and HUD implementation in lua would also need more functionality

Inventory interaction can come later. Checking whether the screen resized is already possible with ScreenDrawer:get_window_size and comparing the difference from last frame. Preventing input events is already planned for this PR, see the checklist where it says "Implement the ability to make events modal, i.e. take the cursor away from the game and allow it to be used externally like in formspecs, and allow changing the cursor." The register_on_formspec_input will come much later after the new GUI has been created, and then the compatibility layer will be made. So yeah, most of the things for the GUI specifically will come later.

Splitting this PR into drawing and event sounds like a good idea imo.

Yes, it's getting kind of big. I'm going to keep it in one PR for now so I can get everything done together.

[v-rob]

Possible future security issue: If we implement the Esc menu in Lua, what's to prevent a mod (either CPCSM or SSCSM) from overriding it somehow? There are multiple avenues that this could happen through, like changing a global function or modifying a table entry containing event/drawing callbacks, and I don't think they can all be secured. So, either we implement it in C++ or we allow modification but implement a universal exit-to-menu keyboard shortcut, or perhaps just Alt-F4. Not especially fond of any of those, so ways to secure the minetest.registered_on_draw/event from being modified directly by mods would be appreciated.

[v-rob]

I am considering removing the register_on_draw callback as anything drawing to the screen ideally ought to be using the future GUI/HUD API instead of registering their own custom drawing which will likely conflict and overlay everything else. ScreenDrawer wouldn't be changed, but will only be exposed in the future element:on_draw callback or whatever.

[rubenwardy]

Really, you should consider any code that is in the same environment (or perhaps even state) as SSCSM as compromised and untrusted

However, the API already allows exiting to the menu - there's a disconnect/shutdown method iirc

[hecktest]

I don't understand, is this a GUI specific PR or a generic input and rendering one? Tying this to GUI in any way might be missing the forest for the trees.

[v-rob]

I don't understand, is this a GUI specific PR or a generic input and rendering one? Tying this to GUI in any way might be missing the forest for the trees.

The events API will not be tied to the GUI in any way, but it will be used by it. On the other hand, I'm still considering whether the drawing API should be tied to the GUI simply because of technical difficulties with sharing drawing between separate mods, specifically the last mod to register a drawing function will draw above everything else, whereas a custom drawn element can specify position in relation to other elements. I don't particularly want them tied together, but we'll see.

Regardless of whether the drawing API is tied to the UI API, since the engine -> Lua code is globally exposed to Lua, there's nothing preventing CPCSMs or SSCSMs from overwriting the API completely to make its own drawing code. Not recommended for the majority of mods as it would break everything, but a specialized game that doesn't need a normal GUI could potentially do that (assuming we get SSCSMs, of course). But really, custom elements should be enough for most of these use-cases.

[v-rob]

I've been experimenting with the idea of custom textures:

https://www.youtube.com/watch?v=87iyesNooFI

I think the idea has a lot of potential, especially if we had SSCSM. Really, it came about as a by-product of my working on a highly capable drawing API. Making the texture available in-game was a small extension. It has multiple bugs, such as the impossibility of texture modifiers (it's pure ITexture, no associated IImage to allow it to be dynamic) making crack_anylength.png broken, some internal problems here and there, the chat shrinking (I really have no idea why that happens... It might be a problem with draw_transformed_texture).

Regardless, it's not PR ready by a long shot. Nor will I fix the bugs and make it a PR yet because GUIs are higher priority, although somewhat less exciting. Custom textures will make it into the GUI PR for e.g. CSS transform or paint-like mods using render textures (simple enough), but not custom in-game textures (which is where the bugs and problems are).

[v-rob]

Closing; I don't expect to work directly on this branch anymore. The first split PR will hopefully be for events, but I have my doubts as I think it might be better to wait for SDL2 events before formalizing an API. So, part of drawing might come first.

[sorcerykid]

I've been experimenting with the idea of custom textures:

Cheers v-rob. Excellent work you've done, and I can see you're on the right conceptual path.

The demo video has me particularly intrigued. I've been wanting to develop a security camera mod, so the ability to render to a node surface would prove invaluable. You've opened a world of new possibilities for modding in Minetest.

[v-rob]

The demo video has me particularly intrigued. I've been wanting to develop a security camera mod, so the ability to render to a node surface would prove invaluable. You've opened a world of new possibilities for modding in Minetest.

I would say that the biggest problem is that of when to update textures. If there are 5 security cameras, that's 5 to update every frame. If there are 500, then that's a lot of textures per frame, all of them rendering a 3D scene. The question is how to keep this number down given that any number of security cameras can be placed? If one were to limit by proximity to the player, there could still be a few hundred within range of the player. Or, in another example, if signs use custom textures for text, should all the text for every sign be generated at the start of the game? That could be a lot of textures to generate on a big world.

[v-rob]

If we get SDL2 input, I worked out a possible API. It's a near exact reflection of SDL's relevant API (although somewhat Lua-ified) meaning it's very large but has all the input functionality anyone could ever wish for.

API

• minetest.register_on_event(function(event))

  • Called every time an event (such as keyboard or mouse input) is fired after the
    engine uses it.
  • If this function returns true, all events handlers registered after this one will
    not be called.
  • To get the state of all events at the current time, use minetest.get_event_state.
  • event: A table describing the event. The table looks like this (example values):

    {
        -- The type of the event. Whatever value this is, there is a key in this table with
        -- the same name containing the event information. No other keys will be present in
        -- this parent table.
        type = "keyboard",
    
        -- A keyboard event. This event should **not** be used for textual input fields;
        -- instead, use the `text_input` and `text_edit` events.
        keyboard = {
            -- True if the key was pressed and false if released.
            down = true,
    
            -- True if the key has been held down and started repeating.
            again = false,
    
            -- The keycode of the button pressed according to the current keyboard layout.
            -- See `minetest.get_keycode_state()` for more information.
            keycode = "K",
    
            -- The keyboard-independent physical location of the button pressed on the keyboard.
            -- See `minetest.get_scancode_state()` for more information.
            scancode = "Y"
        },
    
        -- A text input event. Only happens when `minetest.set_text_input()` is set
        -- to `true`. This may be preceded by one or more `text_edit` events.
        -- Also see <https://wiki.libsdl.org/Tutorials/TextInput>.
        text_input = {
            -- The text that was inputted.
            text = "X",
        },
    
        -- A text compostion event for IME.  Only happens when `minetest.set_text_input()`
        -- is set to `true`. `minetest.set_text_input_rect()` should be used to control the
        -- position of the candidate list.
        -- Also see <https://wiki.libsdl.org/Tutorials/TextInput>.
        text_edit = {
            -- The text that is currently being edited with the IME.
            text = "abc",
    
            -- The cursor location inside the composition.
            start = 2,
    
            -- The length of the composition.
            length = 3,
        },
    
        -- The mouse moved.
        mouse_motion = {
            -- The position of the mouse after the motion.
            pos = {x = 10, y = 10},
    
            The amount the mouse moved relative to its last position.
            motion = {x = 5, y = 5}
        },
    
        -- A mouse button was pressed or released.
        mouse_button = {
            -- True if the button was pressed and false if released.
            down = true,
    
            -- The button that was pressed or released. One of "Left", "Middle", "Right",
            -- "X1", or "X2".
            button = "Left",
    
            -- The number of times the button was pressed in quick succession, i.e. one
            -- for a single-click, 2 for a double-click, etc.
            clicks = 1
        },
    
        -- A mouse wheel was scrolled.
        mouse_wheel = {
            -- The amount the wheel scrolled in the X and Y directions. For mouse wheels,
            -- the value is usually one or zero, but things like touchpads can have lesser
            -- or greater values.
            motion = {x = 0, y = 1}
        },
    
        -- A touchscreen was interacted with.
        touch = {
            -- Touchscreens usually can handle multiple touches independently. This is the
            -- finger/device generating this particular event.
            finger = 0,
    
            -- The type of the interaction. One of "motion", "down", or "up".
            type = "motion",
    
            -- The position of the finger.
            pos = {x = 10, y = 10},
    
            -- The amount the finger moved from its last position. Contains zeroes if not
            -- of type "motion".
            motion = {x = 50, y = 34},
    
            -- The amount of pressure applied in the range [0, 1].
            pressure = 1
        },
    
        -- A touchscreen was interacted with multiple fingers in a gesture.
        touch_gesture = {
            -- The number of fingers/devices involved in the gesture.
            num_fingers = 2,
    
            -- The amount the fingers moved in a rotating motion in (TODO: radians, degrees, what? Needs testing).
            rotate = TODO,
    
            -- The amount the fingers moved in a pinching/expanding motion. (TODO: Again, needs some units)
            pinch = TODO,
    
            -- The center of the gesture.
            center = {x = 49, y = 73}
        },
    
        -- A game controller axis was moved.
        -- Note: Game controllers are a subset of joysticks like Xbox or Playstation
        -- controllers. They have consistent layouts unlike joysticks, which may not have
        -- the same buttons, axes, or layout. Game controller events will only occur for
        -- supported joysticks, but joystick events will occur for all joysticks and game
        -- controllers as well.
        controller_axis = {
            -- The axis that was moved. See `minetest.get_controller_axes_states()` for
            -- possible values.
            axis = "Left X",
    
            -- The value of the axis in the range [-1, 1].
            value = -0.5
        },
    
        -- A game controller button was pressed or released.
        controller_button = {
            -- True if the button was pressed and false if released.
            down = true,
    
            -- The button that was pressed or released. See
            -- `minetest.get_controller_button_states()` for possible values.
            button = "X",
        },
    
        -- A joystick axis was moved.
        joystick_axis = {
            -- The index of the axis that was moved. The value is joystick-dependent,
            -- anywhere from 0 to 255.
            axis = 0,
    
            -- The value of the axis in the range [-1, 1].
            value = -0.5
        },
    
        -- A joystick button was pressed or released.
        joystick_button = {
            -- True if the button was pressed and false if released.
            down = true,
    
            -- The index of the button that was pressed or released. The value is
            -- joystick-dependent, anywhere from 0 to 255.
            button = 3
        },
    
        -- A joystick trackball was moved.
        joystick_trackball = {
            -- The index of the trackball that was moved.  The value is joystick-dependent,
            -- anywhere from 0 to 255.
            trackball = 0,
    
            -- The motion in the X and Y directions, each in the range [-1, 1].
            motion = {x = -0.5, y = 1}
        },
    
        -- A joystick POV hat was moved.
        joystick_hat = {
            -- The index of the hat that was moved.  The value is joystick-dependent,
            -- anywhere from 0 to 255.
            hat = 0,
    
            -- The value of the hat. See `minetest.get_joystick_hat_states()` for possible
            -- values.
            value = "up"
        },
    
        -- A file or text is either dragging over or was dropped onto the window.
        drag_drop = {
            -- The type of operation. One of "start" (the item is hovering over the window),
            -- "end" (the operation has completed), "file" (a file has just been dropped),
            -- or "text" (text has just been dropped).
            type = "file",
    
            -- If the type is "file" or "text", this is the text or file that was dropped.
            -- Otherwise, this is `nil`.
            value = "/path/to/file.txt",
        },
    
        -- An event related to the Minetest window. Only exposes information useful
        -- to games and mods, not external information.
        window = {
            -- The type of window event. One of "resize" (the window size was changed),
            -- "focus" (the window recieved focus), "blur" (the window lost focus),
            -- "enter" (the mouse entered the window), and "leave" (the mouse left the
            -- window).
            type = "resize",
    
            -- If the type is "resize", this is the new size of the window. Otherwise `nil`.
            size = {x = 1024, y = 600}
        }
    }

• minetest.get_keycode_state() -> state: Returns a table of the current state of the
  keyboard indexed by keycodes.

  • The table has the organization { [keycode] = true } for each keycode that is
    currently held down.
  • Keycodes are values according to the current keyboard layout. For instance, a "Y"
    on a QWERTY keyboard would be a "K" on a DVORAK. Keycodes are recommended to be
    used for their meaning, like "I" for inventory.
  • For a comprehensive list of keycodes, see https://wiki.libsdl.org/SDL_Keycode,
    leftmost column.

• minetest.get_scancode_state() -> state: Returns a table of the current state of the
  keyboard indexed by scancodes.

  • The table has the organization { [scancode] = true } for each scancode that is
    currently held down.
  • Scancodes are keyboard layout-independent values based on the physical location of
    the button on the keyboard. Often, this value is the same as the QWERTY keycode.
    Scancodes are recommended to be used for their position, like WASD for movement.
  • For a comprehensive list of scancodes, see https://wiki.libsdl.org/SDL_Scancode,
    leftmost column.

• minetest.get_keymod_state() -> state: Returns a table of the current state of
  keyboard modifiers.

  • The table has the organization { [keymod] = true } for each modifier that is
    currently active.
  • Note that modifiers that are not currently held down may be active, e.g. keys
    like Caps Lock or if the Windows Sticky Keys feature is on, so
    minetest.get_keycode_state() and minetest.get_scancode_state() are not substitutes.
  • Possible modifiers (same name as their keycode/scancode counterparts):
    • "Left Shift"
    • "Right Shift"
    • "Left Ctrl"
    • "Right Ctrl"
    • "Left Alt"
    • "Right Alt"
    • "Left GUI"
    • "Right GUI"
    • "Numlock"
    • "CapsLock"
    • "ModeSwitch"
  • In addition, since it often doesn't matter whether the left or right modifier key is
    pressed, the following exist if either right or left is active:
    • "Shift"
    • "Caps": If caps lock or either shift is active
    • "Ctrl"
    • "Alt"
    • "GUI"

• minetest.set_text_input(enable): Sets whether the text_input and text_edit
  events in minetest.register_on_event should be activated.

  • enable: Whether it should be enabled. Initially false when the game starts.

• minetest.get_text_input() -> enabled: Gets whether the text_input and
  text_edit events in minetest.register_on_event are activated.

• minetest.set_text_input_rect(rect) Sets the rectangle for the OS-drawn IME candidate
  list during text_input events.

• minetest.get_mouse_button_state() -> state: Returns a table of the current state of
  the mouse buttons.

  • The table has the organization { [button] = true } for each button that is
    currently held down.
  • Possible buttons:
    • "Left"
    • "Middle"
    • "Right"
    • "X1"
    • "X2"

• minetest.set_mouse_pos(pos): Sets the current position of the mouse inside the
  Minetest window.

  • Only works when the mouse is not captured. See minetest.set_mouse_capture.

• minetest.get_mouse_pos() -> pos: Gets the current position of the mouse inside the
  Minetest window.

  • Only works when the mouse is not captured. See minetest.set_mouse_capture.

• minetest.set_cursor(cursor): Changes the current cursor.

  • Only visible when the mouse is not captured. See minetest.set_mouse_capture.
  • cursor: One of the following strings:
    • "none": Invisible cursor
    • "arrow": The default arrow pointer
    • "ibeam": An I-beam for input fields
    • "wait": A waiting cursor, like an hourglass or spinning circle
    • "waitarrow": The arrow pointer and wait cursor combined
    • "crosshair": A crosshair cursor
    • "no": A crossed out circle
    • "hand": A pointer hand
    • "sizenwse": A double arrow pointing northwest and southeast
    • "sizenesw": A double arrow pointing northeast and southwest
    • "sizewe": A double arrow pointing west and east
    • "sizens": A double arrow pointing north and south
    • "sizeall": A four pointed arrow pointing north, south, east, and west

• minetest.get_cursor() -> cursor: Gets the current cursor.

• minetest.set_mouse_capture(capture): Sets whether the mouse cursor should be
  invisible and trapped within the Minetest window.

  • capture: Set to true to capture the mouse (like in-game) and false to make it
    visible (like in a menu).

• minetest.get_mouse_capture() -> captured: Gets whether the mouse cursor is currently
  invisible and trapped within the Minetest window.

• minetest.get_touchscreen_state() -> state: Returns a table of the current state of
  each finger on the touchscreen.

  • state: A table of finger states where each finger looks like this:

  {
      -- The position of the finger
      pos = {x = 23, y = 34},
  
      -- The amount of pressure applied in the range [0, 1].
      pressure = 1
  }

• minetest.get_joystick_axes_states() -> states: Returns the value of the axes on the
  joystick.

  • states: A table of axis states where each axis is a value in the range [-1, 1].

• minetest.get_joystick_button_states() -> states: Returns the value of the buttons
  on the joystick.

  • states: A table of button states where each button is true if pressed and
    false if not pressed.

• minetest.get_joystick_hat_states() -> states: Returns the value of the POV hats on
  the joystick.

  • states: A table of POV hat states where each hat is one of the following:
    • "leftup"
    • "up"
    • "rightup"
    • "left"
    • "center"
    • "right"
    • "leftdown"
    • "down"
    • "rightdown"

• minetest.get_controller_axes_states() -> states: Returns the value of the axes on the
  game controller.

  • The table has the organization { [axis] = state } for each axis.
  • Each axis is a value in the range [-1, 1].
  • List of axes:
    • "Left X"
    • "Left Y"
    • "Right X"
    • "Right Y"
    • "Left Trigger"
    • "Right Trigger"

• minetest.get_controller_button_states() -> states: Returns the value of the buttons
  on the game controller.

  • The table has the organization { [button] = true } for each button that is currently
    held down.
  • Possible buttons:
    • "A"
    • "B"
    • "X"
    • "Y"
    • "Back"
    • "Guide"
    • "Start",
    • "Left Stick"
    • "Right Stick"
    • "Left Shoulder"
    • "Right Shoulder"
    • "D-pad Up"
    • "D-pad Down",
    • "D-pad Left"
    • "D-pad Right"

• minetest.set_clipboard_text(text): Sets the text in the OS's clipboard for copy, cut, and
  paste operations

  • text: The text to set the clipboard to.

• minetest.get_clipboard_text() -> text: Gets the text in the OS's clipboard, or nil if
  there is no text in the clipboard.