Simula fails to render subsurfaces/popups flat onto their parents

[georgewsinger]

Context. Currently Simula renders each popup as if it were its own unique surface:

Instead what should happen is that these subsurfaces should render flat onto their parent surfaces (as happens on a normal 2D window manager).

Viewports. In order to do this, we will render each parent plus all of its subsurfaces onto a GodotViewport before applying its texture onto our GodotSprite3D. We've had trouble with this in the past and so I'm here opening up the can of worms again.

Attempt 1: Using a GodotViewport to render parent surfaces yields errors. As a first step towards solving this problem we can try to duplicate our current functionality using a GodotViewport. Once we get this working it should be relatively straightforward to get subsurfaces rendering properly. Here is some starting code:

updateSimulaViewSprite :: GodotSimulaViewSprite -> IO ()
updateSimulaViewSprite gsvs = do
  -- ..
  useViewportToDrawParentSurface gsvs
  -- ..

useViewportToDrawParentSurface :: GodotSimulaViewSprite -> IO ()
useViewportToDrawParentSurface gsvs  = do
  sprite3D <- readTVarIO (gsvs ^. gsvsSprite)
  simulaView <- readTVarIO (gsvs ^. gsvsView)
  let eitherSurface = (simulaView ^. svWlrEitherSurface)
  wlrSurface <- getWlrSurface eitherSurface
  parentWlrTexture <- G.get_texture wlrSurface

  -- Draw texture on Viewport; get its texture; set it to Sprite3D's texture; call G.send_frame_done
  let isNull = ((unsafeCoerce parentWlrTexture) == nullPtr)
  case isNull of
        True -> putStrLn "Texture is null!"
        False -> do renderTarget <- initializeRenderTarget wlrSurface     -- Dumb, but we reset the renderTarget every frame to make sure the dimensions aren't (0,0)
                    atomically $ writeTVar (_gsvsViewport gsvs) renderTarget -- "
                    renderPosition <- getCoordinatesFromCenter wlrSurface 0 0 -- Surface coordinates are relative to the size of the GodotWlrXdgSurface; We draw at the top left.

                    textureToDraw <- G.get_texture wlrSurface :: IO GodotTexture
                    renderTarget <- readTVarIO (gsvs ^. gsvsViewport)

                    -- Send draw command
                    godotColor <- (toLowLevel $ (rgb 1.0 1.0 1.0) `withOpacity` 1) :: IO GodotColor
                    G.draw_texture ((unsafeCoerce renderTarget) :: GodotCanvasItem) textureToDraw renderPosition godotColor (coerce nullPtr)
                    viewportTexture <- G.get_texture renderTarget
                    G.set_texture sprite3D (safeCast viewportTexture)
                    -- Tell the surface being drawn it can start to render its next frame
                    G.send_frame_done wlrSurface

initializeRenderTarget :: GodotWlrSurface -> IO (GodotViewport)
initializeRenderTarget wlrSurface = do
  renderTarget <- unsafeInstance GodotViewport "Viewport"
  -- No need to add the Viewport to the SceneGraph since we plan to use it as a render target
    -- G.set_name viewport =<< toLowLevel "Viewport"
    -- G.add_child gsvs ((safeCast viewport) :: GodotObject) True

  G.set_disable_input renderTarget True -- Turns off input handling

  G.set_usage renderTarget 0 -- USAGE_2D = 0
  -- G.set_hdr renderTarget False -- Might be useful to disable HDR rendering for performance in the future (requires upgrading gdwlroots to GLES3)

  -- CLEAR_MODE_ALWAYS = 0
  -- CLEAR_MODE_NEVER = 1
  -- I think we need CLEAR_MODE_ALWAYS since, i.e., a popup dragging might cause a trail?
  G.set_clear_mode renderTarget 0

  -- Perhaps we should never update the render target, since we do so manually each frame?
  -- UPDATE_DISABLED = 0 — Do not update the render target.
  -- UPDATE_ONCE = 1 — Update the render target once, then switch to UPDATE_DISABLED.
  -- UPDATE_WHEN_VISIBLE = 2 — Update the render target only when it is visible. This is the default value.
  -- UPDATE_ALWAYS = 3 — Always update the render target. 
  G.set_update_mode renderTarget 3 -- Using UPDATE_ALWAYS for now

  -- "Note that due to the way OpenGL works, the resulting ViewportTexture is flipped vertically. You can use Image.flip_y on the result of Texture.get_data to flip it back[or you can also use set_vflip]:" -- Godot documentation
  G.set_vflip renderTarget True -- In tutorials this is set as True, but no reference to it in Godotston; will set to True for now

  -- We could alternatively set the size of the renderTarget via set_size_override [and set_size_override_stretch]
  dimensions@(width, height) <- getBufferDimensions wlrSurface
  pixelDimensionsOfWlrSurface <- toGodotVector2 dimensions

  -- Here I'm attempting to set the size of the viewport to the pixel dimensions
  -- of our wlrXdgSurface argument:
  G.set_size renderTarget pixelDimensionsOfWlrSurface

  -- There is, however, an additional way to do this and I'm not sure which one
  -- is better/more idiomatic:
    -- G.set_size_override renderTarget True vector2
    -- G.set_size_override_stretch renderTarget True

  return renderTarget
  where
        -- | Used to supply GodotVector2 to
        -- |   G.set_size :: GodotViewport -> GodotVector2 -> IO ()
        toGodotVector2 :: (Int, Int) -> IO (GodotVector2)
        toGodotVector2 (width, height) = do
          let v2 = (V2 (fromIntegral width) (fromIntegral height))
          gv2 <- toLowLevel v2 :: IO (GodotVector2)
          return gv2

which yields black textures + the repeated console errors

ERROR: draw_texture: Drawing is only allowed inside NOTIFICATION_DRAW, _draw() function or 'draw' signal.
   At: scene/2d/canvas_item.cpp:790.

AFAIK only 2D nodes have _draw functions, so I'm not sure how this should be handled with our GodotSimulaViewSprite (which inherits Spatial). Nevertheless, if we comment out some Godot lines to get around this error:

    //canvas_item.cpp
    void CanvasItem::draw_texture(const Ref<Texture> &p_texture, const Point2 &p_pos, const Color &p_modulate, const Ref<Texture> &p_normal_map) {
    
      // if (!drawing) {
      //    ERR_EXPLAIN("Drawing is only allowed inside NOTIFICATION_DRAW, _draw() function or 'draw' signal.");
      //    ERR_FAIL();
      // }
    
      // ERR_FAIL_COND(p_texture.is_null());
    
      p_texture->draw(canvas_item, p_pos, p_modulate, false, p_normal_map);
    }

then we get new errors repeated to the console:

ERROR: getornull: Condition ' !id_map.has(p_rid.get_data()) ' is true. returned: __null
   At: ./core/rid.h:164.
ERROR: canvas_item_add_texture_rect: Condition ' !canvas_item ' is true.
   At: servers/visual/visual_server_canvas.cpp:610.
ERROR: canvas_item_add_texture_rect: Condition ' !canvas_item ' is true.
   At: servers/visual/visual_server_canvas.cpp:610.
ERROR: canvas_item_add_texture_rect: Condition ' !canvas_item ' is true.
   At: servers/visual/visual_server_canvas.cpp:610.
ERROR: canvas_item_add_texture_rect: Condition ' !canvas_item ' is true.
   At: servers/visual/visual_server_canvas.cpp:610.
ERROR: canvas_item_add_texture_rect: Condition ' !canvas_item ' is true.
   At: servers/visual/visual_server_canvas.cpp:610.
ERROR: canvas_item_add_texture_rect: Condition ' !canvas_item ' is true.
   At: servers/visual/visual_server_canvas.cpp:610.
ERROR: getornull: Condition ' !id_map.has(p_rid.get_data()) ' is true. returned: __null
   At: ./core/rid.h:164.

Very annoying.

[georgewsinger]

I just created a new popups branch in SimulaVR/Simula to track code experiments.

Right now the branch has added a new GodotRenderTarget NativeScript type that extends RigidBody2D (so that we may jam texture updating logic into _draw command). Right now textures are showing up as black:

and the RenderTarget._draw function isn't even getting called, but this might get us a step closer.

[georgewsinger]

🏁 I started up a post on /r/godot to help figure out why my _draw function is never getting called (from this commit).

[georgewsinger]

✔️ I pushed some working popups code. It's got a few bugs, but I'm going to open up new issues for them to keep things organized.