deko3d: add native video driver for Switch HW render path

[HueponiK]

Description

Add a native deko3d video driver for the Nintendo Switch hardware render path, along with the corresponding libretro HW render interface (RETRO_HW_CONTEXT_DEKO3D / libretro_deko3d.h).

• HW render path — Cores render directly to a DkImage and hand it to the frontend each frame via retro_deko3d_set_image_t, with optional acquire/release fences for async queue sync
• SW frame path — RGB565/ARGB8888 → RGBA8 conversion, staging upload to tile-mode DkImage, composited via dkCmdBufBlitImage
• Menu pipeline — Ozone/XMB/MaterialUI rendered through a 3D shader pipeline (GLSL → DKSH embedded at build time via uam + objcopy), with per-frame bump-allocated VBO/UBO/image-descriptor heaps
• Font driver — stb_truetype atlas mirrored to a tile-mode DkImage, per-glyph blit with premultiplied alpha for OSD/widget text
• Settings-aware — respects video_max_swapchain_images, video_swap_interval, video_hard_sync, aspect ratio / integer scale / custom viewport
• Dock/handheld resize — monitors AppletOperationMode and recreates the swapchain at the new resolution
• Build infra — HAVE_DEKO3D flag, -ldeko3d link, pattern rules for GLSL → DKSH → .o embedding

Missing

• Shaders support
• I'm sure something else...

New enums in libretro.h

• RETRO_HW_RENDER_INTERFACE_DEKO3D = 6 in retro_hw_render_interface_type
• RETRO_HW_CONTEXT_DEKO3D = 11 in retro_hw_context_type

Notes

• I am not sure about C98 standard, do we still need to comply given that this it's switch-specific target.
• Goosestation is so far the only real consumer of this driver. The driver is structured as a standard RetroArch video driver and should work for any Switch RetroArch build with HAVE_DEKO3D=1. I have a vkcube-type demo utilizing it that I could share as well.
• The deko3d API is available through devkitPro (package libdeko3d).
• The 3D-engine blit shaders are simple passthrough; the 2D engine fallback (dkCmdBufBlitImage) is available for both paths.
• Cores that want accelerated rendering on Switch today link libdeko3d directly, create their own DkDevice/DkQueue, and do a software readback to feed retro_video_refresh. This driver lets them hand the DkImage directly, saving the readback + software blit.

Kudos

MASSIVE thanks to RSDuck for duckstation for switch - it was very helpful.

[m4xw]

Great, this was something on my backlog since deko3d was originally released, will take some time to review

[HueponiK]

Thanks. Still draft as I was using another devkit image, and deko3d.h was discoverable there.
I'm also preparing the dk3dtest pr, just for demonstration

[m4xw]

Thanks. Still draft as I was using another devkit image, and deko3d.h was discoverable there. I'm also preparing the dk3dtest pr, just for demonstration

Dont worry about CI, I might need to add some libs there and we probably need to adjust ci template as well

[m4xw]

Side note, for review purposes its easier if you dont keep squashing right now and we squash on merge (regarding the force pushes)

[HueponiK]

okay here's the cube thingy I've used to test the core
https://github.com/HueponiK/dk3dtest

[HueponiK]

Sorry for the pushes, got spooked by docker-cached git clone. This should be ok to review

[m4xw]

Just a note, for switch only files (or if unavoidable codepaths) we target c11 but generic ra code needs to comply with the old ass c standard

[m4xw]

you can't switch the container that's unsupported across the board we ship a modified libnx.

[HueponiK]

you can't switch the container that's unsupported across the board we ship a modified libnx.

no probs, will revert.

[HueponiK]

Would this do as the proof of standards compliance?

# C89 compliance for all modified generic files (exit 0 = clean)
rm -f obj-unix/release/runloop.o obj-unix/release/configuration.o \
      obj-unix/release/gfx/video_driver.o obj-unix/release/gfx/font_driver.o \
      obj-unix/release/gfx/gfx_display.o obj-unix/release/menu/drivers/rgui.o \
&& make C89_BUILD=1 \
   obj-unix/release/runloop.o obj-unix/release/configuration.o \
   obj-unix/release/gfx/video_driver.o obj-unix/release/gfx/font_driver.o \
   obj-unix/release/gfx/gfx_display.o obj-unix/release/menu/drivers/rgui.o
# C11 compliance for Switch deko3d driver (exit 0 = clean)
docker run --rm -v "$PWD:/build" -w /build devkitpro/devkita64 \
  make -f Makefile.libnx DEPSDIR=/build INCLUDE='-std=c11' gfx/drivers/deko3d.o


CC runloop.c
CC configuration.c
configuration.c: In function ‘config_load_file’:
configuration.c:4025:23: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
 4025 |          char *next = strchr(extra_path, '|');
      |                       ^~~~~~
CC gfx/video_driver.c
CC gfx/font_driver.c
CC gfx/gfx_display.c
CC menu/drivers/rgui.c
deko3d.c

[m4xw]

Would this do as the proof of standards compliance?

# C89 compliance for all modified generic files (exit 0 = clean)
rm -f obj-unix/release/runloop.o obj-unix/release/configuration.o \
      obj-unix/release/gfx/video_driver.o obj-unix/release/gfx/font_driver.o \
      obj-unix/release/gfx/gfx_display.o obj-unix/release/menu/drivers/rgui.o \
&& make C89_BUILD=1 \
   obj-unix/release/runloop.o obj-unix/release/configuration.o \
   obj-unix/release/gfx/video_driver.o obj-unix/release/gfx/font_driver.o \
   obj-unix/release/gfx/gfx_display.o obj-unix/release/menu/drivers/rgui.o
# C11 compliance for Switch deko3d driver (exit 0 = clean)
docker run --rm -v "$PWD:/build" -w /build devkitpro/devkita64 \
  make -f Makefile.libnx DEPSDIR=/build INCLUDE='-std=c11' gfx/drivers/deko3d.o


CC runloop.c
CC configuration.c
configuration.c: In function ‘config_load_file’:
configuration.c:4025:23: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
 4025 |          char *next = strchr(extra_path, '|');
      |                       ^~~~~~
CC gfx/video_driver.c
CC gfx/font_driver.c
CC gfx/gfx_display.c
CC menu/drivers/rgui.c
deko3d.c

Just a standard full unix c89 build is sufficient proof, from the looks seems ok but i cant remember if we had to pass it to configure or not, so we can just do a full build

Anyway are you just experimenting or why you now add the deko lib to the docker? Just to be clear its not something we can merge because the pacman registry can be linked against a different libnx version which is undefined behaviour.

if you just do it to experiment, thats fine. I jst need to update the docker image myself, its not something you can do or have access to. This is something post-merge after i run these locally

[m4xw]

Ah seems we now have the c89 check on actions, so yea that passes, but really you dont have to worry about the pipelines for this PR

[HueponiK]

Ah seems we now have the c89 check on actions, so yea that passes, but really you don't have to worry about the pipelines for this PR

Sorry, red builds give me anxiety. We can revert it before merging, but can we keep it meanwhile? Just in case any other updates break it for some other reason?

Alternatively, we can include the proper CI fix here. Just let me know, should we pin the version, or update the dockerfile source for the builder image?

[m4xw]

Ah seems we now have the c89 check on actions, so yea that passes, but really you don't have to worry about the pipelines for this PR

Sorry, red builds give me anxiety. We can revert it before merging, but can we keep it meanwhile? Just in case any other updates break it for some other reason?

Alternatively, we can include the proper CI fix here. Just let me know, should we pin the version, or update the dockerfile source for the builder image?

Yea "for now" is fine but some members here are a bit trigger happy with merges..
Anyway like I said, thats on gitlab, you dont have access there and only me or another team member can prepare that for you