GPU-accelerated Display (Vulkan) #93

osnr · 2023-09-30T17:00:48Z

Replace our software rendering with GPU rendering using shaders. All Display primitives (text, lines, images, filled shapes) should work as before.

Performance is extremely good on the NUC (60fps without any drops, even rotating multiple large images and text, compared to 20-30fps before). (less so on Pi 4, maybe moderately worse than the software renderer but not terrible)

You'll need to install Vulkan on your machine to get this to work; see the README and wiki.

Replaces

pi/Display.tcl + pi/lineclip.tcl + pi/rotate.tcl + virtual-programs/display.folk + vendor/font.tcl

with

pi/Gpu.tcl and a new virtual-programs/display.folk and vendor/fonts (currently just PT Sans; new fonts can be converted with https://github.com/Chlumsky/msdf-atlas-gen)

Implements a new 'GPU FFI' which allows you to write vertex and fragment shaders to implement drawing primitives:

set fillTriangle [Gpu::pipeline {vec2 p0 vec2 p1 vec2 p2 vec4 color} {
            vec2 vertices[4] = vec2[4](p0, p1, p2, p0);
            return vertices[gl_VertexIndex];
} {
            return color;
}]
set circle [Gpu::pipeline {vec2 center float radius} {
        // HACK: just draws to full screen
        vec2 vertices[4] = vec2[4](vec2(0, 0), vec2(_resolution.x, 0), vec2(0, _resolution.y), _resolution);
        return vertices[gl_VertexIndex];
} {
        float dist = length(gl_FragCoord.xy - center) - radius;
        return dist < 0.0 ? vec4(1.0, 0, 0, 1.0) : vec4(0, 0, 0, 0);
}]

# (must be called in between Gpu::drawStart and Gpu::drawEnd)
Gpu::draw $circle {300 300} 20
Gpu::draw $fillTriangle {0 0} {100 400} {200 300} {1 0 0 1}

The first argument to Gpu::pipeline is a list of overall arguments for both shaders. These are accessible from both the vertex and fragment shader. You pass these in at draw time (they're push constants -- sort of like uniforms in GL, but limited in total size to 128 bytes). (The exception is fn-type arguments, which aren't actually passed in; they're meant to be names of Gpu::fn functions you've already made, which are looked up at call time from the surrounding Tcl environment and inlined into the shader. They just fall out and don't map to arguments in Gpu::draw)

The second argument to Gpu::pipeline is the source code of the main function of a vertex shader. It should return a vec2 vertex of a quad in Vulkan triangle-strip vertex order (should be topleft, topright, bottomleft, bottomright, like a Z, not counterclockwise) based on gl_VertexIndex (it will be called with vertex index 0, 1, 2, 3).
The returned vertex should be in screen coordinates, not in [0, 1]. You can access the builtin vec2 _resolution to get resolution of the screen. (In practice, so far we mostly use the vertex shader for clipping so we don't have to touch the whole display on every draw.)

The third (or fourth, if you make fragment shader fn arguments before it) argument to Gpu::pipeline is the source code of the main function of a fragment shader. You can access vec2 gl_FragCoord to get current pixel coordinates. It should return a vec4 color. You can access any of the overall arguments (including _resolution) from here as well.

See Gpu.tcl and display.folk for more examples. You can make sampler2D-type arguments if you want to pass an image in; you can make fn arguments to a fragment shader as third argument.

A pressing TODO is to break the drawing primitives up into separate virtual programs and expose all this better to the user (it's currently hard to use in a user program, because you need to run this code on the Display process).

Substantially rewrites calibrate.tcl to use the new drawing system. Also slightly friendlier print output.

Removes the live camera preview from pi/Camera.tcl -- we'll recommend the web image for now, unless we reimplement it.

Extends c.tcl to automatically generate Tcl-side struct getter functions like image_t data, image_t width, etc (ensemble commands under the struct type name, one for each field).

Extends c.tcl to allow some throwing of Tcl exceptions deeper in C (still WIP).

Extends the Folk interprocess heap in main.tcl to allow freeing of heap allocations (it now is built on top of dlmalloc).
On each allocation, the heap now also stores a random 64-bit 'version' for that allocation (which can then be remembered by the caller). You can query the heap with any address and it will try to give you the 'version' if there is an allocation containing that address. This is used to check for staleness of images that have been copied to the GPU (like camera slices). If the version mismatches the one we stored at previous copy-time, then we know we have to recopy that image to the GPU.
This implementation is pretty inefficient and unsafe (it walks a capped-256 array of all allocations) and we may want to replace it with an interval tree or something at some point. We will probably want to introduce more locking, too.

Fixes #22.

Please test this if you can (and help with documentation if you can); I'd like to merge it by Monday if possible and have at least 1-2 people sign off on it. I know @charlesetc has been running it for a few days without problems.

First step toward rendering multiple objects (using different SDF shaders)

(I want to draw multiple things in a row without having them each clear the screen.)

Remove csubst now that it's part of base C lib

multiple of a shape

Also fix linebreaks in csubst

(next, update images, which is harder)

Now we can standardize on struct representation of C structs in Tcl instead of dict representation... hopefully simpler code too in the end.

WIP: Start on images/sampler2D. Improve char array support in C. Actually put uboFields in Pipeline struct so they don't shimmer out (because we now run the Pipeline data back into C to set up the image right before draw time).

(mostly undoing refactor; also pointer fix in calibrate)

l3gacyb3ta · 2023-10-03T18:23:39Z

Review time!

l3gacyb3ta

99% amazing!

l3gacyb3ta · 2023-10-03T18:24:10Z

README.md

+            $ CFLAGS="-O2 -march=armv8-a+crc+simd -mtune=cortex-a72" CXXFLAGS="-O2 -march=armv8-a+crc+simd -mtune=cortex-a72" meson -Dglx=disabled -Dplatforms= -Dllvm=disabled -Dvulkan-drivers=broadcom -Dgallium-drivers=v3d,vc4,kmsro -Dbuildtype=release ..
+
+            # AMD (radeonsi), including Beelink SER5
+            $ meson -Dglx=disabled -Dplatforms= -Ddri-drivers='' -Dvulkan-drivers=amd -Dgallium-drivers=radeonsi -Dbuildtype=release .. 


-Ddri-drivers='' is unnessary and causes it to fail

l3gacyb3ta · 2023-10-03T18:24:58Z

README.md

+
+     1. See [notes](https://folk.computer/notes/vulkan) and [Naveen's
+        notes](https://gist.github.com/nmichaud/1c08821833449bdd3ac70dcb28486539).
+1. `sudo adduser folk video` & `sudo adduser folk input` (?) & log out and log back in (re-ssh)


maybe put the video and input groups before the vulkan stuff? It tripped me up when the folk user couldn't use the video out, causing vkcube to fail

l3gacyb3ta · 2023-10-03T18:26:12Z

README.md

+   ```
+
+   Go to http://whatever.local:4273/frame-image/ to see the camera's
+   current field of view. Reposition your camera to cover your table.


oh this is awesome! Way better than trying to use the projection itself.

l3gacyb3ta · 2023-10-03T18:27:51Z

calibrate.tcl

    int captureNum = 0;
    uint8_t* delayThenCameraCapture(Tcl_Interp* interp, const char* description) {
-        usleep(100000);
+        usleep(500000);


why is this longer?

l3gacyb3ta · 2023-10-04T00:47:45Z

lib/c.tcl

            variable rtypes {
                int { expr {{ $robj = Tcl_NewIntObj($rvalue); }}}
                int32_t { expr {{ $robj = Tcl_NewIntObj($rvalue); }}}
                double { expr {{ $robj = Tcl_NewDoubleObj($rvalue); }}}
+                float { expr {{ $robj = Tcl_NewDoubleObj($rvalue); }}}
                char { expr {{ $robj = Tcl_ObjPrintf("%c", $rvalue); }}}
                bool { expr {{ $robj = Tcl_NewIntObj($rvalue); }}}
                uint16_t { expr {{ $robj = Tcl_NewIntObj($rvalue); }}}


feels like uint8_t is missing?

l3gacyb3ta · 2023-10-04T00:54:54Z

pi/Gpu.tcl

This all is flying over my head, but it seems great!

l3gacyb3ta · 2023-10-04T00:56:10Z

vendor/fonts/PTSans-Regular.csv

Maybe write up how to generate these?

l3gacyb3ta · 2023-10-04T00:57:41Z

virtual-programs/display.folk

 namespace eval ::Display {
    variable WIDTH
    variable HEIGHT
    variable LAYER 0
-    regexp {mode "(\d+)x(\d+)"} [exec fbset] -> WIDTH HEIGHT
+    if {$::isLaptop} {


Idk if this is relevant to this code, but the laptop stuff still doesn't work on my end

l3gacyb3ta

Seems good to me, but the calibration being broken is wierd

osnr · 2023-10-04T19:09:36Z

Seems good to me, but the calibration being broken is wierd

It's worked for me -- can you post some of the jpegs in the folk folder that get emitted during calibration? I wonder whether it's a timing issue or a repeatability/optics issue. It's definitely janky though

osnr · 2023-10-04T19:10:15Z

The sleep (and extra webcam framegrabs) during calibration is because it takes some time for the projected stripes to get to the real world -> be visible from the webcam (and to appear in the received buffer), but they're kind of guesses...

try more v4l2-ctl stuff. Using Gpu directly seems to fix us dropping the white draw call.

This seems good

osnr added 30 commits July 31, 2023 10:43

Make it work on macOS again(?)

9e4fb87

Make pipeline configurable

9f33273

First step toward rendering multiple objects (using different SDF shaders)

Fix C arrays kinda bc use of csubst (untested)

e56ab28

WIP: Start on refactor so shaders can be created separately

a1b66f0

Finish pipeline/shader refactor

7d1b001

WIP: csubst by default, cut up drawing pipeline

329f7cc

(I want to draw multiple things in a row without having them each clear the screen.)

Can draw multiple pipelines in a row

31b354e

Fullscreen quad

9f26ccd

Draw a circle

5de82f0

WIP: Working on pipeline DSL

9ff89f4

WIP: Continue work on pipeline DSL; make UBO args struct

55ad35a

Remove csubst now that it's part of base C lib

WIP: UBO, descriptor pool, descriptor set work

9c43faa

I can draw a circle

c324e4b

Add support for argument-free shader

db1f452

Use blending so can draw multiple stuff. Fix red alpha

a142b09

WIP: Use new descriptor set for each invocation so we can draw

87a36f8

multiple of a shape

Implement Vulkan line

a8be839

Test struct array. Fix trie ?

87507a3

WIP: Start on multi-binding + image support for pipelines

278fe71

WIP: 'input set' concept

0df5808

WIP: It runs all the way through now

a784bcc

Also fix linebreaks in csubst

WIP: Start working on updating UBO using the new design

ab7d29a

(next, update images, which is harder)

c: Start working on field getters. Throw errors from deep in C!

6cce17f

c: Fix getter for renamed cc

b29a58c

WIP: Finally implement setFromAnyProc, start on UBO memcpy again

8fc3ccd

Now we can standardize on struct representation of C structs in Tcl instead of dict representation... hopefully simpler code too in the end.

Drawing primitives work

e9682c0

Display_vk: Destructure args so you can use the arg names directly

5f3482a

Display_vk: The D in sampler2D is capitalized

58d286e

c: Scope getters so fields w/ same name across structs don't collide

a9d2cfd

osnr added 10 commits September 30, 2023 15:25

main: Document Heap

47ff036

Gpu: Better validation layer problem detect

2557732

README: Integrate Naveen notes

84ce85f

camera,calibrate: Fixing really bad breakage from refactors

82b53f1

(mostly undoing refactor; also pointer fix in calibrate)

README: Minor updates from running through mini

4607279

calibrate: Don't emit file on failure

b548abb

calibrate: Save dense corr to debug, remove dupe jpeg saving logic

02c86aa

calibrate: Just messing around but this works ok

f464de5

README: some Pi updates; try to standardize on Ubuntu Server 23.04

7f131f9

README: Finish Pi 4 directions

c27df63

l3gacyb3ta requested changes Oct 4, 2023

View reviewed changes

README: Remove dri-drivers=''

35c7347

l3gacyb3ta approved these changes Oct 4, 2023

View reviewed changes

README: add libelf-dev

02ffcb5

l3gacyb3ta self-requested a review October 4, 2023 18:57

l3gacyb3ta requested changes Oct 4, 2023

View reviewed changes

c: uint8_t rtype to get rid of warning

adae8bb

osnr added 5 commits October 4, 2023 18:38

calibrate: this works well on folk0 but not folk-convivial

5ed54d5

calibrate: Use Gpu directly; start on autodelay logic;

d2aecad

try more v4l2-ctl stuff. Using Gpu directly seems to fix us dropping the white draw call.

calibrate: Fix brightness test in determineNumberOfFramesToStabilize

a42c3fa

This seems good

Gpu: Warn if DRI file not readable

8b9ea08

calibrate: Return to autoexposure at end?

cb05583

osnr merged commit 5f870b4 into main Oct 5, 2023

osnr mentioned this pull request Oct 5, 2023

Rotation of big images is very slow #68

Closed

osnr mentioned this pull request Oct 14, 2023

Refactor display into multiple programs; make shaders available from userland; run code on subprocesses #103

Merged

nmichaud deleted the osnr/vulkan-display branch November 11, 2023 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

GPU-accelerated Display (Vulkan) #93

GPU-accelerated Display (Vulkan) #93

Uh oh!

osnr commented Sep 30, 2023

Uh oh!

l3gacyb3ta commented Oct 3, 2023

Uh oh!

l3gacyb3ta left a comment

Uh oh!

l3gacyb3ta Oct 3, 2023

Uh oh!

l3gacyb3ta Oct 3, 2023

Uh oh!

l3gacyb3ta Oct 3, 2023

Uh oh!

l3gacyb3ta Oct 3, 2023

Uh oh!

l3gacyb3ta Oct 4, 2023

Uh oh!

l3gacyb3ta Oct 4, 2023

Uh oh!

l3gacyb3ta Oct 4, 2023

Uh oh!

l3gacyb3ta Oct 4, 2023

Uh oh!

l3gacyb3ta left a comment

Uh oh!

osnr commented Oct 4, 2023

Uh oh!

osnr commented Oct 4, 2023

Uh oh!

Uh oh!

Uh oh!

GPU-accelerated Display (Vulkan) #93

GPU-accelerated Display (Vulkan) #93

Uh oh!

Conversation

osnr commented Sep 30, 2023

Uh oh!

l3gacyb3ta commented Oct 3, 2023

Uh oh!

l3gacyb3ta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

l3gacyb3ta left a comment

Choose a reason for hiding this comment

Uh oh!

osnr commented Oct 4, 2023

Uh oh!

osnr commented Oct 4, 2023

Uh oh!

Uh oh!