Vulkan in Processing (draft)

[TeoJT]

(I'm new to PRs so let me know if you see any mistakes ^^")

This is my attempt at making a Vulkan-based renderer for Processing, using LWJGL to access a Vulkan binding. It's part of my final year dissertation where I compare OpenGL and Vulkan renderer speeds so a lot of code is still very messy.

I was also originally planning to release it as a library for Processing, but it appears that moving away from JOGL/OpenGL is now on Processing's roadmap (#881)

The new renderer can be accessed by using PV2D/PV3D in the size() function:

void setup() {
  size(512, 512, PV2D);
}

It works by translating OpenGL commands (through the PGL abstraction layer) to Vulkan commands through a thin OpenGL-to-Vulkan translation layer (under a new package named GL2VK). This layer is optimised for Processing (mostly the immediate and retained modes in Processing). These commands are passed to separate threads (called ThreadNodes) which offload the main thread from potentially expensive operations.
There is also a shader converter which translates OpenGL shaders to Vulkan shaders.

Also, it uses GLFW to create a window and receive mouse inputs... no AWT!

[hx2A]

Wow, @TeoJT , this is an amazing PR!!! Thank you for your hard work here.

It looks like your PGraphicsVulkan class inherits from PGraphicsOpenGL to benefit from all of the geometry and style functionality as well as the tessellation code, and then intercepts what would otherwise go through PGL to use Vulkan instead. Is that correct?

You mention mouse events, does it handle keyboard events also?

How much testing have you done? In general, does it work well?

I can't wait to give this a try!

[TeoJT]

Thanks for the kind words @hx2A :)

You're correct; I effectively haven't touched any (or at least most) of the PGraphicsOpenGL code and linked PGL to GL2VK, which uses Vulkan instead.

It only uses some quick 'n dirty code for mouse events and there's no keyboard or scrollwheel events sadly. The window also cannot be resized yet.

I've mostly tested using the performance example sketches;
DynamicParticlesImmediate, Esfera, LineRendering, QuadRendering, StaticParticlesImmediate, TextRendering all work perfectly.
There's a bug with retained mode so DynamicParticlesRetained, CubicGridRetained, and StaticParticlesRetained work but don't display correctly. CubicGridImmediate doesn't display correctly too for some reason.
Also MeshTweening works which proves that custom shaders works to an extent.

It generally works well on basic sketches as long as you don't use PGraphics, and make sure you call background(0) in the draw loop. It performs on average x1.5 faster than the OpenGL renderer, sometimes even higher.

Let me know how it goes for you!

[TeoJT]

Update- just added mouse clicking and keyboard input just now with GLFW.

[tychedelia]

Hi @TeoJT, stumbled across this pr. Wow! This is a pretty awesome exploration and implementing even a portion of OpenGL in Vulkan is pretty impressive. I'm wondering if you've looked at prior art here like Zink in Gallium? I'm not super familiar with their work, but it might be a good source of inspiration. I'm also wondering about running on MoltenVK which would be essential for macOS and might be a pain for managing the Vulkan dependency.

Looking at the code, I'm noticing some synchronization problems that may cause issues. For example, GraphicsBufffer doesn't appear to do any synchronization with the GPU timeline to ensure resources aren't in use -- in general, this isn't a kind of check that can be performed entirely on the CPU. While it may work in simple cases, it's brittle and can easily cause undefined behavior in the driver. Dealing with synchronization in Vulkan is really hard and can be one of the difficult parts of writing a Vulkan backend. If you haven't already, I'd suggest running things with validation layers enabled in the Vulkan SDK and see what warnings pop up. I'd additionally suggest the following two resources which are great for understanding some of the nuances in Vulkan synchronization:

• https://www.khronos.org/assets/uploads/developers/presentations/Ensure_Vulkan_Synchronization_Vulkanised_Oct_2021.pdf
• https://themaister.net/blog/2019/08/14/yet-another-blog-explaining-vulkan-synchronization/

There are also some synchronization issues on the CPU. For example, in ThreadNode, there's a few TOCTOU race conditions where multiple threads may write to a resource in-between reading and setting an atomic, for example openCmdBuffer on lines 378-392 and getNextCMDIndex where multiple threads could race between cmdID get and set. In general, you want to use proper concurrency primitives like mutexes for critical sections, or lock-free data structures like concurrent queues for communication between threads, rather than trying to build your own synchronization with atomics and busy-waiting. Plain atomics usually aren't enough to ensure thread safety.

The idea of doing some kind of OpenGL translation is really cool. Given the dependency Processing has on OpenGL it would definitely be an awesome way to ensure that people can migrate to a new rendering backend without too many breaking changes. I'm definitely curious to know if other people have had success with this strategy in other contexts? Full OpenGL conformance may be a big task. This is the kind of migration that would definitely benefit from learning from other projects. Hope you keep up with graphics work post graduation. It's a really fun space to work in.

[tychedelia]

Also worth noting that there's been some active movement on Java Vulkan bindings in https://github.com/chuigda/vulkan4j which I've been following.

[TeoJT]

Thanks for the feedback tychedelia, I've definitely learned quite a bit just from your message alone! I did avoid synchronisation to avoid potential slowdowns, since my goal was speed. My attempted solution was to make a separate VBO instead of waiting for a previous buffering operation/draw call to complete. It's brittle and it's no wonder it crashes all the time, haha.

I haven't looked much at Zink but I have looked into Google ANGLE. Speaking of, I'm going to try using ANGLE to move the core away from JOGL; it's not going to be faster than native OpenGL (previous studies seem to suggest that), but it's definitely going to be more stable, especially with sketches that use PGL.

One idea I have is to try to access Vulkan directly from ANGLE, skipping the translation, hopefully getting both backward compatibility (e.g. sketches that use PGL) and performance (the renderer can be rewritten to use Vulkan directly). But I have no idea where to start or if it's even possible given that I'm still quite new to Vulkan. If not, I'll continue working on this renderer as a separate library rather than a replacement to OpenGL.

Also I think it's worth pointing out that during benchmarking, a big bottleneck was single-threaded parts like the tessellator, which I didn't make multithreaded since it wasn't the focus in my study, but that could potentially make things like immediate mode rendering much faster.

Again thanks for the feedback and please do let me know if you spot anything else or know of any other frameworks/libraries that could help!

[tychedelia]

Also I think it's worth pointing out that during benchmarking, a big bottleneck was single-threaded parts like the tessellator, which I didn't make multithreaded since it wasn't the focus in my study, but that could potentially make things like immediate mode rendering much faster.

Yup, tessellation is also a huge bottleneck in Nannou, which has a very similar immediate mode architecture to Processing. This has always been the big point of slowness compared to anything we do on the GPU. Modern hardware is designed to render billions of blades of grass in AAA games or whatever and most of the stuff that creative coding throws at it is pretty trivial in that respect.

One thing I've been wondering about for Nannou is GPU driven tessellation using compute shaders. So basically, we'd just send the paths for each draw primitive to the GPU and populate the vertex buffer there. Linebender has been doing a lot of research into GPU driven font rendering and I've often wondered if we could learn anything from their work in this respect.

Me and my colleagues have also discussed switching from a path tessellation strategy to sending SDFs, but this would represent a pretty substantial breaking change to the rendering model obviously! I love the idea of a Processing-like library for raymarching though.

Again thanks for the feedback and please do let me know if you spot anything else or know of any other frameworks/libraries that could help!

So awesome that you're continuing to look into rendering stuff! One thing I'm really curious about is using WebGPU (either via dawn or wgpu). I'm most familiar with wgpu, but it handles most of the complicated resource tracking and while WASM as a Java compilation target isn't fully baked yet, I think it represents the best truly cross-platform replacement for OpenGL.