Meshlet rendering (initial feature)

[JMS55]

Objective

• Implements a more efficient, GPU-driven (Gpu Driven Rendering By Default #1342) rendering pipeline based on meshlets.
• Meshes are split into small clusters of triangles called meshlets, each of which acts as a mini index buffer into the larger mesh data. Meshlets can be compressed, streamed, culled, and batched much more efficiently than monolithic meshes.

Misc

• Future work: Meshlet tracking issue #11518
• Nanite reference: https://advances.realtimerendering.com/s2021/Karis_Nanite_SIGGRAPH_Advances_2021_final.pdf
  Two pass occlusion culling explained very well: https://medium.com/@mil_kru/two-pass-occlusion-culling-4100edcad501

[superdump]

Awesome!

Two-pass occlusion culling should not be tied to meshlets specifically. We should just do that separately.

[JMS55]

I don't see how they could.

The meshlet system is already setup to do occlusion culling, basically. I already upload the bounding sphere information per meshlet, and have a culling and indirect draw pipeline in place.

I don't see how we could share occlusion culling passes with meshlet meshes and regular meshes without basically duplicating all of the meshlet system and indirect draw setup, bounding info, etc, but changing it to operate on whole meshes instead of meshlets which is just more inefficient. If we were going that route, there would be no point in doing all the work for meshlets in the first place.

[BeastLe9enD]

Gate meshoptimizer dependency behind a cargo feature, or rewrite it in Rust.

If you are looking for a pure-rust implementation, have a look at https://github.com/yzsolt/meshopt-rs

[JMS55]

Unfortunately meshopt-rs does not support the meshlet APIs (it's ported from an older version of meshoptimizer), which is why I'm using a different crate.

[JMS55]

Plan for materials (assuming opaque meshes only, will extend to other passes later):

1. In extract_meshlet_meshes, create a Vec of all meshlet mesh entities
2. In a new Queue system, as part of MaterialPlugin for each material, for each entity from step one check if it's using the material via RenderMaterialInstances, and if so, push the material index (gotten from the hashmap in step 3) to a new Vec in MeshletGpuScene
3. In a new PrepareAssets system that runs after prepare_materials::<M>, as part of MaterialPlugin for each material, upload changed opaque materials to MeshletGpuScene
   3a. Queue new render pipeline for the material, store pipeline ID + material bind group in a Vec
   3b. Use a HashMap to store index of material in the Vec, hashing based on material AssetId
   3c. Could skip materials not used by any meshlet mesh, at the cost of extra queries
4. Optionally (may improve GPU coherence, at the cost of CPU time) in prepare_meshlet_per_frame_resources, sort per-frame-buffers by the material index before uploading
5. In prepare_meshlet_per_frame_resources, instead of a single DrawIndexedIndirect item, make draw_command_buffer hold M items equal to the total material count
   5a. Each DrawIndexedIndirect::base_index needs to be equal to the total count of possible indices (after instancing) for that material
   5b. We can skip draws for materials that aren't used by any meshlet mesh entities
6. In cull_meshlets, for each meshlet, index into the draw command array based on the material index to get the DrawIndexedIndirect for the meshlet's material, and then write to location draw_index_buffer_start + draw_indirect_command.base_index + offset in the index buffer

[JMS55]

This PR is usable atm, and has large chunks of code ready. The plan is once 0.12 is released, I'll start opening smaller PRs with parts of these changes. The goal is to incrementally merge chunks of code for meshlet rendering, instead of one big PR with all the changes.

[JMS55]

Plan to support (occlusion culling, visibility buffer, shadows, forward + prepass, deferred):

• Each view will get a visibility buffer (not to be confused with the vbuffer texture)
  • Stores visible/not visible in view as a boolean for each meshlet, persisted across two frames (previous and current)
  • Instances of meshlets (thread_meshlet) can get their previous visibility via previous_visibility[previous_visibility_id[thread_id]], where previous_visibility_id is uploaded on the CPU during the current frame, using a resource holding the data from the previous frame (might be a better way to do this without the extra indirection and previous_visibility_id buffer...)
• MeshletVBufferNode - Render depth, vbuffer (thread_id + triangle_id), and material depth
  • PreviousOccluderPreparePass - Take meshlets visible last frame, as indicated by previous_visibility, write index buffer to render them
  • VBufferRenderPass1 - Render the 3 node outputs using a single draw_indirect_indexed()
  • GenerateHzbPass - Take depth buffer generated so far, downscale several times to create a hierarchical depth buffer
  • CullPass - Take all meshlets, frustum cull, occlusion cull against hzb, write visibility to visibility buffer for next frame, and if visible write index buffer to render them
  • VBufferRenderPass2 - Render the 3 node outputs using a single draw_indirect_indexed()
• MeshletShadowMapNode/MeshletPrepassNode/MeshletOpaque3dMainPassNode
  • For each material, draw a single triangle/quad using the material depth trick, reconstruct vertex properties, and then shade the fragment

The messiest part is having to use a previous/next visibility buffer with separate indices, instead of being able to assume [object/instance/entity] index is constant across frames, and use a single read_write visibility buffer. Will have to think more on how to do this.

[JMS55]

Additional complication: Vertex data can't be provided to the fragment shader via the vertex output. Fragment shader will need to read the vbuffer pixel, and load all the meshlet and then vertex data. This means we need to modify the fragment shader.

I'll probably have to re-write shaders using naga_oil somehow to load the vbuffer data to construct the VertexOutput, instead of directly reading it as the fragment input.

[JMS55]

Useful reference for visbuffer barycentrics, partial derivatives, and the other complicated stuff: https://github.com/JuanDiegoMontoya/Frogfood/blob/main/data/shaders/visbuffer/VisbufferResolve.frag.glsl

[JMS55]

Next step is to emit "material id" to a depth texture. Visbuffer fragment shader will output material ID to a R16Uint color attachment, and then an extra fullscreen triangle render pass will read that texture and write to a depth target.

[IceSentry]

So I've definitely didn't check that all the meshlet code is correct, but it works. I've tried to make sure to the best of my ability that it doesn't affect unrelated code if meshlets aren't used.

I'm approving this with the context that it is an experimental feature that will still be iterated on because I think merging this now will make collaboration easier.

[cart]

This is definitely exciting work! The approach (while complicated) is surprisingly straightforward given what its doing and what we get. Very curious to see how this performs if you've already done testing (not blocking: I know theres a lot of changes planned).

Makes me want to consider "fragment shader abstractions" so people can write custom fragment shaders that "just work" with meshlets / deferred / prepass / normal forward without needing a bunch of ifdefs + import knowledge. Doesn't seem like a particularly challenging problem to solve given that its mostly import differences and a bit of setup code. (obviously not something we should handle in this PR)

[JMS55]

Performance varies wildly depending on the scene. There's considerable base overhead, but it should scale up to much higher density scenes, especially with lots of occlusion. It's really meant for rendering really dense, ridiculously high resolution scenes where the goal is to render 1 triangle per pixel, so you're never limited by lack of geometric detail. We're not there yet due to lack of lods or software raster though. So performance can be quite good in certain scenes with lots of occlusion atm, but it can't render higher resolution stuff like it's intended to, and on simpler scenes the base overhead dominates compared to bevy's standard renderer. I don't have specific numbers to give ATM, I'll do a lot more in depth comparisons once we have LODs and software raster in.

I agree on the material shader needing some abstraction in the future. You kind of can do that already with our existing ifdefs, but it's not the best. One complication is that derivatives need to be manually calculated when using the visbuffer. We'd probably need some kind of automatic chain rule applier that differentiates functions, like Slang has.

[github-actions]

You added a new feature but didn't update the readme. Please run cargo run -p build-templated-pages -- update features to update it, and commit the file change.