Feature request: Easy way to use shaders to process images

[JMS55]

Say you wanted to add a blur effect to your game. This is easy enough to do: Before you submit your frame to pixels for rendering, calculate the blur, and modify the frame. You have direct access to an array of pixels representing your image, and it's simple.

But wait, it's really slow! You could try adding rayon, but indexing an array in parallel is somewhat tricky, and it's still not quite the speed you wanted. Plus, you're operating on the original image, and not the scaled version that pixels generates, so you will end up with a lower quality result.

Instead, since pixels already uses wgpu, why not do this on the GPU? The problem is that pixels::RenderPass involves learning a ton about wgpu. Instead, I propose a simpler api:

/// Provides a simple way to do parallel processing of images on the GPU
/// 
/// `shader` is a SPIR-V shader, for instance generated with pixels::include_spv!(), that matches the below format
///
/// ```glsl
/// #version 450
/// 
/// layout(set = 0, binding = 0, rgba8ui) uniform uimage2D pixels; // Read only texture generated by the default pixels renderer
/// layout(set = 0, binding = 1, rgba8ui) uniform uimage2D previous_pass; // Read only texture of the previous pass's result
/// layout(set = 0, binding = 2, rgba8ui) uniform uimage2D output; // Writeable texture, write your results to it. For the last render pass, this would be the swapchain's texture.
/// 
/// void main() {
///     // TODO: Show how to get the image size here (all textures would be the same size as the swapchain output)
///     // TODO: Show how to get the current pixel's coordinates here (x, y) (0..IMAGE_WIDTH, 0..IMAGE_HEIGHT)
///     // TODO: Show how to read a pixel (r, g, b, a) (u8, u8, u8, u8) at a given coordinate from a texture
///     // TODO: Show how to write a pixel to output
/// }
/// ```
fn add_simple_render_pass(shader: Vec<u32>) {}

Unanswered questions:

1. I'm assuming this would use compute shaders. Can we write to the swapchain directly like this? If not, maybe we could have the user supply a fragment shader instead. Pixels would provide the vertex shader.
2. Do we want to give more control of input/output textures, and how each pass is executed, instead of just a chain? Maybe look at how specs allows specifying parallel systems for this.
3. Figure out how to fill in the TODO's in the shader
4. How does this integrate with pixels' regular render passes?

[parasyte]

Yep, render passes are designed to provide the full power of wgpu. I agree there are some situations where the default pipeline is good enough and only the fragment shader needs to be changed. 👍

I can provide some extra context for your questions below, but might not have the best answers:

1. A fragment shader can perform the blur fairly easily by multi-sampling the input texture. A compute shader could do pixel buffer transformations (or generate new pixel buffers entirely) without going through the raster pipeline. But there are several open questions relating to texture sizes and formats, and the compute shader would probably be responsible for scaling.

2. Yes, but I would treat that as in independent feature request.

3. This is the tricky bit mentioned above. The raster pipeline handles all of these details for us, but with a compute shader, the inputs would have to be managed by the pipeline and passed to the shader as inputs; "Here's a pixel buffer as a 1-dimensional array, here's the texture geometry for the array, here's an output array for the new texture, here's the texture geometry for the output buffer, ..."

   It feels like a lot of work, and I don't know the specific details of benefits of a compute shader over a fragment shader. I assume the reason is for performance, but how much better does this perform than normal texture sampling?

4. Anything that implements Renderer can be used as a render pass. But for a "I don't know wgpu" feature, we would need to alter the default renderer pipeline.

   • For computer shaders, a new optional pipeline stage would be added that runs the custom compute shader before any of the raster stages.

   • For fragment shaders, a new optional pipeline stage would be added that runs the custom fragment shader after the pixel buffer is scaled but before it is clipped.

[JMS55]

Thanks for the feedback! I'm prototyping this with just wgpu right now. Once I get it working, I'll start adding this to a fork of pixels.

I believe I've figured out how to get compute shaders to work, so we can use those. That should be simpler (for the user) and faster than render passes.

I'm going ahead with making them work in a chain, agreed that a dependency system is too complicated for now.

The order things should end up running is:

1. Pixels default renderer
2. Simple passes (with 1. as input, and the previous simple pass as input)
3. Regular passes (with the result of 2. (or 1. if 2. doesn't exist) as input for the first, and then the previous render pass for the rest)

[parasyte]

A simple way to get more performance on systems with multiple GPUs is to request the faster one. By default, pixels will select the power-efficient GPU. You can request the high performance GPU with PixelsBuilder::request_adapter_options(). An example is shown below.

use pixels::wgpu::{PowerPreference, RequestAdapterOptions};

builder.request_adapter_options(RequestAdapterOptions {
    power_preference: PowerPreference::HighPerformance,
    compatible_surface: None,
});

[JMS55]

Here's the API I've come up with (somewhat pseduo-code) to add a simpler render pass method, and support chaining of render passes. Note that it's a breaking change to the RenderPass API. What do you think?

If there are no extra render passes, DefaultRenderer will render directly to the screen. Otherwise each render pass is setup to write to a texture, forming a chain, and then a final CopyPass runs at the end, to copy the texture to the screen (needed because compute passes cannot write directly to the swapchain afaik, or at least not the way I'm doing it with simple render passes).

struct DefaultRenderer {
    output_texture: Option<TextureView>,
}

struct RenderPass {
    pixels_default_renderer_output_texture: TextureView,
    previous_pass_texture: TextureView,
    output_texture: TextureView,
}

struct CopyPass {
    previous_pass_texture: TextureView,
}

fn create_pixels(simple_render_passes: Vec<SimpleRenderPass>, render_passes: Vec<RenderPass>) {
    let has_no_additional_render_passes =
        simple_render_passes.is_empty() && render_passes.is_empty();

    // Added to Pixels
    let default_renderer = DefaultRenderer {
        output_texture: if has_no_additional_render_passes {
            None
        } else {
            Some(TextureView::new())
        },
    };

    let pixels_default_renderer_output_texture = default_renderer.output_texture.unwrap();
    let mut previous_pass_texture = default_renderer.output_texture.unwrap();

    for render_pass in simple_render_passes.extend_with(render_passes) {
        let output_texture = TextureView::new();
        // Added to Pixels
        let render_pass = RenderPass {
            pixels_default_renderer_output_texture,
            previous_pass_texture,
            output_texture,
        };
        previous_pass_texture = output_texture;
    }

    if !has_no_additional_render_passes {
        // Added to Pixels
        let copy_pass = Some(CopyPass {
            previous_pass_texture,
        });
    }
}

fn render() {
    let swapchain_texture = get_swapchain_texture();

    match self.default_renderer.output_texture {
        None => self.default_renderer.render(swapchain_texture),

        Some(output_texture) => {
            self.default_renderer.render(output_texture);

            for render_pass in self.render_passes {
                render_pass.render();
            }

            self.copy_pass.unwrap().render(swapchain_texture);
        }
    };
}

[parasyte]

I suspect #95 and #96 fixes this. What do you think @JMS55 ?

[JMS55]

I was originally thinking this would be more of a way to just specify a compute shader built off of a template that pixels would provide, and have pixels take care of the rest of the setup like setting up textures and copying to/from other textures. I don't think it's very realistic anymore though, and the new render api is much better, so I'm fine to close this now.