[Flutter GPU] Raster encoding. First triangle! #48314

bdero · 2023-11-22T15:17:47Z

First triangle, in the framework! 🎉

Screen.Recording.2023-11-22.at.6.04.13.AM.mov

Adds shader libraries, pipelines, command buffers, render passes, etc.

Light pipelines/shader objects. No optimization yet, pipeline warming to come.
"Dynamic" command style. Don't re-send bindings if you don't need to. Essentially: [Impeller] Make RenderPass commands more granular. flutter#133179
No need to explicitly encode passes.
Minimal descriptor usage.
Nothing is async, except for the optional command buffer completion callback.

It took a bunch of experimenting to get here, but I think things are starting to look pretty neat. :)

Todo:

Land the shader bundle format/remove the testing hacks & fixtures that piggyback off of the runtime effect system.
Add remaining calls for blend config, clearing bindings, etc.
Inconsistent error handling patterns that need cleanup.
Maybe: Surface exceptions for validation errors.
Handle the texture usage bitmask more elegantly.

This is the test code that's rendering this in the framework:

  void paint(Canvas canvas, Size size) {
    /// Allocate a new renderable texture.
    final gpu.Texture? renderTexture = gpu.gpuContext.createTexture(
        gpu.StorageMode.devicePrivate, 300, 300,
        enableRenderTargetUsage: true,
        enableShaderReadUsage: true,
        coordinateSystem: gpu.TextureCoordinateSystem.renderToTexture);

    /// Create the command buffer. This will be used to submit all encoded
    /// commands at the end.
    final commandBuffer = gpu.gpuContext.createCommandBuffer();

    /// Add a render pass encoder to the command buffer so that we can start
    /// encoding commands.
    final encoder = commandBuffer.createRenderPass(
        colorAttachment: gpu.ColorAttachment(texture: renderTexture));

    /// Load a shader bundle asset.
    final library = gpu.ShaderLibrary.fromAsset('TestLibrary')!;

    /// Create a RenderPipeline using shaders from the asset.
    final vertex = library['UnlitVertex']!;
    final fragment = library['UnlitFragment']!;
    final pipeline = gpu.gpuContext.createRenderPipeline(vertex, fragment);

    encoder.bindPipeline(pipeline);

    /// Append quick geometry and uniforms to a host buffer that will be
    /// automatically uploaded to the GPU later on.
    final transients = gpu.HostBuffer();
    final vertices = transients.emplace(float32(<double>[
      -0.5, -0.5, //
      0, 0.5, //
      0.5, -0.5, //
    ]));
    final color = transients.emplace(float32(<double>[0, 1, 0, 1])); // rgba
    final mvp = transients.emplace(float32Mat(Matrix4(
          1, 0, 0, 0, //
          0, 1, 0, 0, //
          0, 0, 1, 0, //
          0, 0, 0.5, 1, //
        ) *
        Matrix4.rotationX(time) *
        Matrix4.rotationY(time * seedX) *
        Matrix4.rotationZ(time * seedY)));

    /// Bind the vertex data. In this case, we won't bother binding an index
    /// buffer.
    encoder.bindVertexBuffer(vertices, 3);

    /// Bind the host buffer data we just created to the vertex shader's uniform
    /// slots. Although the locations are specified in the shader and are
    /// predictable, we can optionally fetch the uniform slots by name for
    /// convenience.
    final mvpSlot = pipeline.vertexShader.getUniformSlot('mvp')!;
    final colorSlot = pipeline.vertexShader.getUniformSlot('color')!;
    encoder.bindUniform(mvpSlot, mvp);
    encoder.bindUniform(colorSlot, color);

    /// And finally, we append a draw call.
    encoder.draw();

    /// Submit all of the previously encoded passes. Passes are encoded in the
    /// same order they were created in.
    commandBuffer.submit();

    /// Wrap the Flutter GPU texture as a ui.Image and draw it like normal!
    final image = renderTexture.asImage();

    canvas.drawImage(image, Offset(-renderTexture.width / 2, 0), Paint());
  }

UnlitVertex:

uniform mat4 mvp;
uniform vec4 color;

in vec2 position;
out vec4 v_color;

void main() {
  v_color = color;
  gl_Position = mvp * vec4(position, 0.0, 1.0);
}

UnlitFragment:

in vec4 v_color;
out vec4 frag_color;

void main() {
  frag_color = v_color;
}

zanderso

When we're comfortable with the API, there will need to be very thorough docstrings on the public API.

zanderso · 2023-11-22T15:56:44Z

lib/gpu/fixtures.cc

@@ -0,0 +1,124 @@
+#include "flutter/lib/gpu/fixtures.h"


This file needs a copyright header, and a comment with instructions on how to regenerate the arrays below. Instead of hardcoding this file, consider generating at build time like in #46862.

Done. This is a silly temp measure for provisioning a shader lib before I land the new shader bundle format.

bdero · 2023-11-22T22:08:27Z

When we're comfortable with the API, there will need to be very thorough docstrings on the public API.

Agreed. I'd like to continue foregoing docstrings for now while we feel out the rest of the API and have some representative examples that we're happy with.

jonahwilliams

I'm having an increasing fear that added functionality here with light testing will make the refactoring work we'll need to do in the engine later this year and next much harder.

jonahwilliams · 2023-11-23T00:42:22Z

lib/gpu/lib/src/formats.dart

+enum ShaderStage {
+  unknown,
+  vertex,
+  fragment,
+  tessellationControl,
+  tessellationEvaluation,
+  compute,
+}


Can we avoid exposing stages we don't have a plan for yet? I think this could just be vertex and fragment.

jonahwilliams · 2023-11-23T00:42:48Z

lib/gpu/lib/src/render_pass.dart

+  ColorAttachment(
+      {this.loadAction = LoadAction.clear,
+      this.storeAction = StoreAction.store,
+      this.clearColor = const ui.Color(0),


uber-nit

Suggested change

this.clearColor = const ui.Color(0),

this.clearColor = const ui.Color(0x00000000),

jonahwilliams · 2023-11-23T00:44:26Z

lib/gpu/lib/src/render_pass.dart

+    switch (bufferView.buffer.runtimeType) {
+      case DeviceBuffer:
+        success = _bindUniformDevice(
+            slot.shaderStage.index,
+            slot.slotId,
+            bufferView.buffer as DeviceBuffer,
+            bufferView.offsetInBytes,
+            bufferView.lengthInBytes);
+        break;
+      case HostBuffer:
+        success = _bindUniformHost(
+            slot.shaderStage.index,
+            slot.slotId,
+            bufferView.buffer as HostBuffer,
+            bufferView.offsetInBytes,
+            bufferView.lengthInBytes);
+        break;
+      default:
+        throw Exception("Invalid buffer type");
+    }


Patterns like this indicate this should probably be a method on Buffer

Normally I'd agree, but this needs to dispatch to different native methods tied to the RenderPass handle. So then Buffer would end up having something like a reverse _bindTo(gpu.RenderPass, gpu.BufferView) that then calls back into the right RenderPass native method, which feels a lot less straight forward.

It's a visitor pattern 🙂

Ok ok, I'll bite. Done.

jonahwilliams · 2023-11-23T00:45:09Z

lib/gpu/lib/src/render_pass.dart

+  }
+
+  void bindVertexBuffer(BufferView bufferView, int vertexCount) {
+    switch (bufferView.buffer.runtimeType) {


jonahwilliams · 2023-11-23T00:45:45Z

lib/gpu/lib/src/shader.dart

+  int slotId;
+  ShaderStage shaderStage;


Both of these fields feel like they should be final and the class const

jonahwilliams · 2023-11-23T00:48:43Z

lib/gpu/lib/src/shader_library.dart

+
+  ShaderLibrary._();
+
+  Shader? operator [](String shaderName) {


nit: use a method for this. operators, getters, and [] feel cheap even when they aren't.

I think this is sufficiently cheap? It just grabs a handle to an already existing object on the C++ side.

bdero · 2023-11-23T01:18:12Z

I'm having an increasing fear that added functionality here with light testing will make the refactoring work we'll need to do in the engine later this year and next much harder.

Oh, it'll definitely break, but I think it's OK for Flutter GPU to break at this early stage while we're feeling out the API.

If you're working on API changes in the HAL, I actually think it would be acceptable to just remove the Flutter GPU target from the build script if fixing it causes too much friction, and I can come around later to fix it up.

…ine#48314)

…138995) flutter/engine@b50fc4a...7743220 2023-11-25 bdero@google.com [Flutter GPU] Raster encoding. First triangle! (flutter/engine#48314) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC jacksongardner@google.com,rmistry@google.com,zra@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

First triangle, in the framework! 🎉 Adds shader libraries, pipelines, command buffers, render passes, etc. * Light pipelines/shader objects. No optimization yet, pipeline warming to come. * "Dynamic" command style. Don't re-send bindings if you don't need to. Essentially: flutter/flutter#133179 * No need to explicitly encode passes. * Minimal descriptor usage. * Nothing is async, except for the optional command buffer completion callback. It took a bunch of experimenting to get here, but I think things are starting to look pretty neat. :) Todo: * Land the shader bundle format/remove the testing hacks & fixtures that piggyback off of the runtime effect system. * Add remaining calls for blend config, clearing bindings, etc. * Inconsistent error handling patterns that need cleanup. * Maybe: Surface exceptions for validation errors. * Handle the texture usage bitmask more elegantly.

…lutter#138995) flutter/engine@b50fc4a...7743220 2023-11-25 bdero@google.com [Flutter GPU] Raster encoding. First triangle! (flutter/engine#48314) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC jacksongardner@google.com,rmistry@google.com,zra@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

bdero · 2024-01-03T01:04:27Z

This is part of the Flutter GPU MVP umbrella bug: flutter/flutter#130921

bdero added the e: impeller label Nov 22, 2023

bdero requested review from chinmaygarde, zanderso and jonahwilliams November 22, 2023 15:17

bdero self-assigned this Nov 22, 2023

bdero marked this pull request as ready for review November 22, 2023 15:24

zanderso reviewed Nov 22, 2023

View reviewed changes

bdero force-pushed the bdero/flutter-gpu-raster branch 2 times, most recently from f5aa61c to c108fb9 Compare November 23, 2023 00:28

jonahwilliams approved these changes Nov 23, 2023

View reviewed changes

bdero added 8 commits November 24, 2023 02:01

[Flutter GPU] Add shader libraries.

f4657d2

[Flutter GPU] Add CommandBuffer and RenderPass.

7e222b5

[Flutter GPU] Add CommandBuffer submission and completion callback

3db67ef

[Flutter GPU] Start of pipelines.

39a9ded

[Flutter GPU] Command encoding.

964dba8

license header

eb9e0b7

Licenses

3862ae6

build fail

40da2c0

bdero force-pushed the bdero/flutter-gpu-raster branch from c108fb9 to 40da2c0 Compare November 24, 2023 10:02

bdero added 3 commits November 24, 2023 03:14

Build failures

d5bc613

Licenses

73cf7e1

Jonah review

9c22658

bdero force-pushed the bdero/flutter-gpu-raster branch from 39fd774 to 9c22658 Compare November 25, 2023 01:30

bdero merged commit 7743220 into flutter:main Nov 25, 2023

engine-flutter-autoroll mentioned this pull request Nov 25, 2023

Roll Flutter Engine from b50fc4ad6820 to 77432202f766 (1 revision) flutter/flutter#138993

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 25, 2023

77432202f [Flutter GPU] Raster encoding. First triangle! (flutter/eng…

60edbcf

…ine#48314)

engine-flutter-autoroll mentioned this pull request Nov 25, 2023

Roll Flutter Engine from b50fc4ad6820 to 77432202f766 (1 revision) flutter/flutter#138995

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 25, 2023

77432202f [Flutter GPU] Raster encoding. First triangle! (flutter/eng…

c63a8b9

…ine#48314)

	this.clearColor = const ui.Color(0),
	this.clearColor = const ui.Color(0x00000000),

[Flutter GPU] Raster encoding. First triangle! #48314

[Flutter GPU] Raster encoding. First triangle! #48314

Uh oh!

Conversation

bdero commented Nov 22, 2023

Uh oh!

zanderso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdero commented Nov 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonahwilliams left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bdero commented Nov 23, 2023

Uh oh!

bdero commented Jan 3, 2024

Uh oh!

Uh oh!

bdero commented Nov 22, 2023 •

edited

Loading