fgviewer: visualize intermediate buffers by poweifeng · Pull Request #9862 · google/filament

poweifeng · 2026-03-31T06:29:50Z

Intermediate Buffer Visualization: Enabled live monitoring of internal FrameGraph render targets directly in the fgviewer web UI.
HTTP Polling Architecture: Switched from WebSocket binary pushes to native <img> polling (/api/image), eliminating severe frontend memory leaks and GC thrashing.
Robust Resource Tracking: Replaced string-based lookups with (ViewId, ResourceId) composite keys to prevent cross-view collisions and ensure accurate reads.
Format Post-Processing: Extracted readback conversions (HDR tonemapping, depth normalization, MSAA downsampling, single-channel expansion) into DebugServer.
UI Polish: Added a live-updating full-screen image modal and explicitly filtered internal debug passes from the Graphviz/JSON exports to prevent DOM thrashing.
Currently Unsupported:
- Mipmaps & Subresources: Reading specific mip levels or array layers is explicitly skipped.
- Shadowmaps: Variance Shadow Maps (VSM) will physically evaluate to 0.0 and appear completely empty/black in scenes without active shadow casters (due to inverted-Z).
- Stencil: Resolving and reading stencil buffer data is not supported by the backend.

show50726 · 2026-03-31T14:15:05Z

It looks great! Thank you!

Couple questions:

Since a texture could be written several times within a frame, is the image shown on the web page always the final state of the texture, or can we specify the timing?
Is it tested on android as well?

poweifeng · 2026-03-31T21:34:47Z

It looks great! Thank you!

Couple questions:

Since a texture could be written several times within a frame, is the image shown on the web page always the final state of the texture, or can we specify the timing?

Is it tested on android as well?

There is no way to specify timing. It's always the result of the buffer after the last pass (dependents on the output of the previous passes). Which buffer is written more than once?
No, it's not tested beyond MacOS + metal/vk. The changes made here are pretty complicated (to get all the formats working). If ok, I'd like to check this in and then iterate on the other backends and platforms.

show50726 · 2026-04-01T02:26:51Z

It looks great! Thank you!
Couple questions:

Since a texture could be written several times within a frame, is the image shown on the web page always the final state of the texture, or can we specify the timing?

Is it tested on android as well?

There is no way to specify timing. It's always the result of the buffer after the last pass (dependents on the output of the previous passes). Which buffer is written more than once?

No, it's not tested beyond MacOS + metal/vk. The changes made here are pretty complicated (to get all the formats working). If ok, I'd like to check this in and then iterate on the other backends and platforms.

Got it! For your question, like Structure Buffer and Bloom Out Texture in your screenshot are both written by 2 passes?

poweifeng · 2026-04-02T04:51:49Z

It looks great! Thank you!
Couple questions:

Since a texture could be written several times within a frame, is the image shown on the web page always the final state of the texture, or can we specify the timing?

Is it tested on android as well?

There is no way to specify timing. It's always the result of the buffer after the last pass (dependents on the output of the previous passes). Which buffer is written more than once?

No, it's not tested beyond MacOS + metal/vk. The changes made here are pretty complicated (to get all the formats working). If ok, I'd like to check this in and then iterate on the other backends and platforms.

Got it! For your question, like Structure Buffer and Bloom Out Texture in your screenshot are both written by 2 passes?

I see, I think different mip levels and layers are written to by different passes, but I'm guessing, for one layer/level in one texture, it's only ever written to once per-frame.

pixelflinger · 2026-04-03T18:34:51Z

+    static void handleFgReadbacks(FEngine& engine, FrameGraph& fg,
+            backend::Handle<backend::HwRenderTarget> viewTarget, uint32_t viewId);
+    static void update(FEngine& engine, FrameGraph& fg, FView const& view);
+    static void tick(FEngine& engine);


some documentation would be nice

pixelflinger · 2026-04-03T18:38:17Z

+#if FILAMENT_ENABLE_FGVIEWER
+    struct ReadbackRequest {
+        using Callback = std::function<void(fgviewer::DebugServer::PixelBuffer, uint32_t, uint32_t,
+                fgviewer::DebugServer::PixelDataFormat, fgviewer::DebugServer::FormatInfo)>;
+        fgviewer::ViewHandle viewId;
+        uint32_t id;
+        utils::CString name;
+        Callback callback;
+    };
+
+    void requestTextureReadback(fgviewer::ViewHandle viewId, uint32_t id,
+            utils::CString const& name,
+            std::function<void(fgviewer::DebugServer::PixelBuffer, uint32_t, uint32_t,
+                    fgviewer::DebugServer::PixelDataFormat, fgviewer::DebugServer::FormatInfo)>&&
+                    callback);
+
+    utils::Mutex mReadbackRequestsMutex;
+    std::vector<ReadbackRequest> mReadbackRequests;
+#endif


does this needs to be on Engine? Could it be on the new FgviewerManager?

moved to FgviewerManager

pixelflinger · 2026-04-03T18:39:42Z

        case VK_FORMAT_D16_UNORM: return PixelDataType::USHORT;
        case VK_FORMAT_D32_SFLOAT: return PixelDataType::FLOAT;
        case VK_FORMAT_X8_D24_UNORM_PACK32: return PixelDataType::UINT;
+        case VK_FORMAT_B10G11R11_UFLOAT_PACK32: return PixelDataType::UINT_10F_11F_11F_REV;


I would move all the VK changes in a different CL (or PR), seems unrelated.

pixelflinger · 2026-04-03T18:47:13Z

 //    fg.forwardResource(fgViewRenderTarget, debug ? debug : input);

-    fg.forwardResource(fgViewRenderTarget, input);
+    FgviewerManager::handleFgReadbacks(engine, fg, viewRenderTarget, view.getViewHandle());


does this have to be done before present()?

IF not, you could hide all this inside the FRameGraph itself.

Like you could call handleFgReadbacks at the beginging of compile(), update() at the end of compile() and tick() at the end of execute().

It would be all contained inside the FrameGraph, and FgviewerManager could become an implementation detail of FrameGraph.

I've moved all the fgviewer logic to FrameGraph

* Intermediate Buffer Visualization: Enabled live monitoring of internal `FrameGraph` render targets directly in the `fgviewer` web UI. * HTTP Polling Architecture: Switched from WebSocket binary pushes to native `<img>` polling (`/api/image`) * Robust Resource Tracking: Replaced string-based lookups with `(ViewId, ResourceId)` composite keys to prevent cross-view collisions and ensure accurate reads. * Format Post-Processing: Extracted readback conversions (HDR tonemapping, depth normalization, MSAA downsampling, single-channel expansion) into `DebugServer`. * UI Polish: Added a live-updating full-screen image modal and explicitly filtered internal debug passes from the Graphviz/JSON exports to prevent DOM thrashing. * WIP: Currently Unsupported: * GL backend: failed with: OpenGL framebuffer error 0x8cd6 (GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT) in "readTexture" at line 4198 * Mipmaps & Subresources: Reading specific mip levels or array layers is explicitly skipped. * Shadowmaps: Variance Shadow Maps (VSM) will physically evaluate to `0.0` and appear completely empty/black in scenes without active shadow casters (due to inverted-Z). * Stencil: Resolving and reading stencil buffer data is not supported by the backend. * WIP: Untested outside of MacOS+metal/vk

poweifeng added the internal Issue/PR does not affect clients label Mar 31, 2026

pixelflinger requested review from pixelflinger and show50726 March 31, 2026 17:26

pixelflinger requested changes Mar 31, 2026

View reviewed changes

Comment thread filament/src/details/Renderer.cpp Outdated

Comment thread filament/src/fg/FrameGraph.cpp Outdated

poweifeng force-pushed the pf/fgviewer-buffer-view branch 2 times, most recently from f16d488 to a1089b8 Compare March 31, 2026 21:32

show50726 reviewed Apr 1, 2026

View reviewed changes

Comment thread filament/src/fg/FgviewerManager.h Outdated

show50726 reviewed Apr 1, 2026

View reviewed changes

Comment thread filament/src/details/Renderer.cpp Outdated

poweifeng force-pushed the pf/fgviewer-buffer-view branch from a1089b8 to 2dc479d Compare April 2, 2026 04:47

poweifeng requested review from pixelflinger and show50726 April 2, 2026 04:52

show50726 approved these changes Apr 2, 2026

View reviewed changes

pixelflinger requested changes Apr 3, 2026

View reviewed changes

poweifeng force-pushed the pf/fgviewer-buffer-view branch 2 times, most recently from 5a16ffe to 3da9dea Compare April 14, 2026 05:59

poweifeng force-pushed the pf/fgviewer-buffer-view branch from 3da9dea to 496a1ef Compare April 14, 2026 06:08

pixelflinger approved these changes Apr 21, 2026

View reviewed changes

Merge branch 'main' into pf/fgviewer-buffer-view

f61f383

poweifeng enabled auto-merge (squash) April 21, 2026 19:15

Merge branch 'main' into pf/fgviewer-buffer-view

72b0f45

poweifeng merged commit 2ae4dc4 into main Apr 21, 2026
18 checks passed

poweifeng deleted the pf/fgviewer-buffer-view branch April 21, 2026 20:30

Conversation

poweifeng commented Mar 31, 2026

Uh oh!

show50726 commented Mar 31, 2026

Uh oh!

Uh oh!

Uh oh!

poweifeng commented Mar 31, 2026

Uh oh!

show50726 commented Apr 1, 2026

Uh oh!

Uh oh!

Uh oh!

poweifeng commented Apr 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants