This is a list of quite a few different improvements that can be made to the GUI.
- Considering officially support ImGUI as a frontend.
ImGUI is a widely used and well supported UI framework for graphics application. It allows custom backend with no extra dependency if you roll your own backend. This means that ImGUI will work on any platform. There are many more good things about ImGUI, and lots of engines and frameworks are using ImGUI as a basis for their frontend. (e.g. all sorts of games, Google Filament, Nvidia Omniverse, etc.)
Example in action:
- Separating GUI backend and frontend
Currently the GUI rendering code, the canvas, and the windowing system is tightly coupled and determined at compile time. This introduces bad separation / abstraction in API, and it makes it very hard to support multiple windowing system on the same platform. For example, on Linux a user may use X11 or Wayland, on Windows a user may want to present to an DirectX surface instead of a Win32 window.
The best solution is to have the backend (windowing system, querying input states, rendering of list of basic primitives, and present image). Then we can have different frontend as well, this can be the existing Taichi GUI widgets, ImGUI, or other things.
- Zero-transfer presentation
If we have the backend and frontend separated, we can have multiple GUI backend options like Vulkan or OpenGL. This also means with the right combination of APIs or API inter-ops, we can have the GUI backend directly present an image from Taichi, or we can have the copy / image transform done in GPU. Currently the image is transferred to the CPU and eventually the windowing system (Windows, X11, Mac Quartz, etc.) will need to transfer the plane back to GPU again. This is an extremely time consuming and wasting process. (The cornell_box example runs at 1200 samples per second if the transfer happens per 5000 samples, but only runs at 300 samples per second if the transfer happens per 10 samples)
Since this is quite an involved request, I can't just have everything built and send a pull request, therefore I'm requesting for comments from the community!
This is a list of quite a few different improvements that can be made to the GUI.
ImGUI is a widely used and well supported UI framework for graphics application. It allows custom backend with no extra dependency if you roll your own backend. This means that ImGUI will work on any platform. There are many more good things about ImGUI, and lots of engines and frameworks are using ImGUI as a basis for their frontend. (e.g. all sorts of games, Google Filament, Nvidia Omniverse, etc.)
Example in action:
Currently the GUI rendering code, the canvas, and the windowing system is tightly coupled and determined at compile time. This introduces bad separation / abstraction in API, and it makes it very hard to support multiple windowing system on the same platform. For example, on Linux a user may use X11 or Wayland, on Windows a user may want to present to an DirectX surface instead of a Win32 window.
The best solution is to have the backend (windowing system, querying input states, rendering of list of basic primitives, and present image). Then we can have different frontend as well, this can be the existing Taichi GUI widgets, ImGUI, or other things.
If we have the backend and frontend separated, we can have multiple GUI backend options like Vulkan or OpenGL. This also means with the right combination of APIs or API inter-ops, we can have the GUI backend directly present an image from Taichi, or we can have the copy / image transform done in GPU. Currently the image is transferred to the CPU and eventually the windowing system (Windows, X11, Mac Quartz, etc.) will need to transfer the plane back to GPU again. This is an extremely time consuming and wasting process. (The cornell_box example runs at 1200 samples per second if the transfer happens per 5000 samples, but only runs at 300 samples per second if the transfer happens per 10 samples)
Since this is quite an involved request, I can't just have everything built and send a pull request, therefore I'm requesting for comments from the community!