Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[doc] GPU-based GUI API design #68

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from
Draft

[doc] GPU-based GUI API design #68

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1d63690
begin new gui doc
AmesingFlank Jul 6, 2021
544dcac
wip
AmesingFlank Jul 7, 2021
188862c
change camera api
AmesingFlank Jul 7, 2021
1390bd8
drawing scene on canvas
AmesingFlank Jul 7, 2021
1e2931b
small api changes
AmesingFlank Jul 14, 2021
da24e82
wip
AmesingFlank Jul 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
162 changes: 162 additions & 0 deletions website/docs/lang/articles/misc/new_gui.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
---
sidebar_position: 1

---

# New UI system

A new UI system is to be added to Taichi. The new GUI system will use GPU for rendering, which will enable it to be much faster and to render 3d scenes. This doc describes the APIs to be implemented.

A few design principles (to be expanded):

* Immediate mode GUI components.
* When rendering geometries, users will supply `taichi` fields as input. The implementation will avoid any GPU<->CPU memcpy.

## Creating a window

`ti.ui.Window(name, res)` creates a window. If `res` is scalar, then the width will be equal to the height.Example:

```python
window = ti.ui.Window('Window Title', (640, 360))
```

This comment was marked as outdated.

Sign in to view

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're looking at an older version of this file?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, therefore some of the comments are folded. Some other parts marked "outdated" by github still applies but the position in the original document has changed or the object calling on has changed, but the idea is the same. Sorry about this, I made these comments a while ago and did not notice they are not published


There're three types of objects that can be displayed on a `ti.ui.Window`:

* 2D Canvas, which can be used to draw simple 2D geometries such as circles, triangles, etc.
* 3D Scene, which can be used to render 3D meshes and particles, with a configurable camera and light sources.
* Immediate mode GUI components, buttons, textboxes, etc.

## 2D Canvas

### Creating a canvas

```python
canvas = window.get_canvas()
```
this retrieves a `Canvas` object that covers the entire window.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you squint at canvas a bit, it is basically a 2d renderer that draws on a 2d texture / image. So maybe canvas should be created from ti.gui? As for the resolution, it seems canvas is always rendering vector objects, it might be fine to assume the canvas coordinate space is (0..1, 0..1)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So maybe canvas should be created from ti.gui?

I had a discussion with @k-ye about this. The reasoning is that we do not want the users to have to write canvas.render(...) and then write something like window.render(canvas).

As for the resolution, it seems canvas is always rendering vector objects, it might be fine to assume the canvas coordinate space is (0..1, 0..1)

Yes that is the current design. When users call canvas.triangles(...) etc. the ti vertex coordinates and other stuff should be in the [0,1]*[0,1] space.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You would still need to call window.render(canvas) anyways right? Otherwise how do the user decide whether the canvas is drawn on top of everything else or behind things like UI?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to have a simple model: UI widgets are always on top of the canvas.


### Drawing on the canvas

```python
canvas.clear(color)
canvas.triangles(a,b,c,color)
canvas.triangles_indexed(positions,indices,color)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @AmesingFlank ! I wonder could we also use triangles as the function name. You could determine whether drawing triangles with indices by evaluating the function parameters.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. That could work.

canvas.circles(centers,radius,colors)
canvas.lines(start_positions,end_positions,colors)
canvas.set_image(image)
```

The positions/centers of geometries will be represented as floats between 0 and 1, which indicate relative positions on the canvas.



## 3D Scene

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A scene graph system may be the best, making it extensible, relatively easy to use, while being performance & GPU friendly.

Suggested change
# Renderer can render a scene / object
renderer = ti.ui.Renderer()
# Scenes are made up from objects / nodes, each node has a transform matrix and a list of children. The root node is called scene.
scene = ti.ui.Scene()
# Camera
camera = scene.addCamera()
camera.transform = ti.ui.ComposeTransform(...) # Many ways to set transform (e.g. translation / rotation / scale)
camera.fov = math.rad(70) # The FOV of camera
camera.near = ...
camera.far = ... # These should have a reasonable default value
# Primitives
trimesh0 = scene.addTriangleMesh(vertices, numVertices, firstVerex=0) # vertices should (ideally) be a 1D ti.dense
trimesh0.setColor() # This can be a color value, or another ti.dense for per-vertex color
trimesh0.setTexture("textureFile", uv) # UV should be a ti.dense for per-vertex UV. We should also consider supporting using ti.dense images as texture.
trimesh0.transform = ti.ui.ComposeTransform(...) # mesh is a scene node
pointcloud0 = scene.addPointCloud()
trimesh1 = trimesh0.addTriangleMesh(...) # Trimesh 0 contains a child mesh, creating a hierarchical transform
# Lights
pointlight0 = scene.addPointLight()
distantLight0 = scene.addDistantLight()
domeLight0 = scene.addDomeLight() # This can be uniform color or a skybox
# Rendering
# Renders the scene from camera to a target (e.g. window)
renderer.render(target, scene, camera) # target can be the window, another texture, or potentially ti.dense

This is extendable, and should be able to run efficiently on GPU. The potential is also there to tightly integrate real-time rendering into Taichi applications with dense field interactions. The UI backend should talk with the taichi backend to transfer the images / fields implicitly. In the beginning iterations we can force the user use the same renderer / taichi runtime or a few supported combinations. (e.g. gl-gl, vulkan-vulkan, cuda-gl, cpu-vulkan)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be ideal to create a seperate module just to manage API crossing. It is always possible to have shared host-device memory (cpu & gpu visible or even coherent). It is also possible to do supported interops such as Vulkan / GL image sharing or GL / cuda

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a scene graph would be really nice. However, I think this somewhat defeats the goal of having a simple and easy-to-use API.

It would be ideal to create a seperate module just to manage API crossing.

Yes. This will be necessary if GGUI is to replace the legacy ti.GUI completely. Currently we are aiming to support Vulkan(GUI) <-> cuda(Taichi) only. (@k-ye When/if should I worry about other backends?)

### Creating a scene
```python
scene = ti.ui.Scene()
```
### Configuring camera
```python
camera = ti.ui.make_camera()
camera.lookat(pos)
camera.up(dir)
camera.center(pos)
camera.projection_mode(mode)
scene.set_camera(camera)
```
where `mode` is either `ti.ui.Scene.PROJECTION_PERSPECTIVE` or `ti.ui.Scene.PROJECTION_ORTHOGONAL`


### Configuring light sources
#### adding a light source
```python
scene.point_light(pos,color)
```


### 3d Geometries
```python
scene.mesh(vertices,indices,color)
scene.particles(positions,radius,color)
```


### Rendering the scene
a scene is rendered by first rendering it on a canvas.
```python
canvas.render(scene)
```

## GUI components

The support for GUI components will closely follow Dear IMGUI (and will likely be implemented using it..).

```python
window.GUI.begin(name,x,y,width,height)
window.GUI.text(...)
window.GUI.button(...)
window.GUI.end()
```


## Clearing and showing a window
```python
...
window.show()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe have windows constantly show? And make show / hide toggle the window visibility state instead of having to call "show" every frame.

```


## Events Processing
To obtain the events that have occurred since the previous poll:

```python
events = window.get_events()
```

Each `event` in `events` is an instance of `ti.ui.Event`. It has the following properties:
* `event.action`, which could be `ti.ui.ACTION_PRESSED`, `ti.ui.ACTION_RELEASED`, ...
* `event.key`, which is `ti.ui.KEY_XXXX`

To obtain mouse position:
* `window.get_mouse_position()`


## Example Application

```python
import taichi as ti

window = ti.ui.Window("Amazing Window",res)
canvas = window.get_canvas()
scene = ti.ui.Scene()


while window.running:
events = window.get_event()
if ev.action == ti.ui.ACTION_PRESSED and ev.key == ti.ui.KEY_SHIFT:
...

canvas.clear(...)
canvas.triangles(...)

scene.clear()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constantly rebuilding the scene will be costly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily. The implementation is free to do all kinds of caching.

We wish to keep function calls such as canvas.triangles(a,b,c) inside the render loop in order to make it obvious to the users that the newest values of the taichi fields a, b, and c, which might have just been modified this frame, will be used for rendering. This keeps the semantics of the API simple.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the clear() function here means the clearing of the color & depth buffer of the canvas or does it mean clearing of the primitives defined from last frame (essentially a NewFrame() call)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

canvas.clear(): clearing the color & depth buffer of the canvas.
scene.clear(): clearing the geometries in the scene.

camera = ti.ui.make_camera()
camera.lookat(pos)
camera.up(dir)
camera.center(pos)
camera.projection_mode(mode)
scene.set_camera(camera)
scene.point_light(pos,color)
scene.mesh(...)
canvas.render(scene)

window.GUI.begin(name,x,y,width,height)
window.GUI.text(...)
window.GUI.button(...)
window.GUI.end()

window.show()


```