proposal: golang.org/x/exp/shiny: an experimental GUI library.

[nigeltao]

Purpose.

First and foremost, this is an experiment. This is not 'the official Go GUI (Graphical User Interface) library' or the only blessed way to do things. Unlike a well-understood problem like decoding a JPEG image into an in-memory buffer, there are many possible approaches to GUIs, including existing code both under golang.org/x/mobile and elsewhere. This is simply an exploration. I've reached the point where I have an interesting if minimal demo, and I'd like to share and discuss the code, design and ideas.

Name.

There's nothing particularly deep about the "shiny" name. I just like shiny things.

The code would live at golang.org/x/exp/shiny. To emphasize what I've said above, this is an experiment, and this is a GUI library, not the GUI library, so it's not e.g. golang.org/x/gui.

It is under golang.org/x because I want to use the same code review process as everything else under golang.org/x, a process I use every working day.

Overview.

There are two layers: a lower-level window layer, and a higher-level widget layer.

For the lower layer, the primary concept is a Window. On the input side, a shiny.Window looks similar to what the golang.org/x/mobile/app package currently calls an App: you get a channel of events, such as key and mouse events.

The output side is where things diverge. x/mobile/app assumes that you paint pixels with OpenGL, either directly via the x/mobile/gl package, or indirectly via e.g. x/mobile/exp/sprite uses x/mobile/exp/gl/glutil uses x/mobile/gl. Instead, shiny has twin concepts of Buffers and Textures which map naturally to an OpenGL implementation, but also allow other implementations such as one that speaks a pure X11 protocol, without the need for cgo and OpenGL shared libraries. I assume that you could do somewhat similar things on Windows and Mac.

A Buffer is an in-memory, in-process bucket of bytes. A Texture is an opaque handle to something 'out-of-process'. A Buffer's contents can be Uploaded to a Texture, and Textures can be Drawn to the Window. As an optimization, it may be possible to Upload a Buffer directly to a Window.

For OpenGL, you can think of a Buffer as the result of a simple make([]byte, etc) call, and a Texture as an OpenGL texture. For X11, you can think of a Buffer as MIT-SHM shared memory (if applicable) and a Texture as a (server-side) Pixmap.

Uploads are 1:1 between source (Buffer) and destination (Texture) pixels. Draws can be subject to an arbitrary affine transformation (including simpler copies and scales) between source (Texture) and destination (Window) pixels. For OpenGL, affine transforms are simple shader programs. For X11, they are provided by the XRender extension.

For the higher layer, you construct a tree of widgets like buttons, text areas and containers. This shouldn't be overly surprising, except for the fact that the Go language doesn't have traditional object-oriented inheritance. The details of how that works is best explained with actual code, so I'll save it for a CL, if this proposal is approved.

The widget set is pure Go, like Swing instead of AWT or SWT. Sweating the details of a production-quality widget set is an enormous amount of work, and hitting that level of quality is not a short term goal. The primary goal is to explore what's possible in the GUI space with Go, and not necessarily to crank out a production-ready classic Win32 or Cocoa app any time soon.

A secondary goal is to drive development of the Freetype Go port - it has some basic functionality but also some long-standing known deficiencies. Writing a quality GUI text editor is again a substantial amount of work, and I don't intend to build one, but I intend that working on shiny will push the Freetype Go library to be good enough to let someone or some team do exactly that, if they had the time and inclination.

Relationship to golang.org/x/mobile.

There is a lot of overlap between shiny and x/mobile, and obviously I have contributed to both. The key difference, I think, is that the x/mobile code is rightly focused on mobile apps, where the dominant model is each app has only one (full-screen) window, whereas shiny starts more from a desktop perspective. To mix some metaphors, triangulating from different starting points (mobile vs desktop GUIs) can help avoid overfitting to local optima. In the medium term, the two should end up sharing a lot of code, but it's not obvious to me right now whether x/mobile should depend on shiny or vice versa, and it would be premature to pick a winner now. As I keep saying, there are many possible approaches, and I think it's valid to explore more than one of them concurrently.

There is also the operational concern that I would like the freedom for shiny to depend on third party libraries, such as github.com/golang/freetype and github.com/BurntSushi/xgb, but I'd also like to keep the golang.org/x/mobile tree free of such dependencies, at least for now.

Relationship to github.com/google/gxui.

I'm obviously aware of their work, but the two projects aren't strongly linked. As I've said, there are many possible approaches that are worth trying here. Longer term, we might discuss what we can take from each other's codebase, but it's premature to do that now.

Closing Remarks.

I'll repeat that this is experimental and exploratory. I just have a demo and I'd like to share it. I know that there could be a lot of excitement in the community for a 'Go GUI library', but please don't set your expectations too high at this stage, and the existance of this project should not invalidate any other Go GUI projects.

[robpike]

SGTM

[taruti]

What is the license policy of the x/ tree? Probably the FTL (or GPL) requirement is fine for most uses, but to avoid clients accidentally breaking the attribution requirement it should be clearly documented if there is a freetype dependency.

[ajstarks]

Interesting proposal. Can you sketch what a simple client program would look like?

[eaburns]

Looks very interesting. I also prototyped a UI library in Go. It shares many of the ideas that you propose.

It has two halves: the ui package (described below) and the gui package (not written) for simple widgets.

The ui package deals with Windows and Events. It's currently implemented on top of SDL2 using cgo. With ui, one can draw Buffers to Textures and Textures to Windows. However, I call Buffers image.Images and drawing them to Textures happens via draw.Draw. It also has an interesting (I think) solution to the annoying OpenGL thread context issue. In short, one calls Window.Draw with a drawing function that accepts a ui.Canvas. The Canvas has methods for actually drawing to the Window. Window.Draw ships the draw function and Canvas to the thread with the OpenGL context in order to perform all the drawing. This obviates the need for channel operations and context switching for each call to a draw primitive.

Perhaps you'd like to take a peek: http://godoc.org/github.com/eaburns/T/ui.

Also, I'd be interested in ditching my package in order to help out with Shiny. Particularly because the motivation for my ui package is to write a text editor, and you explicitly mention as a target use for Shiny.

[andrewchambers]

My vote for first demo app is an acme clone :)

[crawshaw]

SGTM

Note that x/mobile/app's OpenGL dependency will eventually be generalized to ES2, ES3, Metal, and Vulkan. But all of these share a similar GL style and shiny will need different machinery.

I would like to see the code in x/mobile/app converge in the medium term. There's a lot of subtly in cross-platform window and event management I would like to combine in one place.

[mattn]

Sounds good to me. Are you talking about mobile limited?

[aclements]

In the medium term, the two should end up sharing a lot of code, but it's not obvious to me right now whether x/mobile should depend on shiny or vice versa, and it would be premature to pick a winner now.

Rather than either depending on the other, would it make sense for them to share a common third dependency? I don't have a strong sense of exactly what would be shared between them, but, as an example, the OpenGL bindings seem like a common dependency that is neither mobile- nor GUI-specific, but is required by both.

Put another way, x/mobile is focused on mobile; it needs things like windows and OpenGL, but doesn't (itself) need a GUI toolkit. shiny is focused on GUIs; it needs things like windows and OpenGL, but isn't specific to mobile.

[owenthereal]

👍 Is there any more info on what it is like?

[theGeekPirate]

I've reached the point where I have an interesting if minimal demo, and I'd like to share and discuss the code, design and ideas.

Mind sharing?

[crawshaw]

@aclements I believe that is what Nigel and I are both trying to say, there are just enough moving parts that's it is difficult to describe. Another attempt:

There should ultimately be one shared piece of code for creating windows and delivering events. It may have two separate interfaces, one for mobile and one for desktop, or maybe not. To some extent, it will have to know about the underlying drawing technology the user wishes to use: GL (probably all varieties can be collapsed at this level), Metal, Vulkan, or a *image.RGBA that is fed into the underlying system some other way (the X11 protocol). This is the work mostly being done in the x/mobile/app package right now, and needs to be generalized a little bit and live somewhere other than x/mobile. Both mobile and shiny will use it. In some ways this window+event package is like glfw with two differences: firstly that it's written in Go, and secondly that it doesn't necessarily require GL.

Then there are the packages for accessing the graphics system. Right now we have x/mobile/gl for GL ES 2. It has no mobile dependencies, and should live somewhere else, and can be optionally used by shiny. There are other GL packages out there that expose more features (and a fork of x/mobile/gl that works on GopherJS.) The x/mobile/app and its future generalizations will depend, probably in a "pluggable" way, on these graphics system packages.

[skelterjohn]

Consider treating the input/output (keyboard and mouse/pixels on the screen) as a completely separate issue from GUI widgets.

As an example, I created github.com/skelterjohn/go.wde to do the input/output, and github.com/skelterjohn/go.uik to do GUI widgets using go.wde.

[phonkee]

Sounds interesting, is there anything we can help with?

[andlabs]

Oh boy, this is quite a long thread with a lot to talk about. here goes!

---

@nigeltao original post

Your design is in good company: this is pretty much how GTK+ works; the name of its lower layer is GDK.

What I learned from my own UI package attempt is that the way operating systems give you events and the way Go cahnnels work don't line up fully. With a Go channel, a transaction is considered complete once the message is received, and the code that produces the message can continue operating regardless of what the receiver does. But with a traditional GUI system, the system expeccts that you handle a message fully before asking for another one, and will generate input, painting, and timer events according to that design. In practice, things may seem to work at first, but unless you have an "okay, done with this one" channel for everything, you're eventually going to fall into the trap I fell in with the first version of my package ui: needing to introduce such a channel for painting events and everything deadlocking as a result. I also don't know how x/mobile handles it.

Of course, I could be wrong and this could be a non-issue if done right; only experimentation will tell. (Until then, I've decided to separate my own package ui from Go as much as possible, with the intention of avoiding conflicts between the Go runtime and what the C world expects.) There is definitely room for experimentation here (experimentation I am very interested in).

Next, here is a very very important question. Or rather, two questions whose importance is mutual. Using "shiny.low" to refer to the lower layer and "shiny.high" to refer to the higher layer:

1. Will a shiny.low.Window map solely to a top-level window (such as the one your web browser runs in) or to an arbitrary region of the screen that has its own independent event handling?
2. Will each shiny.high.Widget have its own shiny.low.Window?

If you answer "the latter" to 1 and "yes" to 2, the GTK+ project has done this in the past and found that it doesn't really work well on X11. The link is to a video that shows a window being resized in GTK+ 2 and the massive flicker that resulted as the widgets were redrawn. (Windows and OS X don't have this problem as much because of heavy optimization and API design tricks in the former's case and a similar design to what I'm about to say in the latter.) You may find it more performant to chop up a shiny.low.Window into each shiny.high.Widget's parts and mux out events that way. GTK+ 3 has switched to this model.

Another thing is that for the widget library, you would need more than just a text rendering library; you would also need a full vector graphics library. I know the old code.google.com freetype package had one built in, but I'm not sure how comprehensive it is for the purposes of a GUI package. That also leaves open the question of theming.

Threading is another issue. Windows and OS X (I don't know about the other Unix display servers) assume that all your UI operations exist on the same thread; OS X goes even further by saying that it must be the very first thread the OS creates (though there is an undocumented function to change it; I have yet to list the full side effects). I'm aware of runtime.LockOSThread() and that might be the solution you'll want to go for. (The future of my package ui will lie in having the GUI thread be a C thread that Go knows nothing about, and that uses other methods of communicating with other threads. In my early experiments this seems to work fine as well.)

That being said, I would be happy to contribute to this project if it did exist; I could even possiby (no promises, just a thought) prototype the low level part this weekend using code that I already have in my ui and libui projects. skelterjohn has also tried something like this with his go.wde (the low layer) and go.uik (the high layer) projects; possibly using some of that code might also help.

One big question that looms in my mind is how cgo, moving garbage collection, and arrays and slices will work, since it isn't very well defined (as far as I'm aware, anyway). At some point we're going to have to feed Go data blobs to C. I should have stated this in that one issue; I'm really not against moving garbage collectors so long as the ability to safely feed large amounts of data to the C world in a well-defined manner is provided (or the rules for doing so defined).

---

@crawshaw

See what I said above about how to map a low-level window to a high-level widget. I'm not sure how this will conflict with golang.org/x/mobile's needs... And in fact, wouldn't having a single end-all-be-all OpenGL package be the logical conclusion of this?