yampa-gui: Nested Dynamic Windowing System Library

[Cajunvoodoo]

Inspired by windowing systems like Xmonad and the GUIs of the Yampa arcade, a robust, extensible, and dynamic windowing system would be a great addition to Yampa's capabilities.

I came up with a rough plan of capabilities a window must have along with a number of hurdles.
Capabilities:

1. A window responds to inputs and passes that to its children
2. A window can be focused or unfocused to allow for routing input correctly
3. Windows may be arbitrarily nested and have arbitrarily many children (that are also windows)
4. A window may output events to the parent (like destruction and creation of self and siblings respectively) or for more complex behavior
5. A window may have zero children (e.g. a button or a text box) and can hold data relevant to its function
6. A window must have a Picture element (yampa-gloss) that describes its appearance

When I was thinking it over, windows reminded me of dpSwitch. The hurdle I came across was routing and the nested nature of windows; how do we nest dpSwitches (child windows) inside another dpSwitch (the parent window)?

Another conceptual challenge i came across was data relevant to specific children. Here's a contrived but salient example:

┌──────────────────────────────────────┐
│  ┌────────────────────────────────┐  │
│  │          Count: n              │  │
│  ├────────────────┬───────────────┤  │
│  │                │               │  │
│  │   I'm a blank  │               │  │
│  │     window!    │               │  │
│  ├────────────────┤               │  │
│  │                │               │  │
│  │ ┌────────────┐ │  ┌──────────┐ │  │
│  │ │┼──────────┼│ │  │          │ │  │
│  │ ││          ││ │  │ Bigger   │ │  │
│  │ ││Big Button││ │  │ Button   │ │  │
│  │ ││ Click to ││ │  │ Click to │ │  │
│  │ ││ Decrement││ │  │ Increment│ │  │
│  │ ├┴──────────┴┤ │  │          │ │  │
│  │ │            │ │  │          │ │  │
│  │ │ Also Empty │ │  └──────────┘ │  │
│  │ ├────────────┤ │               │  │
│  │ │  Empty! :3 │ │               │  │
│  │ └────────────┘ │               │  │
│  │                │               │  │
│  └────────────────┴───────────────┘  │
└──────────────────────────────────────┘

This window can be described by the following tree:

               parentWindow
                   |
                 window
               /   |   \
              /    |    \
       win_l  winCount   win_r
      /    |     |          |-> incrementButton -> onClick increment
     /     |   count
    |    blank
   botLeftTriWindow
  /     |          \
"also" "empty :3"   \
                     \
                    winButtonHolder
                       |
                      decrementButton -> onClick decrement

In this example, the count is shared by all of the children of window (and all of its children too). When clicking the buttons, the count would be incremented or decremented, despite being nested so deeply in a child window. But parentWindow may not care about the state window holds. And what if winCount counted some other value alongside count that neither win_l nor win_r needed to manage? What type would these windows have? How should the events of the children be routed in a clean and comprehensible (not to mention extensible and modifiable) manner?

Once these issues related to the structure of the core systems are ironed out, a minimal working implementation could be written and features could be added from there (which all sounds like a lot of fun!).

I think Yampa has the capability to create an excellent interactive GUI system. yampa-gloss provides a great base to build this library off of.

[ivanperez-keera]

It would be really cool to make this happen. Now, implementing a full GUI library is a big undertaking! There are some prior work that I think would be important to take into account. Some of it may offer insights of where pitfalls and opportunities may lie. Other may present tradeoffs:

• Antony Courtney developed Fruit: https://www.antonycourtney.com/pubs/ac-thesis.pdf, https://gist.github.com/antonycourtney/4d10dfd2bdc953d7726c93f25542a527.
• I have long believed that Fudgets could be implemented completely on top of dunai (a Fudget is just an MSF with a side effect in the GUI): https://www.altocumulus.org/Fudgets/dist.html
• There's also the related Gadgets by Noble: https://pure.york.ac.uk/portal/en/publications/gadgets-lazy-functional-components-for-graphical-user-interfaces-2
• Finally, there are traditional GUI libraries. I wonder if it might make sense, and be possible, to design this new yampa-gui such that it could have backends that leverage existing UI libraries like GTK+, Qt, wx, and other native UI APIs.

[ivanperez-keera]

I'm turning this into a discussion for now. I think it's a better place to explore and propose ideas. Once we have a more concrete plan, we can create the issue or issues that will address the proposed change.

[bradrn]

Some ideas around structuring GUIs with arrowised FRP (rewritten from an email sent to @ivanperez-keera):

• GUI widgets intrinsically couple IO operations (namely drawing on the screen/getting user input) with logic. It therefore seems to me that the appropriate abstraction would be monadic stream functions, as provided by Dunai/Bearriver, rather than the pure SFs of Yampa.

• Thinking about a single widget in isolation, it seems reasonable to represent it as a single MSF IO a b which on initialisation sets up the widget to process events, taking input a and producing output b. (This is basically the same approach taken by Reflex and threepenny-gui.) Thus we might have label :: MSF IO Text (), button :: Text -> MSF IO () (Event ()), or textarea :: MSF IO (Event Text) Text. The whole GUI could thus be created by combining widgets together to give an MSF IO () () at the top level.

• It might even be possible to separate out layouting from widget creation, by making widgets additionally produce some kind of View value. This would create considerable flexibility by allowing for layout combinators to be represented as MSFs, along the lines of atop :: MSF m [View] View. The whole GUI would then become a MSF IO () View. The View itself could be some kind of virtual DOM à la gi-gtk-declarative, or perhaps even just a pointer to a C++ class.

• Making the above actually work, however, requires a couple of different components. The first is a way to create MSFs which run a monadic operation at startup only. For this, Ivan kindly provided me with the following code, which looks like it should work:

  onceS_ :: Monad m => m b -> MSF m a b
  onceS_ action = feedback Nothing $ arrM update
    where
      update (_, Nothing) = action >>= \x -> return (x, Just x)
      update (_, Just x)  = return (x, Just x)

  Though he also notes this can also be done with the functions from Control.Monad.Trans.MSF.Except, such as performOnFirstSample :: Monad m => m (MSF m a b) -> MSF m a b.

• The second thing is a way to trigger events from within an MSF itself: i.e. a function along the lines of trigger :: Monad m => m (a -> m (), MSF () (Event a)), where executing the returned a -> m () triggers a corresponding occurrence of the Event a. And this is problematic: that outer monadic context would mess up the niceness of making everything arrowised. (In fact, I wonder if that’s why Reflex and similar libraries use monadic FRP.) Besides, I can’t

• Unfortunately, I don’t see any way to get around that second requirement: the alternative is passing in the Event a from the outside, and that gets horrible quickly if you have lots of widgets which can all emit events. With the existing examples this hasn’t been too problematic, since something like SF (Event Input) Picture suffices if you’re using SDL or Gloss and making all the widgets yourself, but if you want to use a third-party GUI library, it becomes more like SF AllEventsFromAllWidgetsInTheProgram View, which is just nasty to deal with.

• Except, I just got another idea: what if we used performOnFirstSample, or a similar function? That could allow writing instead:

  performOnFirstSample' :: Monad m => m (c, MSF m a b) -> MSF m a (c, b)
  performOnFirstSample' m = safely $ do
      (c, msf) <- once_ m
      safe $ (c,) <$> msf
      
  
  trigger' :: Monad m => MSF m () (a -> m (), Event a)
  trigger' = performOnFirstSample' trigger  -- as earlier, trigger :: Monad m => m (a -> m (), MSF () (Event a))

  Which would be much easier to integrate with the rest of the program. Hmm, maybe I’ll try this and see how it goes… (Though I thought of this idea literally just a few minutes ago, so I imagine there’s all kinds of horrible problems I haven’t thought of yet!)

[bradrn]

Just realised a problem with this idea: it’s easy to implement trigger :: Monad m => m (a -> m (), MSF () (Event a)) so that it’s blocking, but I’m not sure if it’s possible to implement it in a non-blocking way. This means that if you have, say, a GUI with two buttons, each of which do something when pressed, the MSF for the whole GUI will instead block until both buttons are pressed, which is not the desired behaviour. And if non-blocking trigger is impossible to implement, that means this idea

I’m now wondering if it’s possible to pass all the event connection functions (connect button #clicked :: IO () -> IO SignalHandlerId and so on) out through the MSF output, then connect them outside the reactimation such that events are routed to the right place in the signal graph. This is in essence just that SF AllEventsFromAllWidgetsInTheProgram View approach all over again, but it might be possible to do it in a nice way by choosing an appropriate monad to store the events — SF (ReaderT CurrentEvent IO), perhaps?

[bradrn]

OK, this seems to work! The code’s a tad too long for a comment, so I put it in a gist. But here’s the end result:

main :: IO ()
main = runGui $ proc () -> do
    (e1, btn1) <- button "Click me!" -< ()
    (e2, btn2) <- button "Click me too!" -< ()
    btns <- both -< (btn1, btn2)
    swappedBtns <- accumBy1 swap -< (btns, lMerge e1 e2)
    hpaned -< swappedBtns

Basically, this works by maintaining as state a set of events to be handled and events to be registered. Functions to register event handlers are generated alongside the widgets, and passed ‘up’ to the main loop; that then registers handlers which, on each event, run the MSF, passing the identifier of that event back ‘down’ through the MSF until it gets converted into an occurrence of an Event (). This allows e.g. a trigger function to be written which appears non-blocking, with all the advantages that entails.

One significant problem with this approach so far: it can’t handle events with arguments (e.g. keyDown, which would ideally be an Event KeyPress). But I expect that problem can be solved fairly easily using existentials or DMap or some similar technique. I’m also not quite sure how to read or write widget properties — when I try to write stuff my program goes into an infinite loop, though I’m pretty sure that’s just GTK being stupid rather than any fundamental flaw with my approach. But overall, it looks like this may well have real potential!

That being said… looking at @Cajunvoodoo’s original post, it appears I’ve drifted off-topic somewhat, in that most of their points are handled by the GUI library rather than by Yampa (or in this case Bearriver). It does, however, show a working technique to route incoming events down the signal graph until they encounter the right widget.

[bradrn]

Minor update: got reading/writing widget properties to work. The problem was indeed Gtk being stupid — in the case I tried (reading/writing a text entry), each write emitted two events, which then triggered two more write operations in the program, which then emitted yet more events, and so on ad infinitum until the poor program segfaulted. The fix involved keeping track more carefully of exactly when values changed and an update is necessary, so basically returning an Event Text instead of a Text. I’m beginning to see why Reflex uses Dynamics everywhere: they solve exactly this problem while keeping the current value easily accessible at all times. Fortunately, it’s trivial to implement those in Yampa as type Dynamic a = (a, Event a) (well, you’d want to use a record, but same idea).

[bradrn]

Improved the sample code given above: now it’s at https://github.com/bradrn/dunai-gi-gtk. Feel free to play around with it and suggest ideas or improvements!