toolkit-agnostic Core->GUI Communication

[umlaeute]

This is research/brainstorming for refactoring the communication between the Pd core and the GUI to remove Tcl/Tk code from the core. see #1693

Note: This is not a full, open rewrite away from Tcl/Tk but a refactor and abstraction to allow for implementing custom GUIs without worrying about Tcl-specifics. Also, this current work maintains the editing logic within the core.

Please keep the discussion focused on technical considerations. This is not meant to be a GUI feature wish list!

---

currently, the Pd-core constructs Tcl/Tk drawing commands like this:

/* draw a float/symbol gatom */
sys_vgui(".x%lx.c create line %d %d %d %d %d %d %d %d %d %d %d %d "
        "-width %d -capstyle projecting -tags [list %sR atom]\n",
        glist_getcanvas(glist),
        x1p, y1p,  x2-corner, y1p,  x2, y1p+corner, x2, y2,
        x1p, y2,  x1p, y1p, glist->gl_zoom+grabbed, tag);

instead it should issue a command that is oblivious of the actual drawing:

sys_vgui("::pd_canvas::create %x %x floatatom %d %d -width %d",
        glist,
        x,
        x->a_text.te_xpix,
        x->a_text.te_ypix
        x->a_text.te_width);

this is still a valid Tcl/Tk command (even using typical Tcl/Tk constructs like dash-arguments (-width)), but should be easy to parse to be used by any GUI backend)

[umlaeute]

i think, ideally the syntax would be closely modelled after how objects are stored on the disk.

e.g.

pd	gui	description
#X	::pd_canvas::create <glist>	create a new object on the current (#X) canvas (glist)
	<x>	name the new object
floatatom	floatatom	type of the object
137 110	137 110	position
5	-width 5	(named) property

[umlaeute]

Does this involve creating a TCL procedure on the frontend that does the conversion

yes exactly.

How would a message formatted with "::pd_canvas::create %x %x floatatom %d %d -width %d" further evolve in this scenario?

i would create some registration on the GUI side, where an "external" could register a new GUI-object, e.g:

proc ::gui_example::make_nobby {name canvas x y args} {
   # ...
}
::pd_canvas::register "knob" ::gui_example::make_nobby

#...
::pd_canvas::create %x %x knob %d %d

[umlaeute]

the idea is mainly to allow to use the same call on the core for common tasks (e.g. move stuff around), so the actual calls to pdgui_vmess (or sys_vgui) can be removed from the C-implementation.
that probably won't be 100% possible (and the constructor is probably one of those functions that won't go away).

otoh, there's no reason why we couldn't use the on-disk representation (as returned by the object's savefn) for the ctor.

[giuliomoro]

i think, ideally the syntax would be closely modelled after how objects are stored on the disk.

It occurred to me that one thing that's missing here is the number and type of inlets/outlets, as they are not stored on disk. Would these be passed in a separate command?

A wider question I have is: should the GUI know how to draw (1) a rectangle with a fill (and then we call it over and over for each inlet/outlet), or (2) should it be told at once to draw an object with 2 signal inlets, 1 signal outlet and 2 message outlets ?

According to @danomatika 's original post

GUI API drawing specifics out of the core into the GUI (ex: "draw a bng" as opposed to "draw a box, draw a circle, draw an inlet, draw an outlet, ...")

it seems like we are leaning towards (2), in which case we do need to pass significantly more detail to the GUI than what is saved on disk.

[Spacechild1]

it seems like we are leaning towards (2), in which case we do need to pass significantly more detail to the GUI than what is saved on disk.

I think for generic text objects we would definitely need to include information about inlets/outlets. For specific GUI objects, such as bang or number box, this might not be necessary (after all, the GUI is supposed to know what a "bang" or "number box" should look like), but I'm not sure...

[giuliomoro]

[following up in-thread]

As Dan reported below:

Many GUI objects can hide their I/O so passing that may still be needed for them.

So it seems that for all objects we need to send details on their inlets/outlets

[giuliomoro]

As a note, in order to take further advantage of the testing I tested here, it would be extremely helpful to have a place where all the final (after conversion)Tcl/Tk messages are logged, so that we can automate the process of verifying we didn't break anything.

[sebshader]

@giuliomoro I think it should be more like 1. (e.g. draw bang here, draw text object w/ inlets: {signal, message, message, signal} and outlets: {message, signal, signal} etc.)
do you know the 'trace' tcl command? I think you could use that to snoop on messages.

[danomatika]

Many GUI objects can hide their I/O so passing that may still be needed for them.

[danomatika]

This falls into the question of where certain GUI state aspects should sit, ie. is the logic for deciding I/O for IEM GUIs better handled on the GUI side or in the core. I lean toward the core and keeping state in one place and standardizing I/O as part of a message makes sense here. In cases where it’s ignored, passing 0 would be fine.

[giuliomoro]

The overall spirit of this refactoring effort seems to suggest that the state should remain on the server.

[sebshader]

I think it makes sense to keep as much duplicate information as necessary to draw the object on the gui side. (which means: how many inlets/outlets and what kinds for regular objects, what the connections are, and maybe for other objects as well just to make it more consistent in the code/handling).
Hypothetically that means that the only communication that would need to happen in the future is just information about creation, deletion, and connections.

[giuliomoro]

currently, the Pd-core constructs Tcl/Tk drawing commands like this:

/* draw a float/symbol gatom */
sys_vgui(".x%lx.c create line %d %d %d %d %d %d %d %d %d %d %d %d "
        "-width %d -capstyle projecting -tags [list %sR atom]\n",
        glist_getcanvas(glist),
        x1p, y1p,  x2-corner, y1p,  x2, y1p+corner, x2, y2,
        x1p, y2,  x1p, y1p, glist->gl_zoom+grabbed, tag);

instead it should issue a command that is oblivious of the actual drawing:

sys_vgui("::pd_canvas::create %x %x floatatom %d %d -width %d",
        glist,
        x,
        x->a_text.te_xpix,
        x->a_text.te_ypix
        x->a_text.te_width);

One issue specifically with floatatom when the hit detection is kept in the core is that there will be need for a shared knowledge between the core and GUI of the shape of the object: if you place the mouse cursor in the top-right "missing" corner of the object, the cursor ill keep the "regular" shape and only when you enter the actual trapezoid's surface will it turn to the "active" shape. Where is the information about the shape of the box stored and if it's duplicated between the GUI (for drawing) and the core (for hit detection), then how do we ensure they are kept in sync for the rest of times?

I assume there will be similar issues for other objects. For instance, [hsl] and [vsl] need to know exactly where the mouse cursor is with respect to the 'knob' in order to properly handle "jump on click" or "steady on click". If the GUI tries to customise the look in such a way that e.g.: a few "padding" pixels are added inside the rectangle that was prescribed by the core, the functionality's behaviour may be affected. Last, other "subshapes" such as outlets and inlets must be located at a very specific place (with no flexibility left to the GUI for deciding where to draw them) in order for both cursor changes on hover to happen appropriately and for patching to work at all.

So, how is consistency of shapes / sub-shapes between core and GUI ensured when the details of the shape itself are offloaded to the GUI? Should the GUI send back to the core the details of the shape / subshapes it drew so that the core is aware of them?
[Unless all these functionalities are offloaded to the GUI, of course, but that's what this work is specifically not about.]

[umlaeute]

if you place the mouse cursor in the top-right "missing" corner of the object, the cursor will keep the "regular" shape and only when you enter the actual trapezoid's surface will it turn to the "active" shape.

is that an observation or a wish/fear?
because here (on Linux), the cursor doesn't change at all when hovering over the floatatom (neither over the cut-out, nor over the rest of the object).

also, i can drag the numbers just fine when the mouse starts on the cut-out (so it seems, that for all the hit-box-detection, the object is an ordinary rectangle).

EDIT: conversely, a messagebox will not consider the horns to be part of the active region.

So, how is consistency of shapes / sub-shapes between core and GUI ensured when the details of the shape itself are offloaded to the GUI? Should the GUI send back to the core the details of the shape / subshapes it drew so that the core is aware of them?

but yes, somewhat like this.
i think in the general case, Pd-core cannot assume that it will know exactly which size an object will have (fonts for one thing; but consider dan's ncurses based interface, where you simply do not have "pixel resolution".

so far i've only ever thought about rectangle objects, (where Pd requests an object at a given position, and the GUI needs to send back how big the object it created is; it could also send back where it actually created the object).
it should be possible to extend this to any (polygon) shape, but OTOH i'm not sure it's actually worth it (see my observation above, that any cut-outs are purely cosmetic).
hit-box detection in an arbitrarily shaped polygon is much more complicated than within a rectangle...

if a widget adds decoration (e.g. a nice frame around the slider), i would expect this decoration to be "dead" (from Pd's POV) and the actual "active" area is what Pd thinks it is.

[danomatika]

I agree. Anything beyond a hit-box is probably beyond what Pd needs. In interaction-based works, I generally only use distance circle or hit box checking because they are fast and simple to write.

…

On Sep 2, 2022, at 1:05 PM, umläute ***@***.***> wrote: it should be possible to extend this to any (polygon) shape, but OTOH i'm not sure it's actually worth it (see my observation above, that any cut-outs are purely cosmetic). hit-box detection in an arbitrarily shaped polygon is much more complicated than within a rectangle... if a widget adds decoration (e.g. a nice frame around the slider), i would expect this decoration to be "dead" (from Pd's POV) and the actual "active" area is what Pd thinks it is.

-------- Dan Wilcox @danomatika <http://twitter.com/danomatika> danomatika.com <http://danomatika.com/> robotcowboy.com <http://robotcowboy.com/>

[Spacechild1]

[Unless all these functionalities are offloaded to the GUI, of course, but that's what this work is specifically not about.]

Personally, I would really like to have the possibility for third-party GUIs to perform their own collision detection, therefore I think the actual GUI message refactoring should be designed in such a way that Pd's collision detection system can be circumvented. We shouldn't leave this big chance on the table!

Generally, I agree with others that the core should not need to know the exact shape of the object.

However, @giuliomoro brought up a very important point regarding hslider/vslider (and related widgets). Here the object hitbox does not offer enough information, e.g. because of padding inside the object. That's really something to think about!

[giuliomoro]

is that an observation or a wish/fear?
because here (on Linux), the cursor doesn't change at all when hovering over the floatatom (neither over the cut-out, nor over the rest of the object).

macos here, the cursor changes when you hovering over the floatatom (in non-edit mode):

I had done a quick test before posting and it seemed that the cutout was excluded from hit detection, but upon second inspection it actually is included, sorry for the false alarm. And I can confirm that msg boxes will not consider the horns to be part of the active region.

hit-box detection in an arbitrarily shaped polygon is much more complicated than within a rectangle...
if a widget adds decoration (e.g. a nice frame around the slider), i would expect this decoration to be "dead" (from Pd's POV) and the actual "active" area is what Pd thinks it is.

This makes a lot of sense as that seems to be exactly what Pd is doing right now, no reason to make things more complicated. Rectangles only.

so far i've only ever thought about rectangle objects, (where Pd requests an object at a given position, and the GUI needs to send back how big the object it created is; it could also send back where it actually created the object).

It probably needs to send back information about individual inlets/outlets as well, right?

Personally, I would really like to have the possibility for third-party GUIs to perform their own collision detection, therefore I think the actual GUI message refactoring should be designed in such a way that Pd's collision detection system can be circumvented. We shouldn't leave this big chance on the table!

It seems that if the GUI sends back to the core the shape of the created shape and sub-shapes, then we would only need to add a "hit_triggered" message back to the core. If a GUI wants to handle all (or only some!) of the hit detection, it could simply send back empty shapes to the core and instead send hit_triggered when its own hit detection detects a hit. I think.

[sebshader]

It would be really nice to have all of the hit detection/ui geometry/metrics offloaded from the audio thread for performance reasons too..
It would also make hit detection for things like bezier cables (that some people prefer) and segmented cables more viable too I think..

maybe in the distant future some non-blocking memory allocation system can be implemented too and pd will be able to be used for live patching without blocking the audio thread for geometry processing or memory allocation..

[giuliomoro]

I have done a first pass at implementing some tcl procs to move away from passing tk commands. It is my first time at tcl and I don't have a deep understanding of the inner works of Pd, so please bare with me if anything about the style is not ideal. I #define USE_PDTK_CANVAS_PROC in any relevant file. This should be done through configure, but honestly this is easier for developing/debugging so I can switch it off/on per file as needed.

The code can be found here https://github.com/giuliomoro/pure-data-1/tree/1695-tests

I made a few NIT commits to begin with which helped me minimse the diff later on. Two commits are meant to be reverted later on as they are used only to log the actual docmds executed by the frontend in such a way that they can be used for automated testing:
a2f07d9 tcl: print docmds as they get executed. This is for testing purposes (i.e.: to produce consistent logs that are convenient to compare with pure-data-gui-tester) and should not be used in production without a switch to manually enable it
f227d05 tcl: evaluate one command at a time. This is mainly for testing purposes (i.e.: to produce consistent logs that are convenient to compare with pure-data-gui-tester and not thoroughly tested)

Throughout, I have been testing this with my pure-data-gui-testing, whose test patch has been extended to include coverage to all (I think) the tcl calls that have been affected by the commits on this branch (incl e.g.: deleting, selecting and moving objects). Where appropriate, I added workarounds in the tcl code to ensure that the tcl command being generated is exactly the same as in the code I am comparing against (a2f07d9). These can be removed at a later date.

What has been migrated:

• create, select, move, delete: plain objects, symbolatom, floatatom, listbox, commentbar, msg, bang and inlets/outlets. This doesn't include the actual text, which is created via the pdtk_text_* procedures.
• config, update: bang
• all iemguis: iemgui_label_font, iemgui_label_pos, iemgui_draw_iolets, iemgui_draw_move,
• all delete operations
• some move operations
• all text is still generated via existing - unmodified - calls to pdtk_text_*

I tried to stay reasonably close to the existing C functions. This means that I have all of the following procs on the tcl side: config create delete iemgui_label_font iemgui_label_pos move select update. This could surely be rationalised further (e.g.: config and update I think are only iemgui-specific). However in some cases I rationalised it a bit more. E.g.: text_drawborder()'s firsttime flag distinguishes between what semantically are a create and a move operation and so I made that explicit in the code.
I am minimising the number of arguments to be passed to the GUI, but still recreate the same pixel-accurate drawing. E.g.: creating a T_OBJECT passes the x1,y1,x2,y2 coordinates, creating a T_ATOM (float or symbol) passes those plus corner. In order to allow this, a minimal amount of the logic that is in the C data which is needed to draw the shapes from this data has been duplicated in tcl (see get_poly_coords()). Another example of what is needed to minimise the number of parameters passed is that only the obj's ID (ptr) is passed to the GUI in several bang methods, where it is then manipulated to generate the various OBJ, BASE, BUT, LABEL tags (see bang_get_tags()). On this note, pdgui_vmess doesn't seem to allow to pass a plain %lx which would be needed for this case, but only %p or .x%lx, so I have to trim the leading 0x from the argument in the GUI. It would be great if a type could be added that allows us to pass %lx without further processing needed on the front end.

I am currently not following the recommendation above

ideally the syntax would be closely modelled after how objects are stored on the disk.

This is in the first instance because I tried to keep the same argument order as the pdgui_vmess() calls I am replacing (weeding out unnecessary arguments), in order to facilitate my work. Furthermore, most of these calls (all except for create) do not really have a "stored on the disk" counterpart. The parameters I am passing to create include more details than what is stored to disk. E.g.: in the case above of floatatom with a -width of 5, the actual box's coordinates are computed in the backed based on font size and "width". This logic could be moved to the frontend if desired but it's not there yet. 
To complicate the matter further (at least for me), in order to send the object name's (or analogously the atom's or message's or comment's content) alongside the create() call would need some refactoring on the core side, as ::create() is called e.g.: in text_drawborder() while the text is set created in rtext_senditup(). Also keep in mind that if the call to create() looks more similar to how the object is stored on disk (e.g.: passing x0 and y0 coordinates and text), then the computation of the size of the box (rtext_width(), rtext_height()) moves to the frontend and so the hit detection problem manifests itself and needs to be dealt with.

Before doing any further work, I wanted to stop and check in first. Feel free to comment on all of this and the code: this was mostly a time-limited attempt I made in order for me to better understand the issues at hand, but I am not strongly attached to any of the code I wrote.

I thought I'd make a comment here instead of opening a PR because this is veeery far from being mergeable and this is more of a contribution to the discussion that is ongoing here, but let me know if a PR would be a better approach for this and I can do that.

[umlaeute]

please create a WIP-PR, as this allows us to add reviews to the code...

[giuliomoro]

ok done thanks #1763

Btw, as this morning I was locked away with my laptop, I also implemented create and move for msg, listbox and commentbar (and updated the comment above accordingly). Is it best to keep discussing here or on the PR?

[umlaeute]

depends. but i guess we can start discussing over there and then come back here if we come to a conclusion (or not).

[danomatika]

True. I use the UIView hit detection methods in PdParty but honestly I’d be very happy to drop all of that and let Pd do it for me. However, I only every sought to emulate the existing widgets.

[danomatika]

Note: we would probably still need primitive drawing for data structures to some degree.

[umlaeute]

yes, definitely

[mxa]

For Bela it would be very convenient (if not a requirement) to have the IDE running inside a browser. We are considering using svelte to do that, which is under MIT license. For a Pd-GUI, this GUI code may be used and displayed in an Electron app or similar. What are your thoughts on this?

[umlaeute]

my thoughts are limited, but I'm sure I do not want to walk the Electron road.

[Spacechild1]

Me neither. I would like a new alternative Pd GUI to be lightweight and fast - and Electron is neither. That being said, I think it's good if people try out different solutions and then we can see which one works best :-) Once the GUI API is fixed, I will start working on a Qt/QML based prototype.

[sebshader]

Purr-Data seems 'electron-like' in some way.. at least it's some kind of javascript/node engine, right?

[danomatika]

I think it’s Electron, yes. In any case, we want to make it easier to integrate Pd with all sorts of GUIs, Electron included, so I don’t see a need to digress.

[Spacechild1]

I think it’s Electron, yes

PurrData uses nwjs - which is very similar to Electron.

In any case, we want to make it easier to integrate Pd with all sorts of GUIs, Electron included, so I don’t see a need to digress.

+1 Guilty as charged.

[umlaeute]

totally.

i meant to say: I do not want to walk that road. if others do, it's their pick.

[mxa]

The discussion is circling around Electron (which apparently has a bad reputation as bloat?). Sorry, I probably should not have mentioned it, now I derailed the discussion in what might not even be relevant.
According to XPDA there are a few alternatives to it, namely NW.js, Tauri, Wails, Chromely.
Finally there might not even be a need for any of these environments to make an application based on svelte.

[millerpuckette]

As a thought experiment I'm thinking what it would take to adapt Pd to use GTK (which seems the most durable open-source GUI package out there; QT would be another possibility but it doesn't look free enough).
I see two ways to do it: (1) replace s_inter_gui.c with something that spits simple binary messages up and write a GTK program that parses them. Messages from GTK back to Pd can be kept as Pd messages as they are now; or:
(2) use the tcl interpreter to translate the existing messages to GTK ones.
(get some rotten fruit ready to throw at me for what follows):
I think (2) will be vastly easier to carry out because the very same Pd executable can then talk to either kind of front end.
Now... would the same strategy work for a web-based front end? This would entail writing (or finding) a tcl interpreter in JS.
OK, throw the veggies :)

[danomatika]

For the vanilla GUI, I would start by implementing the TK-specific stuff on the GUI side, one call at a time. I believe this was the overall plan and we would at least have a stable reference before testing with entirely new frameworks, at least to Pd.

[danomatika]

And honestly, I'm lazy... I would be happy for most of the logic to remain in the core and I, as someone with a non-TCL platform implementation, can simply just hand off canvas/widget drawing and dialog commands to a native UI framework while simply sending in input events. Then I no longer need to maintain my custom UIs for iOS and I gain updates and features as they are added.

But I don't want to have to parse loads of TCL strings manually, if I can avoid it.

[Spacechild1]

@millerpuckette

Pd to use GTK (which seems the most durable open-source GUI package out there; QT would be another possibility but it doesn't look free enough).

GTK, Qt and wxWidgets are all LGPL licensed and therefore compatible with Pd's BSD license. In my understanding, Qt is the most mature. (I only have practical experience with Qt, though.)

[sebshader]

I think the goal should be to leave as much up to the gui as possible.
I guess all of the graphs/connections would be stored/duplicated on the gui side too.
when pd creates an object it tells the gui to draw it. (also connections)
when the gui creates an object it calls draw itself and tells pd to create it. (& also connections)
idk how it would work to move objects around from pd if pd is unaware of the coordinates though.. I guess there could also be standard functions to 'move to location' or 'move by X pixels'
edit: not sure how patch saving would work either then..
I guess there could hypothetically be a supplemental file/section per patch to save graphical info but that's getting complex/backwards incompatible

[danomatika]

...then we give up IMO universality for performance and, as you mention, we add duplication of state which then needs to be synchronized in some way. I suppose most people here simply want a single nice, fast non-TCL/TK GUI. I want 1000s of GUIs which may not be the fastest but can work on many platforms in many contexts. If you can give me that, I will be happy. I will be disappointed if you overly optimize for a single new framework as I believe the strength of the "libpd-model" would be lost.

[millerpuckette]

If I understand this right, this would be getting rid of tk-isms like "-width" and backslash-escaping, so for instance

set ::sys_searchpath {{/home/msp/class/170/m170-function-library-v10} } ;

would become

set_search_path /home/msp/class/170/m170-function-library-v10

and

.x1d33820.c create rectangle 222 157 229 159 -tags {{.x1d33820.t1d34080o0} {outlet} } -fill black ;

by

create_rectangle .x1d33820.c 222 157 229 159 .x1d33820.t1d34080o0 black

or something like that, correct?

[umlaeute]

hmm. i totally do not see the point of such a conversion. which problem does it solve?

getting rid of tk-isms like "-width"

-width is actually just a un*xism, called "flags" (or options)
there's nothing inherently bad about it. i'm pretty sure i would prefer options (-fill black) instead of positional arguments ("the 7th argument is the background color").
it obviously requires parsing the "command", which is slower than just taking the nth argument. but it is easily extendible if you ever want that. (do you prefer #X obj 70 104 hsl 170 20 0 127 0 0 empty empty empty -2 -10 0 12 #fcfcfc #000000 #000000 0 1; or #X obj 70 104 hsl -width 170 -height 20 -lower 0 -upper 127 -scale linear -steady -labelX -2 -labelY -10 -labelSize 12 -font 0 -bgColor #fcfcfc -fgColor #000000 -labelColor #00000? the latter is much more verbose, and i claim that this is a good thing.)

getting rid of tk-isms like backslash-escaping

well, this is a general problem of transmitting data over a text-based interface with space as a delimiter. if we want to be able to distinguish between a space-delimited list of two symbols foo and bar and a single symbol with a space foo bar, we need some sort of escaping. backslash is just the most universally used. so as long as we have a text-based protocol, we will probably have to keep it.

otoh, the "backslash-escaping example" given is (if i go t this correctly) {{/home/msp/class/170/m170-function-library-v10} } (which uses curlies instead of backslashes).
the outer curlies are a notation for "a single list as an argument" (as opposed to multiple arguments formed by the list elements).
i think (after having implemented s_inter_gui.c) having a "list type" makes a lot of sense, and if we want it we need some way to express it. {} is just one way to do it, but i don't see any drawbacks as compared with e.g. python's bracket notations []. the obvious advantage is, that it just works with the current Tcl/Tk GUI :-)

one could obviously get rid of the inner curlies, and escape any spaces with backslashes. i used curlies because it's much faster to add a prefix and a suffix to string than to do a search-and-replace which will grow the string by an unknown amount.
however, the entire "escaping" is now in s_inter_gui.c so its trivial to replace the quoting.

OSC/FUDI-like patterns

actually i think that .x1d33820.c create rectangle 222 157 229 159 is much more like FUDI than create_rectangle .x1d33820.c 222 157 229 159. it resonates with [; pd quit 1( (note however that i left out the flags in the first example; but then I also left out the typestring in the 2nd example).
I don't really see the point of having OSC-like typestrings, esp. with a positional-argument format like create_rectangle xiiiss .x1d33820.c 222 157 229 159 .x1d33820.t1d34080o0 black, where I assume that the 2nd (real) argument of the create_rectangle proc (222) is the X-position, which will probably always have the same type.

[danomatika]

Good points. I hadn't considered command line-style argument parsing which also works and, as you say, is extensible in the future. I also agree that FUDI-like is good as long as escaping is well-defined (curly braces, quoting, \ , etc). There are tons of precedents for handling an argc,argv pair. Honestly, I don't care how this works as long as it's well thought out and documented (which you've done). I generally reach for OSC first, hence my first hypothetical.

UPDATE: As per FUDI-like, I also like this in that it's working with an established pattern within Pd already. IMO we don't want to introduce a different pattern unless really needed for some technical reason. Keeping similar patterns eases maintenance and understanding of the codebase.

[Spacechild1]

I'm a bit confused... I thought the idea of pdgui_vmess with its explicit type tags was to abstract the GUI messages, so that each GUI backend could implement pdgui_vmess as it sees fit. The current Tcl/Tk GUI creates a Tcl string and sends it over the network. A JavaScript GUI would rather create a JSON string. A Qt GUI would probably create a custom QEvent and dispatch it to the UI thread. Etc.

I don't really see why we would need to specify a general text protocol at all?

In my understanding, a GUI backend would be essentially a plugin that implements a small set of abstract functions:

1. start the GUI
2. stop the GUI
3. send a message to the GUI
4. poll the GUI for messages

[umlaeute]

ah, well.
i think both are are OK.

my general assumption was that most GUIs might want to follow the native transport, and thus leave the pdgui_vmess() implementation as it is (a text-based network protocol).
this would allow for a given Pd-binary to switch to an alternative GUI without any need for "plugins" (as in "externals" that re-implement pdgui_vmess())

iff this is not enough for a given GUI, one might switch to a different pdgui_vmess() implementation, that uses an entirely different protocol (the format could be binary/JSON/...; the transport could be network/some other IPC/in-process calls (to different threads)/...).

for the latter, I thought more about re-compiling Pd with an alternative s_inter_gui.c resp s_inter.c than providing a plugin infrastructure - but then: changing to the latter should be doable once there is an actual need for this.

[Spacechild1]

Ah right, I see now how a text protocol would be useful for GUIs that remain as seperate processes. For traditional "in-process" GUIs it would not make much sense to serialize messages as text, just to deserialize them on the UI thread :-)

for the latter, I thought more about re-compiling Pd with an alternative s_inter_gui.c resp s_inter.c than providing a plugin infrastructure - but then: changing to the latter should be doable once there is an actual need for this.

You're right. Both can be easily added later.

[millerpuckette]

PS yes, keeping the logic in the core will make it much easier to make future adaptations when GUI packages come and go.

[danomatika]

Yes, exactly. IMO it should be possible to provide basic drawing commands through a layer which external authors can utilize. Then externals would theoretically work the same between different GUI implementations using the same pd core... I think it should be as easy to create a new GUI for Pd as it is to run a patch in libpd within a different environment.

[danomatika]

I think we can define and refine using the vanilla GUI and TCL/TK while making it possible to experiment with other toolkits/frameworks.

[giuliomoro]

For instance, one hypothetical OSC-like approach is:
create_rectangle xiiiss .x1d33820.c 222 157 229 159 .x1d33820.t1d34080o0 black

That's not too far from the extensive work that @umlaeute has been doing in #1765.

then convert everything after form string to int etc relatively easily

One of the leading principle behind #1765 was: keep the commands as valid tcl commands, but simplify and abstract them enough that they can be easily parsed and interpreted by "any" frontend language. The current implementation assumes that types are agreed upon as part of the message type (e.g.: create_rectangle is expected to have xiiiss arguments, without need to send xiiiss every time).

[danomatika]

If there is a technical approach to handle both cases, it would be a win overall. Hopefully without splitting dev resources.

It's reasonable to say "any Pd GUI should handle atoms, tcl, bang, etc" in its own way but could fall back to specific drawing commands. I would do this by default for my own implementation since I want it to match the vanilla GUI in look and feel.

Let's think a bit out of the box. The current state of the Pd GUI is really dictated by limitations of Tk that simply don't exist in many other frameworks.

I feel the current state in look and feel is the "baseline" I never suggested any future GUI has to abide by it or how its communication works. I am just concerned that logic of the core will get dropped in favor of "making Pd into Max" and a new reliance on another dependency.

[Spacechild1]

If there is a technical approach to handle both cases, it would be a win overall.

I will try to team up with @umlaeute for a POC.

[sebshader]

maybe some notion of 'object' can be conceptualized on the gui side.
Then gui-non-specific drawing commands for primitives could still be emitted with the object as a 'tag'. Essentially this is what already happens, but it would be generalized to calls that would work with any gui environment.
Kind of like @Spacechild1 suggested, then certain classes could even decide on an implementation-by-implementation basis and set some class-specific flag for if they want to go with their custom implementation and disable sending the primitive draw commands, or to receive them. But even when they receive those commands they would be 'for' discrete objects.
when pd sends commands to tcl, it already knows the class/object. So I think all this would take would be to standardize things and also have tcl send object info to pd as well (send 'pos X' to object Y when a slider moves or something)

[millerpuckette]

"The current state of the Pd GUI is really dictated by limitations of Tk that simply don't exist in many other frameworks." - I disagree. I disdained all the eye-candy GUI stuff that was around in 1996 in favor of a more minimalistic, less visually distracting design. Tcl/TK was the perfect toolkit for me to get Pd looking very nearly the way I wanted it.

[Spacechild1]

@millerpuckette You can get the same minimalistic design with modern GUI toolkits - just with less bugs and much better performance. There is no inherent reason why the GUI has to consist of a series of drawing calls on a (Tk) canvas. You can achieve the same result with SVG graphics, for example. But this probably wasn't a viable option in 1996...

Aesthetically, I'm very much on your side, btw :-)

[millerpuckette]

I think that's right, we don't need the type-strings since the functions themselves will know what type arguments they expect.
Meanwhile this would differ from #1765 in that we would only have functions to draw primitives (lines, rectangles), not entire "cnv" objects - and inlets/outlets/connections would be known to the GUI only as their drawn shapes, not their meanings.
A first step then would be to start implementing a whole bunch of simple tcl/tk functions that do all the things Pd uses TK commands to do.