Dynamic Multi-Way Connection System

[mjholtzem]

Right now outputs can always have multiple connections and inputs can only have one connection. It would be nice if this were more flexible and you restrict an output to one connection or allow multiple connections for an input.

[Seneral]

Can you tell me what you'd use it for? I know there are lot's of other use cases than the standard calculation but I can't imagine any that you couldn't accomplish with the current system... maybe with a few tweaks;)

[mjholtzem]

Well it could be useful even for standard calculation such as adding an
arbitrary number of inputs but I'm thinking in the context of a branching
dialogue system. You generally would not want a single decision to lead to
multiple separate paths but you would want those paths to converge.
On Jun 4, 2016 12:18 PM, "Levin G." notifications@github.com wrote:

Can you tell me what you'd use it for? I know there are lot's of other use
cases than the standard calculation but I can't imagine any that you
couldn't accomplish with the current system... maybe with a few tweaks;)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#74 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABd589v176Tzrx9_o9K9O-qS-1q6J03Nks5qIcGPgaJpZM4IuIcq
.

[Seneral]

Ok, so just to be sure; What do you think will be the best implementation? With preprocessor checks to let the dev switch for their tool, applying this change to the whole framework, or by making an option in connection type or something similar dynamic per knob/node?

[mjholtzem]

I think both the input and the output knobs and having a boolean for allowing multiple connections in the constructor.

[Seneral]

Ok, doing scene saving and improvement of the save system right now, will work on that afterwards:)

[Seneral]

Facing several problems with this right now.
Just consider, when having multiple connections to one input, how do you want to loose one specific connection? Previously you'd click the input to drag the single connection. Now, there isn't one to drag, but possibly multiple. That means, I'd have to find a different way for connection deleting - clicking on curves. Would take horribly long implementing that, trust me! :|
Any ideas and suggestions?

[mjholtzem]

hmmm. Well I have seen one other node editor that drew a button half way
down the connection spline that could be clicked to move the connection.
Might be slightly easier than clicking on the actual curve.

Alternatively (and this doesn't strike me as particularly elegant), if you
restrict it so that you can't connect multi to multi, but just single to
multi, or multi to single, then you could do it similar to how it is done
now and release it by dragging away from whichever end of the connection is
single.

Other than those two options I'm not sure if there is another cleaner way
to do it

On Sun, Jun 12, 2016 at 6:30 AM, Levin G. notifications@github.com wrote:

Facing several problems with this right now.
Just consider, when having multiple inputs to one knob, how do you want to
loose one specific connection? Previously you'd click the input to drag the
single connection. Now, there isn't one to drag, but possibly multiple.
That means, I'd have to find a different way for connection deleting -
clicking on curves. Would take horribly long implementing that, trust me! :|
Any ideas and suggestions?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#74 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABd58zV5-JwEixJsijOuGnqofgqKRGCYks5qK_vTgaJpZM4IuIcq
.

[Seneral]

First option will probably do for now, but only for curves where both knobs are allowed to have multiple connections. I don't know when I'll implement this, though, expect it to be weekend:/

[ChicK00o]

Hi, very new to comment anything on this, so take my words with a grain of salt [i have to check the code also]. the requirement is very promising. So some suggestion that come to my mind for controls is

1. Hold "ctrl" and select the two knobs that need to be disconnected
2. again "joining" already "joined input output knobs" will remove the connection [i.e. disconnect them]
   or some mixture of this could be possible

[Seneral]

Would the concept described in #91 be an acceptable solution for you? Any feedback on that issue appreciated:)

[Seneral]

I think it would be best that in the future it is handled by the traversal system that is used (see #90), so you will be able to use your own and specify own rules. Basically all limitations are then lifted off the framework and you can customize anything that was previously restricted due to how some algorithms work:)

[Seneral]

I've begun implementation of this (latest commit as of writing this is (5a56c5)
Theres something wrong with the serialization taking place when saving to an asset file, because going into playmode (causing an assembly reload) doesn't break anything, but saving and loading a canvas does.
The main thing that seems to break is the connections and connectionrules Lists, on the ConnectionKnob.
I Suspect either I broke EditorSaveManager, or some other code that is run when saving to an asset that isn't lastsession.asset
Could you help out with the serialization? Its fine if you don't.

The CalculationGraph is completely disabled ATM, and will need to be reimplemented. I have held off on this because of the broken serialization.
There are also a few decisions that should be discussed, such as the creation of connections from inputs. I have currently allowed creation of connections from inputs, but the original behaviour was to only allow creation of connections from outputs. To replicate the original behaviour, simply change true to !isInput in CR_Directional.CanStartConnecting() (its marked with a TODO)
GUI images Can be colored using GUI.color, instead of tinting the texture, although I have no idea how big an impact on performance using GUI.color brings.
Might as well add a GIF

I Changed the Knob image to a different image, but that can (and probably should) be set back to the original.
@Nopey in #70

[Seneral]

@Nopey I got an idea on how to edit connections with multiple connections at both knob ends, which is a bit more practicable than the previous approaches.

1. Click once on a knob, and multiple small 'Indicators' pop out of the knob, one for each connection (with color, positions, etc indicating which knob belongs to which connection)
2. Clicking and dragging one each of those knobs will edit the corresponding connection

I think implementing that is a bit easier, but needs a bit getting use to...
What do you think?

[Nopey]

Makes sense, I think maybe the connectionknob should be asked if it wants to do that, and by default the connectionknob would ask all of its rules, but it could then also be overriden by the programmers using this plugin in their projects.
Graphically, I think there should be a small image that is connected to the real knob, that acts as the individual connection. It wouldn't need the connectionRules' graphics drawn on it, but could easily copy the color of the knob by copying its color public variable.
I think the small indicators would fan out in some degree and distance, determined by the number of connections.
There could be a straight line between them and their parent knob, or a bezier curve, with their direction exactly facing the center of their parent knob, and their parent knob's direction being as it is now.
Heres a mockup. It demonstrates the straight line vs the bezier line.

I like the bezier better, but this is just a mockup, and not fully representative of what the finished product would look like.
If we wanted to get fancy with things we could also add an animation for transitioning between no indicators to all of them, by lerping their position from their parent to their final position, and lerping their external direction (for the connection between them and whoever they're connected to) between their parent's direction and their final direction. (and in reverse for hiding them.) Transition should take no more than half a second in any circumstances, If we were to make it.

Another option would be to draw a small indicator so close to the parent knob that drawing a line isn't necessary, but this might cause problems if there are too many connections on the parent knob, it wouldn't have enough surface area to place all these indicators.

Alternatively, we could ditch the indicators idea and move all of that interaction to the connection's curve itself. Removing a connection by middle clicking on a connection, plucking by left clicking on a connection closer to one end than the other, with a zone in the middle where you can't pluck. (maybe have a visual cue?)

Pros and cons exist for all of these, and I'll only go over some of them briefly.
Option 1: frog feet

• +Allows for many connections on a single knob to all have indicators that are clickable.
• -Might overlap with other knobs on the same node, especially if the angle is set very wide.

Option 2: Buckyball

• +No messing with the connection positions, just messing with the color.
• -Indicators might overlap amongst themselves

Option 3: Wires

• +No Added UI Elements, except maybe an indicator of where you can pluck from.
• -Is unintuitive, because there is less visual representation of what you're doing.
• +Few hooks required
• -Collision detection for bezier curves. uh oh..

I like the wires the best, but implementing the click detection could be a nightmare. gonna go research that, I'll post again to report my findings.

[Nopey]

The easiest method would be to do collision detection on a low resolution version of the bezier curve's line segments. theres fancier ways, but I have not yet learned some of the math used, so unless you have enough knowledge of math to really dabble with the research papers, I'm just gonna say we use line segments.

[Seneral]

Hi, sorry for not responding. Still processing your first response...
Don't get into bezier click detection to fast. I don't think it will turn out too intuitive for what it will take to implement. It is simple line segment click detection, but still not optimal...

Option 1 or 2 sounds better, although we would need to make a visual mockup first;)
Might be as simple as adding buttons like a few pixels along the curve, although that could cause other clipping problems.

[Nopey]

I think that using a hybrid between the first two may be the way forward, because the the buckyball one is optimal before you run into more then 5? connections, and then the frog feet becomes necessary. one (final?) option would be to expand them into a flat array parallel to the nodes side. This would only work with one knob expanded on a node at once, and is probably the easiest to implement as it requires no angle Math.

…

On Dec 30, 2016 4:31 PM, "Levin G." ***@***.***> wrote: Hi, sorry for not responding. Still processing your first response... Don't get into bezier click detection to fast. I don't think it will turn out too intuitive for what it will take to implement. It is simple line segment click detection, but still not optimal... Option 1 or 2 sounds better, although we would need to make a visual mockup first;) Might be as simple as adding buttons like a few pixels along the curve, although that could cause other clipping problems. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#74 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEVU-4puGSZ_GGJ9ygmci3fIO2GnEtm2ks5rNaJdgaJpZM4IuIcq> .

[Seneral]

Sorry for the late response. After the last merges I now have time to try and implement a prototype of 1. said structure of NodeKnob and subclasses and 2. editing interface for multiple connections :)
Will update when I have something to decide which to use!

[paintbox1]

Is this really such a huge problem? Like mjholtzem said, "if you
restrict it so that you can't connect multi to multi, but just single to
multi, or multi to single, then you could do it similar to how it is done
now and release it by dragging away from whichever end of the connection is
single."
I still cant figure out any usefull scenario for multi to multi.

[Seneral]

Good point. For now, we can assume that this is the case to get most users satisfied, and then expand on that later. Because you never know how users end up using this framework, so we have to be prepared;) But still, very unlikely use case indeed...

[Seneral]

But still, when allowing one output connections and multiple input connections like in a dialogue editor case, I'd imagine the current approach isn't quite intuitive anymore. You'd have to drag the connection from output to input, the exact reverse way...
So for that, an approach like we discussed earlier could actually be better or atleast an option IMO.

[paintbox1]

I think it should always be possible to create a connection from output to input.
Why should I be allowed to connect A to B but not B to A?

[Seneral]

That's the problem. When you have a multiple-single connection dependency, no matter which way, you currently only have one way to draw a new connection, from multiple to single, and only one way to edit a connection, from single. Both is done by simply dragging from the knob.

When we want to allow it the other way around, we have to add additional options besides dragging. I want to avoid shortcuts (eg. Shift-drag) and especially when editing from a multiple connection knob, where the connection amount is variable, that won't work anymore. So the only way I see is some sort if graphical solution, a GUI popup that appears once clicked on a knob, where you can create (center) and edit multiple connections (fan around center), like we discussed above.
Hope you understand now why we made this decision:) If you have any ideas how to make it better, very much appreciated;)

[Seneral]

Haha, now I understand:) Sorry...
But you see, even when we get the NodeKnob fully modular, it's not the perfect fit for every case...

[paintbox1]

@Seneral I totaly see your point regarding the real world cable analogy and that might be the most intuitive one for people who have never worked with a computer interface. I just imagine myself working with these editors hundrets of hours and want to be able to use it as fast as possible. Thus I'd consider dragging the newbie approach and clicking the pro user approach. Therefore I think, it's ok to not be able to do all cases using the newbie approach. As I'm writing this, I'm recognizing that I should have pointed it out this way earlier. Please bare with me^^

Regarding structuring: I'm not sure, I understand Nopey's suggestions completely but the way I see it, those modifiers would also strip away connection types from the knob class, which I think is a good thing. Mainly because it is solely useful for calculation, which is just overhead for statemachine use-cases like mine. In fact, I'd love to see all the calculation stuff gone from the core.

Finally one suggestion for "creating" knobs: Manual instantiation like that should be necessary only for dynamic knob counts. (For Example a user adding new choices in a dialog node). For static knobs, which always exist on that nodetype I'd like to add them as fields or properties with Attributes like

class MyNode : NodeBase
{
 [Knob(Name = "My Special Knob", NodeSide = NodeSide.Right)]
 [ConnectionTypeModifier("float")]
 [ConnectionDirectionModifier(Dir.Out)]
 [ConnectionLimitModifier()]
 ConnectionKnob mySpecialKnob;
...
}

That way, it would be even more modular and readable. Plus you dont need to find your special knob in the knob list, when you need to do something with it.

[Seneral]

Yep, have also been thinking of attribute based knob creation. But then again, with so many attributes needed to specify a knob, this would be of no benefit.
I would again go with some sort of mix between strongly-typed subclasses of NodeKnob/ConnectionKnob that is defined through a set of Knob Modifiers. I like that modularity, but I need some sort of strongly-typed behaviour which 1. matches the principles of C# and 2. allows for short and easy knob creation.
Will make a draft once I have more time:)

[paintbox1]

But then again, with so many attributes needed to specify a knob, this would be of no benefit.

Yes that example would be a bit too much for me too, I just stressed it to the extreme to show, that nothing is missing from your example above.

I imagine an ordinary Knob more like this:

[OutputKnob(Name="Sum")]
[ConnectionType(typeof(float))]
private ConnectionKnob sumKnob;

What exactly do you mean by "match the principles of c#" ?

[Seneral]

The example I posted was an ordinary knob with features it needs in order to work like the current ones, so no extreme stress - in the end, definitely too long for a normal knob declaration.

So we need to find a different way, maybe some kind of preset. This is what I meant with a mix between both. We could have a subclass of ConnectionType specify the modifiers it has and then use a single attribute and instantiate the knob based on the ConnectionType subclass (the attribute taking additional modifiers for flexibility sake:) ).

That way we have the benefits of both sides combined, IMO. You can target ConnectionTypes strongly-typed if you want, have short and easy creation but still have the benefit of the modularity of the modifiers.
Compability check would only need to be done in ConnectionKnob by checking the requirements of each individual modifier, not individually by each subclass.
So for a knob with a type-modifier to accept a connection the other knob needs a type modifier with the same type.
For a knob with a Dir.Out modifier the opposite one needs a Dir.In modifier (or none?).
A ConnectionLimit modifier only needs to check if there is already a connection.
And so forth.

[JoshAshby]

I'm a little behind on following this issue but I'll go ahead and throw my thoughts out here as maybe they could influence some of the design for this.

I've got a pretty different knob/connection and input system setup in my own framework but I solved the mutli-to-multi issue pretty easily I think. Knobs have an "expanded" state that is triggered on the first click, this causes all the positions for the connections to be fanned out next to the knob. On the second click, the knob initiates building a new connection (although this could be changed to say middle or right click or even a keyboard shortcut if I wanted).

When a knob is expanded, the connection draws the curve from the offset position AND it draws its own "knob". Subknobs are sorted by the connections other knobs position, to ensure they are always neatly organized. When this "subknob" is clicked it deletes the connection. Heres a gif of making a number of connections:

(The circle in the knob icon just means that these are bi-directional knobs, they ge arrows when they are input or output only :-) and the brief red curve means that the knob it was over wasn't available, in this case because it was set to a connection limit of 1 whereas the other knobs are all set to unlimited connections.)

And what it looks like normally, ie when the knob doesn't have focus

And finally another gif of the removal process, even with a messy multi-to-multi on the unlimited (1) knob:

I think most of this could be accomplished with the current state of the code base in this project, although I don't know about input handling on the knobs and "subknobs"/connections. You'd probably want to basically make a copy of NodeKnob.GetGUIKnob that returns a connection specific location if the knob is selected, and update Node.DrawConnections to handle getting its specific location back. At least that's approximately how I solved it. I also have a bool Available() call on knobs that connections check when they are being initialized, which has a check for if the max number of connections has been reached for that knob or not.

[Seneral]

Wow that's like exactly what we settled on (the expanded state of the knob with the connections fanning out). Really nice to see it working fine as I didn't have the time to create a mock-up yet.
Thanks! :)

[JoshAshby]

If I get some free time soon I can certainly take a look into seeing just how far I can get if you want since I've implemented this once before quick successfully; although I suspect I might hit some friction around clicking on the "subknobs" seeing as in my framework they are part of a Connection object that can be returned as the current target to the input control system, and I haven't seen the equivalent of that in here. Maybe that's just a matter of making a smarter knob that knows which "subknob" is at the mouse position instead?

[Seneral]

Sorry @JoshAshby for not answering... Go ahead if you want:)
I think it would indeed be better if the knob would handle the subknobs as you proposed, which would require moving the drawing to the knobs.
Would like to help you but have very few time while I'm preparing to release my first tool. Will try to work on it after most of the trouble is done though:)

[Seneral]

I'm currently finally implementing this:)
I did settle on a mix of the fully modular approach and the fixed-subclass model. I call it: Subclass'n'Property-based Connection system :D

Let me tell you why. The complete modifier based connection knobs have the advantage of being super flexible, but it also is weakly typed, it needs complex, repeated setup for each and every knob and all routines accessing these knobs are fragile because they need to search a list to know a knobs property.
The mix-based Knobs, on the other hand, have the advantage of being strongly-typed, requires less individual setup and is easier to work with in code because properties are declared and directly read out. Only disadvantage is that it is not FULLY modular, but IMO it is just as flexible.

Here's why:
The new base class is ConnectionPort, which is basically a simple connection between nodes, whithout the concept of knobs at all. Say, statemachine transitions.
It is extended by ConnectionKnob, which introduces the placing on the node sides and positioning with regard to the NodeGUI, and bezier curves.
Finally, theres ValueConnectionKnob which extends that even further to be limited to a type and to contain and pass a value.

The concept of hard-coded Inputs and Outputs, however, is fully gone. Ports in general can have an input or output flag set but also none if desired. This is totally ok with the calculation routines, checking for input/output is no big deal. It also does not limit a certain port to have only one connection, this is later improved to get UI support ala @JoshAshby .

Also, creating knobs in nodes is now way easier. The approach of declaring the Knobs (or just ports) in the node from @paintbox1 is taken, they are read in and created and kept up to date with their declaration in the node. This means adding/removing knobs in the source code will finally adjust existing saves. Dynamic knobs are seperately handled and do not underly such restrictions.
For each of these connection stages, ConnectionPort, ConnectionKnob and ValueConnectionKnob, an attribute is delivered, which serves as a specification for the connection.
Additionally, we might be able to extend that to allow attributes for knob lists which adapt to some other list and allow easy dynamic knobs for most cases.

Btw, the removal of the explicit Input/Output allows completely flexible routines to take place, passing values backwards up the graph, whatever. If you do not need to pass values, you can use ConnectionKnob and extend it yourself if need be.

Hope that's clear and a good solution for all. The fully modular approach was a bit radical to me and had some flaws that we would probably later regret...
Still, hope to hear your suggestions. I came far today but still can adjust if requested:)
Seneral

[paintbox1]

Hi @Seneral, I'm surprised, you want to take the approach from my example. Have you found some sort of compromise with less attributes? In fact, I'm excited to see a code example of your new approach :)

[Seneral]

Well there are only three attributes with parameters, some core options, like type/value handling, which are based off the choice of subclass.

[Seneral]

So, although it is not fully implemented yet (especially the port creation from attributes), here's what the Calculation Node would now look like:

public const string ID = "calcNode";
public override string GetID { get { return ID; } }

public override string Title { get { return "Calc Node"; } }
public override Vector2 DefaultSize { get { return new Vector2 (200, 100); } }

[ValueConnectionKnob("Input 1", Direction.In, "Float", ConnectionCount.Multi, NodeSide.Top)]
public ValueConnectionKnob input1Knob;
[ValueConnectionKnob("Input 2", Direction.In, "Float", ConnectionCount.Multi, NodeSide.Top)]
public ValueConnectionKnob input2Knob;

[ValueConnectionKnob("Output 1", Direction.Out, "Float", ConnectionCount.Multi, NodeSide.Bottom)]
public ValueConnectionKnob output1Knob;

protected override void Init () 
{
	// Set up other stuff
}

Do notice some other changes I made to the creation, all node properties are now actual overridden properties - this means, changes made to the source code now actually reflect in already created canvases, this includes the ConnectionPorts aswell since the knob declarations can be validated on each load easily.
Also, the attribute properties are not all mandatory, ConnectionCount and NodeSide are optional and in this case could well be left out to get back the default behaviour of current knobs.
Finally, be aware this is a calculation example, there are two more (base) types that have other fuctionality, each with their own attribute and custom properties...

Feedback appreciated:)
Seneral

[Seneral]

So I finally got together a working version yesterday, took a bit longer because of a startup error I had to find a workaround for (when window is open on unity startup, there seem to be two initialization stages with a weird reset in between which caused errors).

Anyway, it is looking way better now. Node knob (or rather, port) definitions are now concrete because of the variable-based definition and attribute-based creation.
This does mean save types are nit compatible, though. Opening an old save erases node position and all connections (not knobs as these are now property-based).
But it has the benefit of not having to create a hacky second node on initialization for checking with the node knobs, instead it works on the node knob definitions, fixing #72
After cleanup and finishing import/export for #43
I'll create a PR with everything included jn seperate commits for testing:)
Seneral

[paintbox1]

Sorry to answer late. Those attributes look like a nice solution. How do the other base types look? Im particularly interested in these because of my dialog editor :)

[Seneral]

So you have a hierarchial structure with increasing features bit also complexity. Each has their own attribute for creation and their own styling level (all inherited). More on that later.
So you have the base ConnectionPort which solely serves for a connection between nodes, without the concept of knobs. So it is drawn as a line between node body to node body like a transition. Only mandatory attribute parameter is name, but you can also optionally specify a direction, style preset, maximum connections, etc.
ConnectionKnob extends ConnectionPort to introduce the knob concept. You now also need to specify the direction the knob has, in or out and can specify a fixed position. The style now includes the knobTextures.
Finally ValueConnectionKnob extends that even further to pass a value along the connection. You now have to specify the style because it includes the value type.
The styling works very similar to the previous connectionTypes, but for styles and only for the ValueConnectionKnob a type.
More informations when it's ready:)

[paintbox1]

Sounds very promising. I will check it out, when its done.

[Seneral]

Finished. Will create the PR tomorrow morning, need sleep now:)

[Seneral]

Officially merged and committed:)
Really happy how it turned out. Please give me lot's and lot's of feedback:D
Seneral

[Seneral]

Next up is multi-multi-connection editing.
I think I'm going a different way than what @JoshAshby proposed and what we had settled on.
Simply because that behaviour would have to be the default for ALL multi knobs, because we can't differenciate between multi-multi and multi-single there. And that extra, theoretically useless extra step for normal editing is just too much IMO.
Besides, I think I found an easier and consistent, maybe even more common way to edit the connections. We could allow creating a connection by drawing/dragging from a knob as before, but from both multi and single, in two ways. Deleting connections could then be done with a connection hit test (probably easier to implement than the extra knob popup). Hit-Test would not need to be performed on hover, but only when a specific event occurs. For example right-clicking a connection - it could either bring up a context menu with a delete option or even directly delete it...
That means creation and deleting would be consistent for all combinations of multi and single, has no direction (is mirrorred) and is cleaner.

What do you think?
Will take some more time nevertheless, am busy atm.
Seneral