Separate UI and processing v2, Simple approach

[ly29]

Why?

This is is a standard paradigm in all system so I don't is needed to argue why it is a good thing but I will state briefly why it is a problem.

Today processing and some UI handling is contained in node update() function. This function is called from two contexts, by blender directly on editor changes and from the Sverchok update system. The intended function of the update function is clearly stated in the documentation: Update on editor changes. Editor changes in this context is: creating sockets or linking sockets. Moving a node, renaming, selecting a node etc isn't an editor change.
However, every time you link a node or create/move/delete a socket a new update event is issued, there isn't any recursion check going on here. Note that this also includes the def init(self, context): function, update will be called for each socket you create, it is needed to check for the existence of sockets on every call to update to avoid a cascade off non critical errors.

From Centers polygon node for example.

    if self.outputs['Centers'].links or self.outputs['Normals'].links or \
KeyError: 'bpy_prop_collection[key]: key "Centers" not found

These errors doesn't affect the operation of sverchok but they are annoying.

How

There are a two main alternative approaches in my head:

The simple

We move the processing to a new function in every node which will be called from the sverchok update system.

Pros

• simple to implement
• no new classes
• direct access to inputs etc

Cons

• No clear separation of ui and processing code,
• harder to compile the node tree and optimize

Example code from the in range function which isn't such good example since doesn't need an update function.

    def update(self):
        inputs = self.inputs
        outputs = self.outputs

        # outputs, end early.
        if not 'Range' in outputs or not outputs['Range'].links:
            return

        param=[inputs[i].sv_get()[0] for i in range(3)]
        f=self.func_dict[self.mode]
        out = [f(*args) for args in zip(*match_long_repeat(param))]
        self.outputs['Range'].sv_set(out)

Becomes:

    def init(self, context):
        self.inputs.new('StringsSocket', "Start").prop_name = 'start_'
        self.inputs.new('StringsSocket', "Step").prop_name = 'step_'
        self.inputs.new('StringsSocket', "Stop").prop_name = 'stop_'

        self.outputs.new('StringsSocket', "Range", "Range")
        self.state = "ACTIVE"

    def process(self):
        inputs = self.inputs
        outputs = self.outputs
        if not outputs['Range'].links:
            return
        param=[inputs[i].sv_get()[0] for i in range(3)]
        f=self.func_dict[self.mode]
        out = [f(*args) for args in zip(*match_long_repeat(param))]
        outputs['Range'].sv_set(out)

The complete

Compile every node into a something that can be executed and optimized on the way, for example several math nodes could be optimized into one expression, fine tracked control of what needs to be updated is possible.

Pros

• Longterm
• Possible for fine grained control
• We can switch to a different back end

Cons

• More complicated to design right
• Harder to implement, bigger changes

I am sure that the second solution is the right one but I an not certain that I am up to the task of designing it correctly.

[ly29]

This is a new issue since the old one got longer than needed and contained many other things. #152 wasn't moving forward.

[zeffii]

does process only get called by sverchok if state==ready ?

[ly29]

That is my idea, this would happen for separate graphs on a layout, process would only be called if a graph changes. Separate layouts and unconnected graphs in on layout are done independently already today in preparation for this kind of behaviour.

>>> ng.get_update_lists()
([['Random Vector', 'Vectors in', 'Vector Math', 'Viewer Draw'], 
  ['Integer', 'List Range Int', 'function', 'Debug print']], {})

[ly29]

Doing the simple one would solve many problems and work of restructuring could begin like that, I think I will make branch to test that approach.

[nortikin]

to test simple solution?

[ly29]

Yes.

[ly29]

It is working, well only 3 nodes, and before expanding it I want to make easier to code etc. You can have a look at the ui-separate branch if you want.

I'm still not convinced that this is the right approach but I want to test a bit.

[nortikin]

yes, it must be grounded and rethinked some times before implementation.

[ly29]

A node group tree type for sverchok, hidden from the user interface.
Where sverchok nodes also can live.

>>> D.node_groups['SV-GROUPS'].nodes['Debug print'].bl_idname
'SvDebugPrintNode'

Just playing with some ideas

[nortikin]

and now when you did one list rulez all nodes, my shift+A not showint popup lists

[ly29]

Yeah, something went wrong with that sync, I am working on something nice, will sync again in a while. There isn't much to see here yet.

[ly29]

Prototype node groups

Rough and not nice looking at all. Probably buggy also. But it works.

[ly29]

Lot of work left to make this work well.

processing Integer
processing Group
Node List Range Int.004 is ready
processing List Range Int.004
processing Integer.001
processing Group.003
Node List Range Int.004 is ready
processing List Range Int.004
processing Debug print
[[1, 3, 5, 7, 9, 11, 13, 15, 17, 19]]
[[1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39]]

[ly29]

We can implement something like a for and do while loops with similar techniques as these. And many other things.
The nodes takes over the flow control in this case, doesn't feel quite right but was simple.
https://github.com/nortikin/sverchok/blob/ui-separate/nodes/basic/group.py

[ly29]

@nortikin
If you want to test it is kind of working with the few nodes it has, very much a work in progress. A lot of debug printing.

[ly29]

Simple iteration test setup. This is really just a proof of concept, this isn't necessarily how it should work.

processing Integer.003
processing Iteration
Node function.001 is ready
processing function.001
Node function.001 is ready
processing function.001
Node function.001 is ready
processing function.001
Node function.001 is ready
processing function.001
Node function.001 is ready
processing function.001
Node function.001 is ready
processing function.001
Node function.001 is ready
processing function.001
Node function.001 is ready
processing function.001
processing Debug print.001
[[9]]

#242

[ly29]

I starting to think that packing the recursive parts into a node group is pretty good way to deal with these scenarios.

[zeffii]

Exciting developments @ly29. I'm getting flashes of PureData and Buze (Buzz clone)

[ly29]

@zeffii
It is very fun to rethink what is possible in the sverchok system, of course it is a bit hackish right now but I am very open for suggestions.

[zeffii]

Cool. Open Floodgates.

• right-click selection, make new sub_node_group
• sub group will have a left to right convention
  • when in iteration mode left hand side inputs must match right-hand side outputs
  • else they can be asymmetric and the sub_node_group is more like an abstraction container.
• sub_node_group gets its own layout, maybe with simple bgl identification to show that it is a sub group (colour border of window)

The Iteration node and sub_node_group node, one would inherit from the other.

[zeffii]

holding shift over an input socket in the sub_node_group, and click-dragging away would create a communicator,

I don't know if the communicator or 'relay' node whatever it's called .. should only ever be one node which grows depending on the need for new inputs.. or if it's good to have multiple separate input nodes.

[ly29]

The general idea plan right now for this branch:

• Convert some more nodes with different requirements and look for pattern to simplify implementation
• Write documentation
• Make update system more clever and only update nodes that lead to an output
• Only reprocess nodes that depend on an input *
• Cleanup some code for the various nodes being converted
• Play with node groups and iteration techniques.

@zeffii
Good ideas all of them

rightclick selection, make new sub_node_group

Of course that is the proper way, and actually only sane way, of creating node groups, building them by hand has serious limitations.

sub_node_group gets it's own layout

My plan is to let them live in frame while being edited, but let them live permanently in the hidden Sverchok tree type shown above, so we can have node groups without them being visible but still use all current nodes (with conversion as shown above).

The group input and output grows and shrink according to need, but only linked socket are propagated to the node group/iteration node. And only when it is "linked", might change that terminology and how it works a bit.

[ly29]

A simple Fibonacci setup.

[ly29]

Group in a group in used for iteration.
This is actually working without any extra work.

[zeffii]

the only correct response is an animated gif: http://i.imgur.com/1KsMK.gif

[ly29]

Well enough play with this for now, I consider the concept proven.

[ly29]

A do while loop or more advanced scenarios also fit into this scenarios. There is not theoretical limit for node groups depth or recursion, practical limits of course exist.
I am starting to think this approach is good enough for now, it gives us several practical benefits while being an quite easy conversion. Of course managing state etc is a little bit more complicated but for many nodes this would be good enough.

[ly29]

Fibonacci seq illustration. Of course this is like hello world, but wasn't possible before with the nodes so it is worth to show.

[enzyme69]

Svelegant, I should say.

[ly29]

I think this basic approach has some things going for it.

• can get more clever about processing,
• can turn off updates while editing
• having the nodes still existing allow bpy properties to be exposed in other blender windows
• doing simple node groups this way allow flow control tools, if, while, for and node groups works as functions, allowing for many simple programming techniques to be exposed, including recursion etc
• it is quite simple to convert the nodes to work in this slightly different way

[ly29]

So the question is, should I work on this or does it seem useless?

Doing this make sverchok turing complete, recursion etc is possible. If we decide to do this we should.

1. Work out the interface for the nodes for managing state.
2. Convert all the nodes.
3. Fix the update system a bit.

So yes or no? If yes I think we can convert everything quite quickly and then explore further techniques, even if in the long term it isn't enough it will make the next step easier. So I say yes.

[zeffii]

Can it be manipulated to use the builtin grouping feature?

[zeffii]

until then I guess sticking those parts in a Frame is an OK visual solution

[ly29]

As far I understand that part of pynodes isn't ready yet, if we build ourselves we make an incentive for the blender developers to do that I think, it should also be quite easy to convert our node groups to proper node groups when the time comes. There are some details to work out of course but I think it is quite doable.

[ly29]

Also it is a good time implement reroute support etc that involves changing a lot of nodes.

[zeffii]

do any nodes get excluded from this?

[zeffii]

it wouldn't surprise me if this produces undefined behaviour in SN.. But you might have to do a bunch of nodes so we can see through .diff what to change. 100+ nodes is quite a lot, but if this (and I think it does) has multiple benefits.. it's worth it - at least give it a good shot.

[ly29]

the 3dview props node should work without any changes.

Types of nodes when it comes to update.

• Nodes that are always ready, like circle, int, float range - basically update can be renamed to process
• Nodes that depend on some input to be ready: mesh separate, soldify, view draw etc.
• Multi input nodes or socket changing nodes, list nodes, wifi nodes
• Some nodes doesn't really need process since they are pure syntactic sugar, like wifi nodes, they can be abstracted away by the update system

I'm all for doing a bunch for nodes and finding out what needs to change, the most important part is that the node tree and blender doesn't control processing but only the ui part.

[nortikin]

should we create own classes for this processing?
if you explain what to do -i ready to rewrite nodes.

[ly29]

There are still some styles issues to be worked out but I will explain, soon.
I don't think we need our own classes, we can get many benefits with a small restructuring of our existing code. Writing our own classes is perhaps a good idea for later but is a much bigger task. It also a good time to get rid of some cargo cult code.

[ly29]

@zeffii
re script node, I think it fits quite well into this since it is operator driven or simply processing. of course some scripts that break the sandbox and play with blender or other nodes might break...

[zeffii]

I'm quite happy to modify my nodes, and if that goes well start picking off the others..some of which i've never looked at the source for.

[ly29]

So I will write up some notes and convert a few more nodes, then we can do the conversion quite easily I think.

[nortikin]

in separate branch

[AgustinJB]

If you guys need help I also can rewrite nodes. This topic sounds like necessary to me.

[ly29]

Some updates.
So, started the a new branch since a lot has happened in master since the old one, it is called to-process not so much has happened yet, mostly some basic stuff that changed, using mix in class for the nodes, finally I understand how to do inheritance properly with blender classes (see http://blenderthings.blogspot.se/2014/07/inheritance-and-mixin-classes-vs.html ) which resolve the problem of how to set state nicely. Next step is to bring over the changes from the old the ui-separate branch.
Some notes:

• updateNode is replaced with update_node, which is now a class a member method in the SverchCustomTreeNode class and no longer needs to be imported from data_structure. Also not that this is ONLY for user exposed bpy.props, and those should never ever be changed inside the new process function
• creating a new socket, there so need for doubled name argument. inputs.new("StringsSocket", "data")
• a node is responsible for managing its state, it is set using self.set_state(ACTIVE), many nodes will never have to do this since they are always active.

Some further changes are coming, I'm doing a full write up of how to write a well behaved node.

Also propose to call this version 0.5

[nortikin]

yes, but what we have to change? update function?

@classmethod
def updateNode(cls, self):
    update_node

will do what you write to do with nodes.

[ly29]

Very confused by me. It isn't a class method, but is placed in the SverchCustomTreeNode class as a method, renamed to keep better naming and keep track of changes needed. Since it a method that is somewhat over used today.
Otherwise it works just like updateNode, just placed in a more logical place.

[nortikin]

ok, than we should delete all updateNode in nodes?

[ly29]

@nortikin
updateNode is replaced by update_node which doesn't need to be imported. It could just be replaced with global sed but it also needs review, it should only be used on user exposed properties which shouldn't be changed inside of the process function.

[nortikin]

now watching https://docs.python.org/3.3/library/threading.html#module-threading
It is what we need, maybe. when all separation will be, this librery can ease working process to blender.