Flow-control concept

[honix]

Im considering to develop flow-control mechanics for pyno.

Right now pyno do the update of all nodes 60 times per second. It is good for real-time systems, but bad for other classes of programs, for example calculator. It is also a waste of processor time.

This aproach also seen in vvvv visual language. It also reminds me functional nature of algorithms. In some cases it seems more natural to just define operators, put some data and dynamicly (!) see the process. Dynamic nature is one of main goals of pyno.

Another approach is flow-control - the way to execute nodes in order you need. It will allow to do condition branches and loops (can be main loop done this way?).

Examples is UnrealEngine's blueprints, cables.gl and VL from vvvv guys.

They differently solve this concept. I mostly like cables.gl approach, it bulds around signal concept as i understand it. There is signal and data wires. Signal wire activates execution of node. The semantics of signal wire may vary from node to node.

My goal is to do easy to understand but powerfull flow-control concept, so pyno will be used for broader classes of programming.

Ideas are welcome.

[drtrigon]

I did a first very hacky approach in "flow_control_delay.pn". This might be a way to go BUT also produces non-trivial side effects: notice 1 field gets updated immediately, the other 2 NOT even though 1 of it should. Think this is related to the nodes list order.

[bilderbuchi]

I think the update frequency should be factored out/customizable, so that people can choose when to update.
For example (and this is without having used pyno yet), I would imagine using this to have some regular python code controlling e.g. some experiment setup (with serial connection, etc.), but giving the user the option to have parts of the program as (stateful) pyno patches - so e.g. some automated process runs a machine, and every once in a while advances the pyno section of the program (which could be some data processing) one tick, passing it some inputs and collecting its outputs.
The bigger aim here being to make something similar to LabView, just with Python.
Flow control as I understand it (maybe badly) should be automatic/user invisible, from inputs to outputs. I don't know how e.g. vvvv handles this internally when there are several inputs.

One thing (possibly only a tangent) I think would be very useful, if pyno could automatically create nodes by wrapping existing functions from a given module (doing some signature analysis on the way, possibly relying on typing information).
This way, it would be easy to build up a node library, e.g. from callables like python builtins, numpy functions, etc., without having to write them all yourself. Imagine e.g. being able to just use numpy.sin. Probably some blacklist or somesuch will be necessary to reject functions that won't work, as the dynamic nature of Python (especially on the return type) will make it quite difficult to do this reliably.
What this would open also is that eval will probably have to be used less often (which is good), and possibly the pyno patches could essentially be compiled down to plain python programs with chained function calls by some AST magic (but that's waay out there now).

[drtrigon]

Am 6. November 2018 16:42:55 MEZ schrieb Christoph Buchner <notifications@github.com>:

The bigger aim here being to make something similar to [LabView](http://www.ni.com/labview), just with Python. Flow control as I understand it (maybe badly) should be automatic/user invisible, from inputs to outputs. I don't know how e.g. vvvv handles this internally when there are several inputs.

Flow control in LabView is NOT invisible. The opposite is the case it is actually THE reason for most strange constructs like reference variables passed in and out e.g. file read write blocks or using sequences by just passing variables in and out not doing anything to them, etc. Have a look a the delay block I mentioned before - it uses one of these LabView techniques... ;)

One thing (possibly only a tangent) I think would be very useful, if pyno could automatically create nodes by wrapping existing functions from a given module (doing some signature analysis on the way, possibly relying on typing information). This way, it would be easy to build up a node library, e.g. from python builtins, numpy functions, etc., without having to write them all yourself. Imagine e.g. being able to just use `numpy.sin`. Probably some blacklist or somesuch will be necessary to reject functions that won't work, as the dynamic nature of Python (especially on the return type) will make it quite difficult to do this reliably.

I was thinking the same, but changed my mind for the moment as this is quite some work for doing and maintaining and in the end the one block you need will for sure still be missing (murphys law). Everybody can build and share subs or collections of subs already e.g. on github (like Icestudio).

[bilderbuchi]

Ok, so maybe I don't exactly understand what is meant with flow control. I'll take a look at the delay block.

Being able to wrap existing python callables (or objects in case of fields) does not mean you can't write your own code where needed! it just means you won't have to implement half the python stdlib or numpy functions before you even get started, and code reuse would be much better (e.g. of other functions you have written before). Also, it would enable you to just call() the callable instead of having to go through eval and friends. But I'll grant you, doing a reliable analysis to determine the set of arguments and returns could be hairy.

[bilderbuchi]

Also, speaking from personal experience, node-based languages are full of strange constructs if you come from text-based programming. I'm happy that I groked vvvv after a while so that I used it for several years, it was overall an awesome experience!

[drtrigon]

I don't used vvvv - I used LabView quite lot some years back (v4, v6 to 8, 9 may be up to 2013). Comming from LabView I would say using Pyno on the level std funcs would need e.g. better wire or connection draw facilities (like on schematic or PCB CAD tools as mentioned in #3 (comment)).

For now I am quite comfortable with the higher level character using multiple std funcs in one single node, makes the design a bit more abstract and may be better understandable. Of course reuse is bit more of a problem. Finally Pyno should have all you propose I agree on that actually! This is of course just my opinion.

As said thanks to the new subs you can already start wrap your favourite std func in "blocks" and share them e.g. via github. There is nothing stopping you from doing that. I think @honix would even include them into this repo if you issue a pull request - but that is just a guess. ;)

[drtrigon]

About dataflow I can say that LabView has lot of beautiful tools and tricks but most of them include having to introduce new elements and LV ended up having so many that basically everything can be done in several different ways. I don't think this is where Pyno should end up - we should keep everything as simple as possible and only make it as complex as needed. That is why I started experimenting with the delay example.

[honix]

So, im doing some work in separate branch.

Subjects of change:

• simplify node's new_code method, there is no parsing any more, just python reflection
• remove piece of processor's need_update code, when function get its environment, this now happens in node's new_code

Any node now will have call binding, call will be a function.

So you can use more than one function and bind main function to call, node will represent this main function:

def helper(a, b):
  return a + b

def add(a=0, b=0):
  result = helper(a, b)
  return result

call = add

It can be lambda expression:

call = lambda x: x

Or make fast bindings to libraries (only pure functions for now, maybe methods later):

import calendar
call = calendar.isleap

Bad news is that i drop function tuple return separation, as really function returns only one value, and separation of tuples is only syntactic sugar. Almost all examples used multiple return values is broken now.

[honix]

We can try to reflect return type notation to restore multiple returns feature.

def process(response: int) -> Tuple[int, str]:

[drtrigon]

Can you elaborate a bit what's the reasoning behind these changes? What about supporting the old "single function" node as well - just having both (in the same element)?

[honix]

This way is little more explicit to user. We want to take user function, but dont know what to take. Let user make this step.

Also it more adaptive to foreign functions. User will not make wrapper, but just bind needed function to call.

And techinically this is most "okay-ish" way to do pythonic reflection. Just take what we waiting from user, read signature, and make node representation.

upd:

It is possible to auto-bind function if there is just one user function. But it is little lack of consistency i think.

[honix]

Progress report. I done with type exctraction 8e78777. Here is screens.

Multiple return values using type annotation:

Usual tuple outputs as tuple itself:

It is really one way to do it properly and it is. How do you think guys?

[bilderbuchi]

Looks great! Did not think it would be so "easy"/fast to do what I suggested. :-O

[drtrigon]

That would allow whole python script in one single node. Interesting. I'm not familiar with reflection (even though I learn from wiki I was already using it before ;) or return type notation of python.

What about the scope? Is the import or are the helper functions available in other nodes too?

I the old mode could be supported along the new it would keep compatibility with old code... (however honestly to say there are not too many progs out there to modify). Or use a default wapper to make the old code compatible with the new node?

[honix]

@drtrigon scope is node closed. I think it is good for bunch of reasons. First of all node's code becomes self-contained and everything-with-me.

Yes, i considering to modify old patches. There is some breaking-rules changes. Maybe it is not all of them.

[drtrigon]

I think node closed scope makes sense.

About compatibility with old nodes and pyno-files, basically if you get code w/o call defined like:

def newNode(a=0, b=0):
result = a + b
return result

just add call = newNode to it should do the trick for most of the cases. Can that be done automatically in new_code?

The other thing I am reasoning about is whether typing should be imported automatically (implicitly) in order to make the syntax for type exctraction/annotation support simpler and shorter.

[drtrigon]

LabView Dataflow Programming Basics: http://www.ni.com/getting-started/labview-basics/dataflow

Just found that document - this is the language I got used to, may be helpful here.

[drtrigon]

Helpful would be naming of node output values, may be from comment?

Importing via fields is obsolete now, correct?

I implemented the flow control example in new style (pull request will follow) and noticed that it makes more sense now, but the fps are by about a factor 10 lower than they should be given the delay around 1...3 fps (using 0.1s should result in about 10fps, right?).

[honix]

Yes, default return names are not helpfull now. I will look for some signature's arguments naming.

Nodes is scope closed, nothing to leak in leak out (except G['item'] for globals). Field can import something if it used in this field (inter-field will work too i belive).

Fps may show wrong. Need to check.

[honix]

Thanks for LabView link!

[honix]

Return naming is possible now! d53ec7c

[drtrigon]

https://www.researchgate.net/publication/221914355_Graphical_Programming_Techniques_for_Effective_Fast_and_Responsive_Execut

[drtrigon]

I thought a bit about flow-control again. Consider following example:

• sync: Node A and B are naturally "synchronized" by node C - both paths A and B have to be executed before node C will.

• sequence (LabView): This is one of the most important flow concept, e.g. for hardware access, etc. Look at the example and consider A, C, D only. A has to be executed before C and C has to be executed before D. This is very important for e.g. file access, usb serial port access, etc where order matters. In all these cases you have to open, work and close the device in exact that order. In most cases there is naturally an object that has to be passed from one node to the next, e.g. the file or serial class object. There are cases where this does not hold, consider a time.sleep to wait for the hardware (e.g. between write and read operations). In these cases the user can easily add a dummy variable (=None e.g.) that is not used but passed from node to node (such variables could be called "device" or "resource").

• loop (VisTrails): Looping (for, while, may be map also) is a bit harder to implement. LabView for example does this by introducing new elements that are frames where you can put the elements to loop into also adding a lot of dedicated special in-/output elements like shift-registers etc. I do not think this is the way for pyno as it involves a lot of changes and new concepts (also for the user to learn). I would like to propose to concept VisTrails uses. They just added a self output to all nodes/subs. By connecting this self output to another node it is possible to manipulate and execute the node as many times as wanted. This is clever and elegant IMHO. The drawback is they need to fiddle around with state and condition variables that are passed in the background hidden and silently (using the self object). For pyno this would mean we have to add the self output always or optionally (I vote for always for reasons of simplicity). I think we need to find a better way to handle the state variable as VisTrails does in order to make it more intuitive for users. May be require to wire them graphically? Doing this in the background is bad, confusing and needs wrapper objects to make it convenient.
  Btw.: Adding this self object might also allow for recursion constructs.
  

[honix]

@drtrigon yes, self pin is the way i want to do it.

Node will have self pin, which is pointer to the node. Then we can query node to execution by for example then(nodeC). When current node execution is done, Pyno engine will execute this quered nodes, maybe in parallel (nodes have only private state). And so on.

Need to check it for theoretical errors.

[drtrigon]

Does executing in parallel mean mutli-threaded? As this could end up in hundreds of threads running in parallel when comming to subs containing other subs etc. LabView e.g. does not work mult-threaded but processes elements in sequence - just the user never really knows the order of execution despite the points noted before. But the order of A or B and also of D or E is not know. (Usually in LV the order is always the same, so as soon as you executed the code once you know the order, but on any change it may change as well.)

[bilderbuchi]

In the end, i think your patch/collection will have to form a Directed Acyclic Graph. You should be able to use some graph theory package like networkx to easily obtain an ordered representation of the nodes in your patch. This you can then process in turn (and in parallel where proper, e.g. with concurrent futures and friends). If you get cycles, you have to break those up with a delay-by-one-step node (cf. vvvv).
This is straightforward for pure function nodes, will probably need a bit of thought for stateful objects.

[drtrigon]

I have to correct myself, LV also works in parallel, see "clumps":

NI LabVIEW Compiler: Under the Hood: http://www.ni.com/tutorial/11472/en/#toc3

[iperov]

Hi.

For the last year I've been exploring visual-programming for real practical applications.

My initial idea is to give DeepFaceLive (https://github.com/iperov/DeepFaceLive) users the ability to modify the logic of program nodes interactively, without resorting to source code.

The conclusions I came to are:

1. visual programming of simple operations is a failure. 2-3 lines of text in python, in nodes it will be like an entire screen of spaghetti nodes. It's pretty, but not practical.

2. visual-programming limits the speed of writing programs. By reading text, we load a larger volume of algorithms into our imagination. When reading nodes, we first take a long time to mouse and scroll, memorize the diagram, then translate it into our internal representation of the algorithm in our imagination. As a result, we produce a lot of redundant actions in the brain compared to reading text code.

3. Nodes should contain large programs. The user only guides the logic of the data.

Therefore, the nodes should be interactively programmable in a lightweight language like python.
In this case the system can provide 3 types of programming:

• No-Code. The user downloads ready-made apps and uses them.

• Low-Code. The user makes programs using ready-made nodes, or modifies downloaded apps, adding functionality to suit his needs, such as outputting an image to a stream or saving a sequence of images in a folder.

• Full-Code. Same goals as above, except the user programs nodes interactively using hot-reloaded text inside the same studio.

4. The more abstract the node system is, the smaller the range of tasks it can accomplish. The greater the range of tasks a node system can accomplish, the more it is like directly writing lines of code, but with Basic(lang) flowcharts, which again are difficult to read compared to text code.

5. A vertical node system can accommodate more nodes and requires mouseover on each node to read the I/O.
   Horizontal is easier to read but requires frequent horizontal scrolling.

6. The advantage of node system over text code is that we can see the result of any function execution interactively. It is like Debug mode of code execution, only the result of any line of code is available at any moment by simply pointing the mouse.

A node system that has a fixed pipeline and has fixed views for users, such as buttons, text and numeric fields, is not difficult to implement.
The pain begins when the code produces dynamic interactive views waiting for input from the user.
For example, selecting the type of Neural Network model in a ComboBox, and depending on that, different settings appear below. Or some settings that spawn other settings.
In this case ViewController should be built in a separate window by the program/nodes itself, increasing the size of the program, and mixing Model code with ViewController code.

Conclusion.

Is it possible to design such a general-purpose node system that would be both intuitive for the user
(vvvv is not intuitive even for programmer),
able to implement MVC/MVVM code of any complexity, and async/greenlets/multi-thread friendly?

Or is it just fundamentally impossible?

[honix]

@iperov welcome!

My thought is that visual programming is good for simple tasks, realtime interactive systems.

Like it's good for searching right combination of nodes to get the effect you need. But it almost always is about combination a to b, a to c to b, and because of that it limited to such simple tasks.

It is good for realtime, because almost any state of your nodes on canvas is syntactically correct, so you can iterate freely. Sometimes you don't understand the logic, but love the effect.

To contrast with textual programming, the last has more ways to connect things. It is harder to write in textual, but expressiveness is better. It's kind of combination too, but the size of possibilities is huge.

Text is always abstraction, and you must learn a language to recognize and use its forms and relations, while visual things is limited by links and nodes, that way one can express only primitive constructs.

As you mention, when we try to make visual language low level-ish, it become spaggeti pretty fast. Text is compact and it usually hides complex semantics, when visual thing usually don't work with semantics well because of like more physical approach of arranging words.

This is why I'm no longer a visual evangelist. I still love the systems like vvvv. When I just started to program, this thing was like mind blowing, because of fast iteration cycle and great premade examples anyone can modify. But text has a cypher in it, and it is it's great power, that allows one to insert in it almost any semantics you need.