New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Visual and UI components #66

Closed
bergie opened this Issue Apr 18, 2013 · 26 comments

Comments

Projects
None yet
4 participants
@bergie
Copy link
Member

bergie commented Apr 18, 2013

For that NoFlo runs also in the browser (see #63), it is time to start preparing for visual and interaction components.

The way UI designers use Quartz Composer for prototyping is a good benchmark here.

Essentially we'll need components that manage a DOM element (or a higher-level Component/jQuery UI/Bootstrap widget), and also components for listening to DOM events (for example, touch).

This also concerns #1 in the sense that the NoFlo UI needs to have a "preview mode" where you can run these visual flows.

There is a set of video tutorials on Quartz Composer that are quite illustrative of this use case.

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 18, 2013

QC uses a push/pull model.

A QC patch execute method is given the current patch time and a GL context. It can then read from its in ports and write to its out ports.

A QCPatch (or a subclass) may also have its own subgraph. It can call executeChildren.

A Macro Patch in QC is simply a regular QCPatch with children. It executes its children once when it itself is executed.

An iterator is a QCPatch subclass. It accepts an iterations index from an in post and then executes its children that many times in its execution method.

An entire QC composition is the same as a single Macro Patch.

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 18, 2013

Yep, the concept of Macro Patches is very similar to subgraphs in NoFlo, and a regular patch would be the same as a NoFlo component. Publishing ports on a macro patch is also similar to NoFlo's exported ports (see #59)

There are some subtle differences. For instance, I don't see why inside a subgraph the published outports would need to be exposed via a special "Feedback component" when simply a Splitter and looping connection would do.

Similar thing with the special way of connecting visual components and interaction components together, should in my view be a regular port connection. As should be the handling of z-indexes (layering).

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 18, 2013

The z-index layer number on blue parches in QC is really just the execution order. When a QCPatch executes its subpatches, it does so in execution order as defined by the layer number on each blue patch.

A blue patch then pulls the values of its in ports. Nothing executes unless it's a blue patch or connected to a blue patch.

Blue patches cannot have out ports. They are purely consumers only.

Purple patches often execute on idle, pulling in data from hardware like the mouse or the network. When one of its output ports is updated, the runtime marks dirty every node in the graph that is connected to it so that they dump their cache and execute again when their own out ports are next needed.

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 18, 2013

@subtleGradient thanks for the clarification! I think that part will work a bit differently in NoFlo, as we will be manipulating DOM instead of working with "display PDF". So visual components will probably have an outport where their containing DOM element is sent so that you can then place them in the chosen places on your document or documentFragment.

The benefit here is that you could quite easily embed NoFlo-driven parts to almost any regular web page. Or if you're prototyping a UI, just write everything to the body :-)

Interaction components (purple patches) executing only on idle is quite close to how JavaScript's event loop works.

As for things not executing unless connected, that is already covered by normal FBP concepts like most components requiring some input before they actually start sending stuff out.

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 18, 2013

A push only model usually causes havok and poor performance in the DOM.

The trick is to figure out a way to separate all the reading and writing to and from the DOM in at least two phases no more frequently than in a requestAnimationFrame loop.

See how MontageJS does it. Their bindings system is almost flow based programming, but not quite.

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 18, 2013

Will do! I want to do some experimentation with visual components next week.

And yeah, for performance requestAnimationFrame and WebWorkers will both be quite relevant for NoFlo

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 18, 2013

It may be as simple as making a node 'class' that has a frame number input and buffers all its other inputs until the frame number changes. By default plug the frame number up to a single requestAnimationFrame loop node with a frame number out port.

That way all the event driven push stuff doesn't cause any havok in the DOM and everything happens at the right time.

Also, you could choose to manage time separately for one part of the graph if you want to do something custom.

This doesn't solve the problem of something like canvas drawing instruction execution order though.

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 18, 2013

@forresto shared a link to Toolkitchen which could provide some interesting ideas for the visual NoFlo components

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 18, 2013

@subtleGradient this is something we could probably do quite easily with the throttling support that already exists in https://github.com/bergie/noflo/blob/master/src/lib/AsyncComponent.coffee

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 18, 2013

There are a million frameworks for building UI components. Ideally NoFlo would work with them all

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 18, 2013

@subtleGradient it will, just like you can wrap any Node.js modules into NoFlo components. But we need to start with something that gives us reasonable widgets to demo with :-)

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 18, 2013

Whew, good. I'm really excited!

bergie added a commit that referenced this issue Apr 19, 2013

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 19, 2013

Baby steps. GetElement is our first visual component. Essentially runs document/element.querySelector and sends out the resulting element. Next up: mouse and touch interaction listeners.

bergie added a commit that referenced this issue Apr 23, 2013

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 23, 2013

@subtleGradient @forresto @d4tocchini highly initial demo of some touch interaction with NoFlo on browser: http://noflojs.org/demo/touch.html

So far tested only on Chrome on an Android device and Safari on iPad. Note that there is currently no mouse listening active, so you can move the image only with touch-capable devices

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 24, 2013

👍

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Apr 24, 2013

Alright, now there is also a demo of springiness added to http://noflojs.org/demo/touch.html

Makes the graph slightly more complicated, but a lot of that can be removed by moving stuff to subgraphs.

cc / @subtleGradient @d4tocchini

bergie added a commit that referenced this issue Apr 25, 2013

bergie added a commit that referenced this issue Apr 25, 2013

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented Apr 26, 2013

👍 * 💯

@d4tocchini

This comment has been minimized.

Copy link

d4tocchini commented Apr 29, 2013

You're the man @bergie!

@bergie

This comment has been minimized.

Copy link
Member

bergie commented May 3, 2013

I was thinking about QC components like Sprite and Billboard a bit. In NoFlo the best way to see these would probably be a set of "templated HTML" components that provide their own input and output ports, and which you could easily place into the DOM.

For example, a Button component could have a "label" inport, and outports providing "element" and "click". With the element outport you can then place the newly-created button element anywhere in the DOM (other option would be to provide a container via inport), and the click would send an event whenever the button is pushed.

To construct visual components like this, we first need some barebones DOM handling components, like something for creating new elements and placing content to them, and the positioning/interaction components we already have. With those, each widget could just be a packaged subgraph :-)

@subtleGradient

This comment has been minimized.

Copy link

subtleGradient commented May 5, 2013

There are lots and lots of ways to do it. The essential thing is that at the design level, the implementation details are cleanly abstracted away and encapsulated in the patches. You may want to have a DOM+CSS renderer, but you may want to swap it out for Canvas or WebGL or something else. You need to be able to make the decision about renderer completely separately from the composition itself.

As long as all the implementations of "Sprite" use the same interface, you could easily swap implementations by sed 's/CSSDOMSprite/WebGLSprite/g'

@bergie

This comment has been minimized.

Copy link
Member

bergie commented May 6, 2013

@subtleGradient we certainly need to start with quite low-level components (which would then be specific to DOM, canvas, or whatever), but those can be used as the building blocks for more generic components/subgraphs.

For Sprite, can you provide some use cases? That would help in design, as my experience with QC is quite limited.

@d4tocchini

This comment has been minimized.

Copy link

d4tocchini commented May 13, 2013

on a side note, @bergie, here's that remix of your springy drag demo that works with mouse and touch. I just hacked @desandro's Draggabilly into a NoFlo component

@forresto

This comment has been minimized.

Copy link
Member

forresto commented May 13, 2013

Smooth.

@d4tocchini

This comment has been minimized.

Copy link

d4tocchini commented May 13, 2013

Front-end FBP is a pandora's box, there are too many ways of doing things! From the looks of the hacky work arounds in those Facebook Home Quartz Composer Tutorials, some best practices are badly needed...

Should we have fat components with many inports/outports, or thin ones that pipe sprites around and add functionality ie "decorator pattern"? @subtleGradient in #67 you seem to be implying fat, generic ones.

Shouldn't Sprite inports also be exposed as exports? I found it irritating in QC that Sprites had no outports. It would be nice to be able to pipe size & position in and out of Sprites to capture more dynamic layout logic without subgraphing or additional components. Does this cross the encapsulation line? Or, maybe it's not in place because circular dependencies arise? In which case, we can bust out some Cassowary!

@bergie

This comment has been minimized.

Copy link
Member

bergie commented May 21, 2013

The DOM handling components have been moved from core to https://github.com/bergie/noflo-dom

@bergie

This comment has been minimized.

Copy link
Member

bergie commented Sep 25, 2013

Individual component libraries and requests for new components are better handled in the issue trackers of their respective libraries

@bergie bergie closed this Sep 25, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment