Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Glimmer Engine #10501

Merged
merged 321 commits into from May 5, 2015
Merged

Implement Glimmer Engine #10501

merged 321 commits into from May 5, 2015

Conversation

@wycats
Copy link
Member

wycats commented Feb 21, 2015

Glimmer is the next-generation rendering engine for Ember.js. Built on top of HTMLBars, streams, and an entirely rewritten view layer, Glimmer offers the best performance of any of the major JavaScript frameworks while remaining backwards compatible with Ember 1.x apps.


The Virtual DOM showed the world that render-time diffing delivers awesome performance by minimizing DOM updates and keeping unchanged elements stable. But like any abstraction, it comes with a cost. In this case, constructing the virtual DOM to compare requires at least some subset of components do a full re-render, and building virtual DOM nodes for static areas of your markup that will never change for every change requires many more allocations and increases GC pressure.

Glimmer analyzes templates at compile time, relying on the fact that Handlebars' declarative syntax clearly differentiates between static areas, which make up the majority of a template, and the dynamic areas that can change. Instead of invoking a render() method and diffing the result, Glimmer only walks the (much smaller) existing dynamic tree.


Handlebars' declarative syntax means that from the perspective of an app, the template is being "re-rendered every time", but we can take advantage of static knowledge to reduce the work that we need to do. User code, like helpers and computed properties, are executed during the walk, but only to get (primitive) values which can be === compared, not to build up a new tree. In other words, the programming model is equivalent to "render every time", but we take advantage of the declarative nature of Ember's APIs to reduce work.

Internally, instead of creating a virtual DOM, Glimmer builds a tree of streams, each stream pointing to a node in the DOM. On initial render, we track the last value (always a primitive) that we inserted into the DOM. The streams are not exposed to application code; Glimmer executes user code when necessary and pushes the new value into the stream.

When re-rendering, we walk the tree and flush the streams. If the primitive value produced by the stream has not changed, we do nothing. If it has changed, we write the change to the DOM. Like Virtual DOM approaches, this is a "write-only" algorithm.

One of the benefits of this approach is that we can naturally fuse the Ember/Polymer model of observing individual properties with the React model of explicitly re-rendering areas of the template. No matter what happens, the process of revalidation walks the tree, looking for dirty nodes, and revalidates any dirty nodes it finds.

The only difference is how much of dynamic tree is marked dirty before revalidation begins. Also, because observation can only dirty dynamic nodes (and schedule a top-down revalidation), multiple observers and re-renders that occur during a single run-loop still only produce a single batch of DOM updates.

In essence, the biggest difference between Glimmer and traditional virtual DOM approaches is that we diff values, not DOM. This approach not only lets us avoid the algorithmic complexity of tree diffing (instead, we just === check two values), it also lets us avoid doing many render-time allocations.

Slides from the talk at EmberConf

Work List

(this looks like more things than it actually is)

  • merge subscription change into our branch
  • linkRenderNode can mark stability
  • Ember should return true from linkRenderNode
  • make sure all child streams correctly disconnect from parents
  • add stream pruning logic that correctly prunes ancestors
  • add scope teardown hook to HTMLBars (recursive upon clear or remove)
    • this is also responsible for view layer teardown
  • Propagate correct visitor through descendent hooks (so dirty checks can be avoided for a subtree)
  • Do not dirty check render nodes that were just created (fixes regression)
  • Add "classify" hook to decide which hook content et al should delegate to
  • Make sure that #view is idempotent
  • Components and views should create a new scope frame
    • Implement custom scope in Ember (so that ambient view, controller, etc. can be stored)
    • Eliminate *component* hacks
  • Lifecycle hooks on the renderer
  • Move code duplicated between #view, #component and Ember.View into the Renderer
  • Support reflecting attributes onto angle-bracket components
  • Pass mutable bindings as "mutator" objects
  • Support foo={{action "bar"}}
  • Legacy support: Implement unknownProperty/setUnknownProperty to work with mutator objects
    • Deprecate setting non-mut attributes, and support (mut foo) sexprs in legacy Handlebars syntax
  • Investigate executing teardown callbacks at a "quiescent" state ("vent plasma")
    • Make sure to consider the rpflo stress test situation
  • Implement a zip and map abstraction for streams to clean up ad hoc stream creation
  • Add tests for memory leaks
  • Make sure that custom render functions that mutate the buffer continue to work
  • options.template should not exist if there is no template
  • feature flag all the things
  • Handlebars helper compat
  • Template function that returns a string (legacy)
  • Context shifting #each
  • collection helper
  • {{input}}
    • TextField
    • Checkbox
    • TextArea
  • Ember.Select
  • unbound
    • non-block form
    • block form (partially done, but not completely)
  • #view and view helper
  • #view and view helper with back-compat edge cases
  • #with helper
  • with helper with back-compat edge-cases
  • yield without a block
  • {{render}}
  • ContainerView
  • Top-level view APIs
    • createElement
    • appendTo
    • replaceIn
    • destroyElement
  • isVisible
@wycats
Copy link
Member Author

wycats commented Feb 22, 2015

React Hook Ember Hook Server? Initial Render? Rerender? Purpose
componentWillMount init Yes Yes No Set initial component state without triggering re-render
componentDidMount didInsertElement No Yes No Provides opportunity for manual DOM manipulation
componentWillReceiveProps willReceiveAttrs No No Yes React to changes in component attributes, so that setState can be invoked before render
shouldComponentUpdate Maybe N/A No No Yes Gives a component an opportunity to reject downstream revalidation
componentWillUpdate willUpdate No No Yes Invoked before a template is re-rendered to give the component an opportunity to inspect the DOM before updates have been applied (example)
componentDidUpdate didUpdate No No Yes Invoked after a template is re-rendered to give the component an opportunity to update the DOM (example)
componentWillUnmount willDestroyElement No No Yes The inverse of componentDidMount/didInsertElement; clean up anything set up in that hook
N/A willRender No Yes Yes In Ember, executed both after init and after willUpdate*
N/A didRender No Yes Yes In Ember, executed both after didInsertElement and didUpdate*

* These hooks can be used in cases where the setup for initial render and subsequent re-renders is idempotent (e.g. $().addClass) instead of duplicating the logic in both places. In most cases, it is better to try to make these hooks idempotent, in keeping with the spirit of "re-render from scratch every time".

@wycats
Copy link
Member Author

wycats commented Feb 22, 2015

The above comment is an attempt to classify the hooks in React and Ember so that we can make sure that the new Ember components implement at least the hooks that React folks have identified as being important in the programming model.

@rwjblue
Copy link
Member

rwjblue commented Feb 22, 2015

All new API's will also need to be feature flagged.

@wycats
Copy link
Member Author

wycats commented Feb 22, 2015

@rwjblue yessir

@Linicks
Copy link

Linicks commented Feb 22, 2015

The init hook name is a little unclear compared to componentWillMount. Could it be named initElelement, or something similar ?

@stefanpenner
Copy link
Member

stefanpenner commented Feb 22, 2015

The init hook name is a little unclear compared to componentWillMount. Could it be named initElelement, or something similar ?

I would prefer leaving it as init, as it clearly describes what it handles, initialization of the current entity.

That entity just happens to be a component. initFoo as a pattern feels like an exercise in over specification. Especially when you consider the rest of the collaborators all having a unique initFoo method, initModel initRoute etc.

Also worth noting, as we continue to align ourselves with >= ES2015 init will likely become constructor

@stefanpenner
Copy link
Member

stefanpenner commented Feb 22, 2015

@wycats @tomdale good job guys :)

@alexspeller
Copy link
Contributor

alexspeller commented Feb 22, 2015

😍

@alexdiliberto
Copy link
Contributor

alexdiliberto commented Feb 22, 2015

This is amazing 👍 👍 👍

@Linicks
Copy link

Linicks commented Feb 22, 2015

The problem with just using init is that init is used in so many places in the tech world. In an Ember only world it would be clear and concise, but in an ever increasing complex tech stack it's nice to have that extra bit of clarity. I think that's why FB chose an even more verbose naming scheme than my proposal. I have had the chance/misfortune of maintaining allot of legacy software, and appreciate when the code is self describeing as practical. This is really a preference in either direction, but after being in this game for so many years I'm getting a little opionated :) Either way this is exciting stuff, and look forward to seeing this come together in Ember.

@ebryn
Copy link
Member

ebryn commented Feb 23, 2015

@Linicks if you look at their ES6 syntax, you'll see they're utilizing the constructor to initialize this.state

@fivetanley
Copy link
Member

fivetanley commented Feb 23, 2015

What is a "ShadowRoot"?

@gonvaled
Copy link

gonvaled commented Feb 23, 2015

This being a "DOM write-only" implementation, how does Ember deal with changes to the DOM done by third-party components, like jQuery based changes, which are not controlled by Ember?

@davidlormor
Copy link

davidlormor commented Feb 23, 2015

"the Dynamic Tree only includes nodes that may have changed (...) and never needs to compare static content that will never change"...sounds fast! 🚀 👍

@stefanpenner
Copy link
Member

stefanpenner commented Feb 23, 2015

This being a "DOM write-only" implementation, how does Ember deal with changes to the DOM done by third-party components, like jQuery based changes, which are not controlled by Ember?

It doesn't and likely cannot support this. It would be impossible to know what those plugins are doing

@mgenev
Copy link

mgenev commented Feb 23, 2015

@stefanpenner that means we can no longer use bootstrap (jQiery dependent) in an ember app unless a bootstrap-for-ember version is written, right? Same goes for foundation etc...

@stefanpenner
Copy link
Member

stefanpenner commented Feb 23, 2015

@stefanpenner that means we can no longer use bootstrap (jQiery dependent) in an ember app unless a bootstrap-for-ember version is written, right? Same goes for foundation etc...

No, this is no different then today. If you component is re-rendered your bootstrap and jquery code needs to re-run.

@wycats
Copy link
Member Author

wycats commented Feb 23, 2015

@stefanpenner @mgenev I think there's a little bit of confusion here.

From new-world Ember's perspective, as long as the element for a component still exists in the DOM, it doesn't matter where it has been moved, whether it has been decorated, etc.

There are a few restrictions on moving elements around, but they're very minor and would be easy to abstract. Stay tuned as we continue to make more progress.

@tomdale tomdale force-pushed the idempotent-rerender branch from c542bd7 to 27ed45f Feb 23, 2015
// and generate:
//
// - {{#each p in people}} (legacy)
// - {{#each people as |p|}} (legacy)

This comment has been minimized.

Copy link
@ef4

ef4 Feb 24, 2015

Contributor

why is {{#each people as |p|}} marked as legacy? Are we not keeping that syntax?

// want to require non-enumerability for this API, which
// would introduce a large cost.

return intern(debugName + ' [id=' + GUID_KEY + Math.floor(Math.random() * new Date()) + ']');

This comment has been minimized.

Copy link
@ef4

ef4 Feb 24, 2015

Contributor

Are there really platforms where Math.random() is so broken that we need to multiply by new Date?

This comment has been minimized.

Copy link
@tomdale

tomdale Feb 24, 2015

Member

Dunno, let's have our IE friends run http://jsbin.com/lexonidara/2/edit?html,js,output overnight.

This comment has been minimized.

Copy link
@jdalton

jdalton Mar 4, 2015

Ran this in IE11 overnight and no collisions.

This comment has been minimized.

Copy link
@loopmode

loopmode Mar 6, 2015

I came from using millis to multiplying by random in that particular order, so the idea was to allow for more possible values during a single CPU cycle. Maybe the author had a similar purpose in mind.

@tomdale tomdale force-pushed the idempotent-rerender branch from 222dd70 to cdb8894 Mar 1, 2015
@rwjblue rwjblue changed the title [WIP] <my-component> syntax and stable full re-renders [WIP] Implement Glimmer Engine Mar 3, 2015
@rwjblue
Copy link
Member

rwjblue commented Mar 3, 2015

Updated title + description based on EmberConf presentation.

@toblender
Copy link

toblender commented Mar 3, 2015

Minor correction: teack === track?

@morenoh149
Copy link

morenoh149 commented Mar 3, 2015

@toblender yeah. I thought he meant teach

@darkbaby123
Copy link
Contributor

darkbaby123 commented Mar 4, 2015

@wycats You mentioned a lot about diffing values to check if they're dirty. I'm curious what's the difference between Glimmer's diffing logic and Angular's dirty-checking.

As I know Angular dirty-checking also checks values (expressions) one by one and execute corresponding callbacks (mostly change doms). But:

  1. Angular doesn't have run loop so it's can not batch update doms.
  2. watchers need to be triggered by $apply() manually sometimes.
@drogus
Copy link
Contributor

drogus commented Mar 4, 2015

I'm curious what's the difference between Glimmer's diffing logic and Angular's dirty-checking.

These two diffing situations are not happening at the same level. Angular's diffing happens outside of a view layer in order to check if a template change is needed. So let's say that you have a user = { login: 'drogus' } object. And you use it in a template as <span class="login">{{user.login}}</span>. Now, in order to update the template automatically you need to know if a value of the user object was changed. In Angular it's done by dirty checking all of the properties used in templates. In Ember properties are observed, so it knows when a property is changed.

This PR and diffing described here happen in a view layer when a framework already knows that it needs to rerender something, but it needs to figure out what to rerender exactly. DOM is slow, so regenerating the entire template may be costly and here comes the diffing algorithm that React started with.

@bitinn
Copy link

bitinn commented Mar 5, 2015

One interesting side-effects of React (or other similar dom-diffing) approach is you can worry less about what's being updated in DOM (as oppose to update them manually on value change).

What sort of limit will value-diffing approach impose on DOM manipulation? Would you say this approach is closer to something like Riot than React?

Update: to me, it seems the key differences here are:

  1. dom-diff approach allow user to define almost arbitrary dom update logics, then perform it through a full render.
  2. value-diff approach capture those logic via template/helpers, and can minimize the need to go through static dom tree.
  3. value-diff is conceptually closer to traditional template renderer, though Glimmer use batch dom update (what dom-diff commonly do) to improve performance.

Apologize if I am totally mistaken :)

@Adriaaaaan
Copy link

Adriaaaaan commented May 6, 2015

Great work! although our app is currently very broken still I'm already seeing a huge improvement in performance in my list views. Now i just have a million deprecation warnings to fix

@wycats
Copy link
Member Author

wycats commented May 6, 2015

@Adriaaaaan That's great news! Please do report any notable breakage. We want things to be pretty clean by the time we ship 1.13 final :)

You might also want to run in prod mode to check out perf. Those deprecation warnings aren't free to generate ;)

@chrism
Copy link

chrism commented May 6, 2015

Great Work!

Anyone using Liquid Fire should check that ember-animation/liquid-fire#276 has been resolved before updating as it is not yet glimmer compatible.

@jmurphyau
Copy link
Contributor

jmurphyau commented May 6, 2015

@wycats any help you can provide on unbound helpers? A helper that produces a stream like value that can be invalidated causing the value in the template to update? Trying to get ember-get-helper to work with glimmer.. Simply returning a new child stream (e..g objectStream.get(key)) doesn't cause a new value to render in the template when that value changes.. Do I need to look into some of the hooks or somehow subscribe to the new stream?

@jmurphyau
Copy link
Contributor

jmurphyau commented May 6, 2015

I can't seem to find that really detailed description of all the HTMLBars lifecycle hooks etc.. Do you have a link to that that you could provide?

@wycats
Copy link
Member Author

wycats commented May 6, 2015

@jmurphyau right now there is no public way for a helper to express dynamic dependencies, but that's a good idea.

Glimmer intentionally doesn't expose the internal stream machinery; it's changing quite a bit still, but it makes sense for a helper to be able to tell Ember when to invalidate it.

It also kind of makes sense for us to include this get helper in Ember itself honestly. It's a primitive, like {{component}}.

@wycats
Copy link
Member Author

wycats commented May 6, 2015

@jmurphyau as I said in the blog post, more info on the life cycle hooks is forthcoming :)

@Adriaaaaan
Copy link

Adriaaaaan commented May 6, 2015

@wycats Hmmm have looked into it more, I'm now seeing a rather significant performance regression. related to interacting with sub resources templates from a list. It seems that interacting with it causes the the whole tree to redraw making it unusable if there are a lot of rows per page. 1.11 is faster, its most notable when selecting the rows (the detail page is very laggy at appearing, the more rows in the list the longer it takes). Although it is clearly much faster at rendering (the page appears much faster) interacting with it is much worse as it seems to be doing more work everytime something changes. I'll see If I can make a simple jsbin

@EnglishCriminal
Copy link

EnglishCriminal commented May 6, 2015

@jiyinyiyong
Copy link

jiyinyiyong commented May 7, 2015

Hi, you mentioned there's slide in EmberConf 2015. But is there a video?
I can't find it in the list "EmberConf 2015 by Confreaks"
https://www.youtube.com/playlist?list=PLE7tQUdRKcyacwiUPs0CjPYt6tJub4xXU

@igorT
Copy link
Member

igorT commented May 7, 2015

@jiyinyiyong it's the keynote, towards the end

@jmurphyau jmurphyau mentioned this pull request May 7, 2015
@jiyinyiyong
Copy link

jiyinyiyong commented May 8, 2015

@igorT I don't get it. Could you point it out in the video?

found it, thx https://youtu.be/o12-90Dm-Qs?t=3077

@blessenm
Copy link
Contributor

blessenm commented May 8, 2015

Stupid question. So there is no virtual DOM in ember right? Just value diffing to find which part of the DOM needs to be manipulated.

@axemclion
Copy link

axemclion commented May 14, 2015

Glimmer is pretty fast - almost 25% faster !! Ran some tests and results here - http://blog.nparashuram.com/2015/05/performance-boost-in-emberjs-from.html

@jdurand
Copy link

jdurand commented May 15, 2015

@blessenm instead of diffing DOM it diffs a tree of possibly mutable states that are bound to the real DOM. It doesn't need to maintain a virtual DOM because it has a deeper knowledge of your apps state.

@blessenm
Copy link
Contributor

blessenm commented May 15, 2015

@jdurand thanks for the clarification.

@ballPointPenguin
Copy link
Contributor

ballPointPenguin commented May 18, 2015

The Glimmer Twins

@juggy

This comment has been minimized.

Copy link
Contributor

juggy commented on 5a1c5e1 Jun 2, 2015

Why not expose componentNameMap as a global config to enable custom inputs using the same convenient {{input prop type="custom-comp"}} ?

This comment has been minimized.

Copy link
Member Author

bantic replied Jun 3, 2015

@juggy Not sure exactly what you are asking. Using another type should "just work". E.g. {{input type="color"}} should be fine.

This comment has been minimized.

Copy link
Contributor

juggy replied Jun 3, 2015

type=date-picker to load the component date-picker for example with the value and properties set. But now I realise this is just sugar that might not be good in the long run. Just call the component directly.

I guess what bugs me here is that input either create a checkbox or a textfield, you also have textarea to create a text area. At the end we should get rid of all that and just call the components. Another discussion altogether. So nevermind for now.

This comment has been minimized.

Copy link
Member

rwjblue replied Jun 3, 2015

@juggy - {{textarea}} creates a <textarea>

This comment has been minimized.

Copy link
Member Author

bantic replied Jun 3, 2015

@juggy The type="checkbox" isn't actually intended to only be sugar to simplify creation of check box inputs. It's to reflect the fact that both text inputs and check box inputs are the same html tag (input) but have somewhat different semantics. The meaning of the value property is somewhat different between the two, and a checkbox input has a boolean checked property that is meaningless for text inputs. For custom inputs making your own component would definitely be the best way to go (i.e. {{calendar-date-picker-input}}).

@jmurphyau
Copy link
Contributor

jmurphyau commented Jun 12, 2015

@wycats someone has asked a relatively simple question on stackoverflow and in slack - 'what is a stream'.. I answered as best I could but wanted to bring it up here just so I could get a better understanding - what is a stream? Did the name 'stream' come from anywhere/anything in particular?

I mean - I do know what it is (I think) - just not sure how to best describe it..

@antigremlin

This comment has been minimized.

Copy link

antigremlin commented on 5e1787a Jun 16, 2015

Hi Robert,
Do you know of any future plans to support viewName? Is anything going to change with the upcoming deprecation of views in Ember 2.0? I have posted a question on Ember forums here but no one seems to know the answer.

@matthewp
Copy link

matthewp commented Jun 19, 2015

Is there somewhere that explains how Glimmer works in more detail, with links to code maybe? Given how many commits are included in this PR it's hard to figure out how the details are implemented in practice. For example you say:

On initial render, we track the last value (always a primitive) that we inserted into the DOM.

But I can't find in the code where this happens. And I don't know what you mean by a primitive value. I assume you mean that you track attribute string values and text node's string nodeValues but am having a difficult time finding where this happens in the code.

Is there maybe one or two commits that contain most of the work that I can look at?

@ef4
Copy link
Contributor

ef4 commented Jun 19, 2015

@matthewp Much of the infrastructure is actually in https://github.com/tildeio/htmlbars

There's not really a single commit to point to -- the work was extensive and many people put in hundreds of commits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.