Reactive graph transformations

[kmalakoff]

I'm really excited to see what you have planned for the "Reactive graph transformations" (well, I took at look at the code/tests, too!).

I'll explain my use case a little more for food for thought...

As chokidar is scanning for folders and files, it is emitting a stream of changes at a high rate so rather than processing them in an incremental fashion (which could be inefficient to render), I'm currently batching them up and them periodically merging them into the scene graph (the "source" representation) and then triggering a full scene render (the "target" representation) which then gets passed down through React to render "reasonably" statically. The way I am currently triggering the render is 1) by using a "modifiedAt" observable timestamp at the top level since there is a single root node and I'm using transactions both in the scene graph and rendered representation phases 2) whenever the view settings change (like the search filter).

So I think there are three main needs:

1. ease and efficiency of an incremental representation - because I'm batching change streams at the moment, I have made child relationships simple arrays instead of observable arrays because I could have peers of 500+ folders that would keep triggering changes during the batching process. If this was made incremental, I would need to search for where to insert the child into the scene graph and then incrementally determine if the generated child needs to be added to the rendered representation and where to put it.

2. an efficient way to regenerate the full rendered representation from scratch when the view settings change significantly

3. synchronicity and render management - because there is an asynchronous flow of chokidar events which may themselves cause more async file system calls, there still might be a small batching step where changes are being resolve asynchronously and them merged into the tree. If many changes appear at once, there may need to be a certain amount of rate limiting based on a framerate target. Like in computer graphics rendering, you sometimes drop frames instead of trying to keep up with an increasing backlog of work. Right now, I'm just measuring each phase and using it to set the maximum number of nodes to process and throttling updates accordingly.

If any of this is way too complicated for the short term, too edge-case, or there are good shortcuts, let me know. It's easy for my to imagine best case scenarios! I'd be happy to discuss and beta test...

[mweststrate]

Expect a more extensive answer from me later today, but a quick question: the current setup with recurse doesn't really encourage reuse and requires one 'master' transformation. So what would you like better as api (same test case):

    var transformState = function(state) {
        stateCalc++;
        return state.name + state.todos.map(function(todo) {
            return m.transform(todo, transformTodo);
        }).join(",");
    }

    var transformTodo = function(todo) {
        todoCalc++;
        return todo.title.toUpperCase();
    }

    m.autorun(function() {
        mapped = m.transform(state, transformState);
    });

or

    var stateTransformer = createTransformer(function(state) {
        stateCalc++;
        return state.name + state.todos.map(function(todo) {
            return todoTransformer(todo); // note: could be passed first class to map
        }).join(",");
    })

    var todoTransformer = createTransformer(function(todo) {
        todoCalc++;
        return todo.title.toUpperCase();
    })

    m.autorun(function() {
        mapped = stateTransformer(state);
    });

I think I slightly prefer the second option. It makes especially clear that you should be putting inlined closures into m.transform.

I don't like the autorun in this snippet bty, so maybe there should be an utility api that hides this (the autorun is the one that now causes the transformation to stay alive; mobservable will always try to switch to lazy evaluation if there are no active observers)

Maybe instead of autorun something like:

const transformedStateController = stateTransformer.root(state);
console.log(transformedStateController.value); // always returns the up-to-date transformed graph
transformedStateController.dispose(); // abort the transformer

[mweststrate]

I think your usecase is a nice one for graph transformations. The nice thing (theoretically, yet to be proven) is that the overhead of a single change in the data tree is (almost) constant, now matter how many items there are. This fits nicely into the @observer decorator which in itself also already exhibits this behavior.

I think after introducing transform I will also investigate feature like array.reactiveMap; so that a change in an array does not trigger a complete array to be mapped again (which will already be a lot more efficient when using transform as unchanged items will directly be returned from the memoized transforms). A reactiveMap will just return a map once, and then that map will start observing the source array so that only updates (splices) need to be processed. You can btw already achieve that manually by using observableArray.observe which emits events similar to ES7 Array.observe.

For very fine grained control over when updates should be propagated it is always possible to not use observable structures and change an observable timestamp or tick counter to push updates.

Reactive views and values can currently be in three different states: ready, stale and pending. If mobservable allows to create values for which you can control this state yourself that could provide a useful way to manage batches as well (a view will never recompute as long as one of its used values remains stale). But I'm not sure whether this wouldn't collide too much with core values of mobservable; glitch-free, synchronous updates.

But anyway, I'm very interested how the reactive transformations will work out, so I'll focus on that first ;)

[mweststrate]

I've pushed an initial implementation to this branch:

Would you mind trying it? Just clone the repo and run npm install in the checkout and after that npm install <cloneddir> in your project.

The api:

mobservable.createTransformer(transformation, cleanup): give it a one argument function that transforms an observable thing into something else. Returns a function (transformer) with the same signature that memoizes the reactive transformation. The cleanup function is optional but will be invoked as soon as the transformation is no longer part of the graph. Receives both the source and transformation result and can be used to clean up resources associated with the objects.

transformer(object) use this somewhere in a reactive context to transform an object into something else and track the transformation.Can be used inside reactive functions, autorun, @observer, other transformers etc.

transformer.root(object): just like transformer(object) except that it will never fail back to lazy evaluation if nobody used the transformed graph. Returns an object with two properties: value (with the transformed graph) and a dispose() function to dispose the thing.

Note that the created transformer functions act as cache, so make sure to create them only once!

Only one argument might feel like a limit, but note that you are free to access any reactive value from the source object, the function closure etc. The transformers will respond to that.

Quick example:

const transformState = m.createTransformer(state =>
    state.name + state.todos.map(transformTodo).join(",")
)

const transformTodo = m.createTransformer(
        todo => todo.title.toUpperCase(),
        (todo, text) => console.log("Bye", todo.title, "aka: ", text)
);

// use the transformation
var transformedState;
m.autorun(() => {
    transformedState = transformState(state);
});

// or:
var transformController = transformState.root(state);
transformedController.value; // contains the transformed graph, note that the pointer might change over time!

// or just in some class:
@observable get transformed() {
        return transformState(state)
}

Any feedback is welcome! Note that there is no extensive test suite yet, so if you run into any issues; don't stare too long at them and just report them :)

Thanks for trying out in advance!

[ds300]

I will also investigate feature like array.reactiveMap; so that a change in an array does not trigger a complete array to be mapped again

I solved this problem already for Derivables. Took me a whole mind-bending day but I wrote a fairly verbose discourse on how I got to the solution. Might save you some time :)

[kmalakoff]

@mweststrate I'm just seeing this now so will look at it and provide feedback in the next few hours.

Thank you!

Update: ran out of time and will pick up in the morning (PST).

[mweststrate]

@ds300 thx for the pointer. I think it is a bit simpler to solve in mobservable as I can just observe the splice events and map those. Cool thing is that this can be done to most other common operations as well, filter, slice, sort etc. Did something in the past already (in a project that was further too complicated to be viable :-P.)

Biggest question for me is how you can avoid / cache when people create the map inside a closure using the closure (they might be used to do that). For example the following snippet would create a 'new' reactiveMap all the time:

** Naive approach **

class stuff {
    @observable projects = [ /* projects */ ];
    @observable get filteredProjects() {
           return this.projects.reactiveMap(project => project.favorite);
    }
}

** Create the reactiveMap only once **
In contrast, the following is the correct approach

class stuff {
    @observable projects = [ /* projects */ ];
    filteredProjects = this.projects.reactiveMap(project => project.favorite);
}

** Memoizing reactiveMap **
And this one could be made to work correctly out of the box by memoizing the reactiveMap (similar to createTransformer, assuming that the same map function is passed in each time (the first example is impossible to memoize as it passes a new closure in each time reactiveMap is called):

class stuff {
    @observable projects = [ /* projects */ ];
    @observable getFilteredProjects() {
           return this.projects.reactiveMap(project.isFavorite);
    }
}

But the differences are quite subtle, so for me the primary question is: how to make this clear / educate the users?

[mweststrate]

@kmalakoff: Conceptual question: currently the graph transformation doesn't keep parts of the graph up to date as long as there are no observers (such as autoruns or @observer components). Is this nice for effeciency or would it be more predictable / efficient if the whole graph is always kept in sync?

So suppose your project tree contains 1000 transformed folders, but your UI displays only 10 of them. Would it make more sense if only those 10 transformed folders are kept in sync (making it cheaper to keep the graph in sync but more expensive if you start displaying 10 other folders) or should they be kept all in sync at all times (making it far more easier to inspect the transformed graph in non-reactive code as no transformations need to be applied lazily)?

I guess the later makes more sense as it is easier to understand / more predictable what the transformation does? The current implementation takes the first approach.

[kmalakoff]

@mweststrate let me give a concrete scenario...let's say I have a TreeNode like:

class TreeNode {
  @observable children = []; // tree
  @observable isCollapsed = true; // filter-relevant observable
  @observable icon = 'folder'; // render-only observable

  constructor(name) {
    this.parent = null; // not observed
    this.name = name; // not observed
    this.tags = []; // not observed
  }
}

I would expect that only accessed observable properties are tracked by the transformation and that this is the main method of letting the transformation know what to watch. (Note: in KnockoutJS we had peek() so that if you need to traverse the children to find a node or reduce something like does any child or myself have a specific tag the whole subtree isn't inadvertently watched).

Scenario 1: Add a subfolder / child to a TreeNode

In this case, I would push a new node into the parent's children potentially triggering the reprocessing. If the parent node is visible, it's children should be tracked and so the transformation would be triggered on that node again. If the parent isn't visible, its children would not be tracked so the tree insertion should not re-trigger the transformation.

Conclusion: I think that means the former...a dependency on children

Scenario 2: Change the tags filter

The root node and all nodes up to where the subtree does not contain the tag (using peek to avoid unwanted dependencies in the subtree tag reduce), but no tracking should be done on the node tags since the tree needs to be rebuilt when the search filter changes.

Conclusion: The node tags shouldn't be tracked, but the dependency should be only made on the search tag...I think that means the former...a dependency on searchTags (but not TreeNode tags) and to trigger a full transformation from root.

Scenario 3: Change isCollapsed on a TreeNode

This is a local dependency that is may cause a batch of subtree Nodes to be added or removed. If the transformation is to an array (not sure if array is the only target type, but it is enough for me), those new nodes should be spliced in / out somehow.

Conclusion: I think that means the former...a dependency on isCollapsed

I think everything should be incremental and Scenario 2 is the interesting one. As for developer productivity, special tooling might handle it (like toggling an observable in the console and logging the transformation actions) rather than needing to track more for inspection purposes.

I'm not sure if this helps (or actually answers your question), but this is what I would expect!

[kmalakoff]

@mweststrate on the reactiveMap question But the differences are quite subtle, so for me the primary question is: how to make this clear / educate the users?...

Personally, I think this is probably a non-issue since it is based on a general understanding of JavaScript.

If I use a getter like @observable get filteredProjects(), I expect the code inside the function to be fully executed on each get so it would smell wrong when I considered writing it and I would notice the inefficiency the first time I stepped into the code (duh, it's recreating the whole reactiveMap on each get...better never do that again!).

If you draw attention to it in the docs, you can save people time to figure it out the hard way.

Would this be a good workaround?

    @observable get filteredProjects() {
           return this._filteredProjects = this._filteredProjects || this.projects.reactiveMap(project => project.favorite);
    }

[mweststrate]

@kmalakoff @dslmeinte

Let's call the different approaches eager and lazy.

Lazy: only evaluate if there is an observer (switch to eager) or if somebody accesses the value (just evaluate lazily but don't track further)
Eager: always keep transformations fresh, even if nobody is observing.

The risk of lazy is that you have a lot of unecessary evaluations if observers hop on and hop off and in the meantime the outcome of the derivation doesn't change.

The risk of eager is that you have a lot of unecessary evaluations if parts of the graph are never observed.

Lazy is the current behavior as it is the default behavior of mobservable.

Scenario 1:
Both in eager and lazy the behavior would be the same; the change only results in a transformation if the parent node is visiable

Scenario 2:
Ha this is an interesting one! I'm interested why you want to peek here: If you don't peek it means that your transformation graph will automatically rebuild, but only the parts where the visibility is changed as result of a folder toggling from invisible to invisible (or vice versa).
I think this is more efficient than having to peek walk the whole tree upon each filter change. Of course it would be interesting to actually measure that assumtion :)

Mobservable gurantees that if something like that happens each transformation will be recalculated only once by ordering them correctly. So if both parent and child needs a re-evaluation the child will be re-transformed first.

Scenario 3:
Quite similar to 1.

---

Too look more philosophical at scenario 2, I would also depend on TreeNode tags:
Its basically a difference between imperative search or declarative search (@get isVisible = folder is visible if current search tag is in the folder's tags). My philosophy is that there is always a minimal amount of computations you have to do to achieve something, like a search like this. This might be faster if you build it manually, as mobservable will always have a little overhead for the administration of relations between data. But on the other hand mobservable will optimize all possible data mutations that affect the outcome. For example if you add a tag to a folder while a filter is active, mobservable will not re-evaluate the whole tree while a manual implementation would, unless you build specific exception for such cases, in which case you have to be sure you know all of these cases.

---

Good points on the reactive map question, probably people will figure out soon enought when it is clear that reactiveMap itself returns a reactive structure. Your workaround is valid indeed.

[kmalakoff]

@mweststrate I'll respond in a second.

I've been struggling a bit to get the test finished (troubleshooting dependency cycles when I start using observables on the DisplayNode) but I thought I'd share my work-in-progress rather than delay further: kmalakoff@688b1f8

One change I needed to make was to not force the array items to be mobservables: kmalakoff@688b1f8#diff-510d154cf95f37822e100bfd4e7a03cdL84. This restriction was a little too strict because an object may have observables, but not be an observable.

Let me know how you'd like to work on the tests with me...I've added you to my repo to keep things simple.

[kmalakoff]

I've added lifecycle checks. Another piece of feedback is that maybe the argument order should be reversed since you are likely to be more interested in the transformed node (at least that was my first intuition):

    var transformNode = m.createTransformer(function(node) {
        nodeCalc++;
        return new DisplayNode(node);
    }, function cleanup(node, displayNode) { // KM: maybe the transformed node should be the first argument?
        displayNode.destroy();
    });

It is not really important though, more of an observation.

At any rate, it seems to be working totally smoothly as one would expect! Really awesome!

[kmalakoff]

@mweststrate OK. I've written a bunch of tests and there's only one problem with a transformed node not being released.

Summary

1. I loosened the need for transform nodes to be observables since I was using asStructure and classes: kmalakoff@688b1f8#diff-510d154cf95f37822e100bfd4e7a03cdL84
2. There was a lifecycle bug with an observable that creates a dependency to regenerate the node: https://github.com/kmalakoff/mobservable/blob/feature/transform/test/transform.js#L543
3. Maybe the transformed node should be the first argument: https://github.com/kmalakoff/mobservable/blob/feature/transform/test/transform.js#L166
4. Ideally, I'd like to do a direct assignment here, but but it creates a cycle and would need to preserve asStructure: https://github.com/kmalakoff/mobservable/blob/feature/transform/test/transform.js#L170
5. Ideally, I'd like to use an observable set instead of map: https://github.com/kmalakoff/mobservable/blob/feature/transform/test/utils/transform.js#L18 and https://github.com/kmalakoff/mobservable/blob/feature/transform/test/utils/transform.js#L84

This is really awesome! Thank you!

[kmalakoff]

On the eager vs lazy discussion:

The risk of eager is that you have a lot of unnecessary evaluations if parts of the graph are never observed.

I am planning on using the transformed result to make exactly what I want to render so I need to step outside my thinking to imagine when this would come up. My gut says that unless there is a concrete, compelling use case (I haven't spent much time thinking about this, but maybe there is?), it should probably match the default, eager behavior of mobservable since it is what people probably expect, eg. sensible defaults, and when a use case presents itself through a GitHub issue, implement lazy and enable it through an option or mobservable.lazyTransformation.

On Scenario 2, I skipped it in my tests so I'll need to implement it and probably some more similar tests. I won't have time for this until Thursday. I was thinking that the whole tree could change when the search tags change so every node needs to be visited anyways, eg. unlikely to benefit from incremental algorithms.

[kmalakoff]

@mweststrate I found some time and wrote the tag tests. I made a static (eg. global filter) and dynamic version (incremental), and it seems amazing how optimized this is. When I put it in my application, I'll try performance testing.

That said, in order to use dynamic tags, I'll need your feedback on because just adding a dynamic array slowed things down in the creation of node: #68

I added a dev dependency on lodash.intersection for the tags test: https://github.com/kmalakoff/mobservable/blob/feature/transform/test/utils/transform.js#L3. If you don't mind small dependencies, you can leave it in, but if you try to keep everything minimal, maybe a special, hand-crafted version could be written (so I put it with the utils).

Also, I wasn't sure how to implement collapsed or expanded on the display nodes themselves in a clean way so I went with an external map (although a set would have been the better choice). The problem is guaranteeing persistence while the nodes are created and destroyed with the filter so the external representation may be the best choice, but also because in the transform function, I am only passed the node, but because the transform hasn't been completed, I cannot easier find the display nodes in it. I could cache the display node on the tree node, but ideally multiple trees of display nodes could be created if you want to transform the tree in different ways. Not sure if there is a better way, eg. passing the inflight transformed results perhaps?

Finally, I'd love if you can review my test code and look for better ways to do things to provide me feedback. For example, I never used the transformState pattern or transformController / root pattern so it wasn't clear if they were actually needed. Really, I just want to keep the state.renderedNodes in sync with the transformation pipeline with the least boilerplate possible (ideally state.renderedNodes = state.root ? state.root.map(transformNode) : [];) or maybe a getter? Let me know what you recommend.

---

PS: I looked at #2 (above):

There was a lifecycle bug with an observable that creates a dependency to regenerate the node: https://github.com/kmalakoff/mobservable/blob/feature/transform/test/transform.js#L543

Basically, a display node was not being destroyed, but I found that changing where I tracked the dependency made the problem go away:

Bad

    var transformNode = m.createTransformer(function(node) {
        nodeCreateCount++;
        node.icon();  // icon dependency
        return new DisplayNode(node);
    }, function cleanup(node, displayNode) { displayNode.destroy(); });

Good

    // custom transform
    TreeNode.prototype.transform = function(iter, memo) {
        node.icon();  // icon dependency

        memo = memo || [];
        memo.push(iter(this));
        this.children.forEach(function(child) { child.transform(iter, memo); });
        return memo;
    }

So I don't understand understand why one is the good place to track a dependency and the other one is not! I'm not sure if there is a self-evident way to handle this or if docs are enough or if the dependency can be tracked in either.

Anyways, it means all of my test cases are passing! Very awesome...I feel like I should migrate my app over! I can't wait for your blessing on when it is ready...

[mweststrate]

Thanks for the extensive testing! Really useful!

I'll try to answers the questions in order, just let me know if I missed something:

---

Concerning (1): The looser is observableCheck is fine. Imho. Probably we should still check if value !== null && typeof value === "object".

Concerning (2): I don't think the link resolves anymore

Concerning (3): Swapping the arguments of cleanup makes sense.

Concerning (4): that is actually what the .root is for; if your root transformer will return new objects the root transformer will make sure it is stored in its .value property.
Is there a specific reason that you define children etc asStructure? Since transform will return (and recycle) reactive DisplayNodes it is more efficient to not use structural comparison.

Note btw that you can also expose children and icons as properties in TreeNode by doing
m.extendObservable(this, { icon: 'children' }) etc.

Concerning (5): Could you file a separate request for observable sets?

---

For eager versus lazy, I'm currently writing some benchmarking test so that a more educated decision can be made on which approach to choose.

---

Concerning #68, what is the findOrCreate in the profiler output?

Adding more development dependencies is no problem.

The collapsing state is interesting. At least it should be state somewhere; that is either on the state object or on the source folders. It could be state of the displaynode as well, but then the state would be lost as soon as the displayNode is no longer visible. I'm not sure whether that is acceptible in your case, so I think your current solution is fine.

Concerning transforming the first item; a getter is a nice way to keep the render pipeline clean (assuming you observe that getter somewhere).

I'll try to review your code more in depth later today, but I think in general this setup is fine!

---

Concerning the transformer difference; there are actually two styles of transformers: The ones that return a reactive data structures and those that don't. For the first (your case), you don't want the transformer to run ever again because the returned DisplayNode can react directly to the source TreeNode and there is no need to create a new one (unless it was really disposed). For the second category of transformers, which don't return reactive data structures (like the first tests in the test suite) you want the transformer itself to always run again to keep the transformed data in sync with the source data.

The return new DisplayNode statement does not observe anything; it just creates some new reactive functions but that in itself doesn't establish a reactive relationship. However, the node.icon() will cause the transformer to actively observe the node, so after that both the transformer and the node returned by the transformer will be observing the source node. I think that explains roughly the difference, although I didn't dive into the details of the second transformer yet (what are memo and iter?). Probably I'll take a deeper dive into this later today.

[kmalakoff]

This is fun...you make it look easy!

(1) The looser is observableCheck is fine. Imho. Probably we should still check if value !== null && typeof value === "object".

Great. I wasn't sure about it but scratched my head for a while trying to minimize the hoops I was going through syntactically. In Knockout, the API is a little more symmetical - get() and set(value) so with the various options available to me in mobservable, I sometimes find myself trying to figure things out. For example:

Based on my experience with Knockout, these feel like they should be equivalent, but the second one has an error that map (my instance function) isn't defined...I'm still trying to understand conceptually the difference...

var store = observable({root: null});
state.renderedNodes = state.root ? state.root.map(transformNode) : [];) 

var store = {root: observable(null)};
state.renderedNodes = state.root ? state.root.map(transformNode) : [];) 

(2): I don't think the link resolves anymore

Sorry about that. I inlined the It bad / good case in a subsequent post. You've continued the thread with The return new DisplayNode statement does not observe anything;

(4a): that is actually what the .root is for; if your root transformer will return new objects the root transformer will make sure it is stored in its .value property.

Hmmmm. The syntax / boilerplate wasn't totally clear to me for two reasons:

1. return state.name + state.todos.map(transformTodo).join(","); - what does this line do? I can understand state.name += state.todos.map(transformTodo).join(","); would update the name, but why the return, why + instead of +=, and where does it know where to store the result?

2. how to I get the .value property back into my store? It looked like the transformController was floating independently of my store so it wasn't clear how to pipe it back in.

(4b): Is there a specific reason that you define children etc asStructure? Since transform will return (and recycle) reactive DisplayNodes it is more efficient to not use structural comparison.

Like in #55, it seems like mobservable's default behavior should to leave it to me to choose if I want recursive conversion to observables...opt-in (default is shallow) instead of opt out (asStructure). I sort of think opt-in is better because I use the class notation and hand-craft which properties to observe and so I want to block the recursion meaning I've been using asStructure. It could be a problem with my understanding of mobservable, but as soon as I saw that mobservable was recursively creating observables while I was also hand-selecting them (using classes and property decorators), I read the documentation to try to figured out how to turn off recursion to give me fine-grained control...hey mobservable, hands-off my hand-optimized class instances...I'll let you know when I want you to help! 😉 (maybe I'm missing something, but there have been zero cases so far where I wanted recursive observables)

It's like in Knockback, I allow all attributes and nested relationships to be made observable by default which is great for quick prototyping:

 /* the model's attributes can possibly include nested and cyclic relationship graphs */
var vm = kb.viewModel(model);

but in production code, I recommend hand optimizing each ViewModel to observe only what is needed because observables are expensive and because recursion can create deeply nested trees (even though I resolve the cycles to the same ViewModels to allow reuse and to avoid infinite recursion):

 /* the model's attributes can possibly include nested and cyclic relationship graphs */
var vm = kb.viewModel(model, {statics: ['id'], keys: ['name'], factories: {'related': RelatedViewModel, 'children.models': ChildViewModel}});

(4c): Note btw that you can also expose children and icons as properties in TreeNode by doing
m.extendObservable(this, { icon: 'children' }) etc.

I did use that syntax once, but prefer a symmetric syntax like:

this.name = name;
this.observed_name = m.observable(name);

This could be related to my experience with Knockout. I wrote Knockback where I got in the habit of adding additional basic and computed observables in the longhand for readability even though I could have been syntactically more sugary using extend because they could be documented and grouped more easily for self-documented code. Decorators and instance variables make this type of self-documenting approach even better (instead of ES5)! Also, this could be because of my problems understanding the differences in (1) - I'm lean towards symmetry and ease of readability.

(5): Could you file a separate request for observable sets?

Done: #69

---

MW: Concerning #68, what is the findOrCreate in the profiler output?

  findOrCreate(path) {
    let projectNode = this.projectNode(); // this makes sure there is a root node
    let node = projectNode.find(path);
    if (node) return node;

    // this traverses splits the path and traverses the tree to find the node and will generate the full tree down to it if the path does not exist. It will create the default observable array of tags for each new node based on the name of the node and assign the passed tags.
    node = this.projectNode().findOrCreate(path, (attributes) => {
      return new LocalNode(_.extend({project: this.project, path, attributes, tags: [NAME_TO_TYPE[attributes.name] || 'module'])); // this only gets called if the node is not found. It extends properties into the class
    });
    return node;
  }

The constructor:

  constructor(...args) {
    super(...args); // this will set the non-observed tags property
    this.tags = observable(asFlat(args[0].tags || [])); // this will re-set the tags property with a non-recursive tags observable 
  }

I construct the tags like this for two reasons: 1) the caller doesn't need to pass in observables, eg. slightly shorter syntax, but more importantly 2) to tell mobservable that this is not a modification, but a create so nothing is observing it yet and you can go ahead and use a create code path to optimize the creation of a values-only (asStructure) observable array instead of worrying about subscriber bookkeeping yet.

---

MW: Concerning transforming the first item; a getter is a nice way to keep the render pipeline clean (assuming you observe that getter somewhere).

KM: Can you show me what you would do in a modern ES syntax considering (4a) above?

---

MW: The return new DisplayNode statement does not observe anything; it just creates some new reactive functions but that in itself doesn't establish a reactive relationship. However, the node.icon() will cause the transformer to actively observe the node, so after that both the transformer and the node returned by the transformer will be observing the source node. I think that explains roughly the difference, although I didn't dive into the details of the second transformer yet...Probably I'll take a deeper dive into this later today.

KM: Great. I'm looking forward to what you find out...If what I did when it wasn't working properly is problematic (eg. shouldn't set up observable relationships with the transformer), maybe in a debug mode you could wrap the transformer in an untracked statement and see if any subscriptions were created and warn the user?

MW: what are memo and iter?

KM: iter is the transformation function and memo is to collect the results recursively. Think of memo just like in reduce for an array to collect results instead of to reduce them (memo is what underscore calls it in reduce), but for recursive tree mapping. It avoids temporary and concatenating arrays.

Array.prototype.map(fn) {
   var results = []; // is available to all iterations because there is no recursion
   this.forEach(x=> results.push(fn(x)));
  return results;
}

TreeNode.prototype.map(fn, results) {
   results = results || [];
   this.forEach(x=> results.push(fn(x, results)));
  return results;
}

Results or memo needs to be passed through the tree to optimally collect results, but you want to call it like var transformed = root.map(fn); instead of var transformed = []; root.map(fn, transformed);

Instead of creating and concatenating arrays at each node:

TreeNode.prototype.map(fn, results) {
   var results =[];
   results.splice.apply(this, [0, results.length].concat(this.map(x=> fn(x))); // or some other merging logic
   return results;
}

---

I think this is really turning out great! I really appreciate you devoting so much time to making this awesome...

[mweststrate]

Regarding:

var store = observable({root: null});
state.renderedNodes = state.root ? state.root.map(transformNode) : [];) 

var store = {root: observable(null)};
state.renderedNodes = state.root ? state.root.map(transformNode) : [];) 

The difference between the first and the second is that the root reference itself will be observable. So in the first case you can safely do store.root = [] and it will be picked everywhere (e.g. to consumers it would be the same as store.root.splice(0)). While in the second case nobody would pickup the change as the store object itself isn't observable, nor its root reference to the list.

---

1. return state.name + state.todos.map(transformTodo).join(",");... its a transformer that doesn't produce a new object, but just a string (the "michelBISCUIT,TEA") thing. So it isn't about changing the state itself (derivations should never change state imho) but transforming the state into a new value, which happens to be the concatenation of name and all the todo values.

2. Yes that is a bit clumpsy. You need either a getter (but in that case you don't need the transformation controller) or use autorun (and in that case you don't need the controller as well). So I guess the whole root() construction just doesn't have added value. I introduced it because autorun looked to me a bit clumpsy to wire up the transformation, but probably it isn't that bad. (although an observable getter is even nicer).

In ES6 / typescript that would be:

class Store {
    @observable get renderedNodes() {
        return state.root ? state.root.map(transformNode) : []
    }
}

---

Ah I think you got a misunderstanding about what asStructure does, what you need is asReference. asStructure compares values structurally, so assigning a different object that does look the same doesn't result in changes being propaged, while asReference treats every value as 'just a reference to observe' without recursing. Now I can understand this is totally confusing. The opt-in is basically to get people up & running quickly but it might not be a sane default in real projects. I guess I should change it in 2.0. (I'm also not completely excited about the asXxx functions, it's ok-ish for ES5 users but in ES6 it is nicer to make these modifiers arguments of the decorator).

---

It is perfectly fine indeed to not use extendObservable. You just have to be aware that you don't accidentally assign a new value to the property but invoke the setter instead. But as plainly created observables are functions I guess that is pretty clear.

... answer to be continued, have to dine ;)

[kmalakoff]

Makes sense. One quick note (I'm on my out for the day)...

MW: Ah I think you got a misunderstanding about what asStructure does, what you need is asReference.

KM: I do find them confusing! I totally prefer to get rid of asX and use decorator options, but since if shallow is not the default in 2.0 and because I use decorated classes and because I don't like to have to think too much about basic decisions, I probably will ever only use one (rather than having to care if I am passing a value, struct, class instance)...like observable(raw([])) so maybe having mobservable inspect to figure out what it is.

I'm not sure about the other asX variants since I haven't had a use case yet...maybe they should be creation helpers instead of asX? eg. m.structArray(values) or basic types? Eg. m.nestedObservable, childObservables, etc. Agreed that asX could be better!

[mweststrate]

Good points. I will definitely revisit. For now just go for asFlat for
arrays (1 deep reactivity; the values of the array) and asReference for the
rest :)

On Wed, Dec 9, 2015 at 5:39 PM, Kevin Malakoff notifications@github.com
wrote:

Makes sense. One quick note (I'm on my out for the day)...

MW: Ah I think you got a misunderstanding about what asStructure does,
what you need is asReference.

KM: I do find them confusing! I totally prefer to get rid of asX and use
decorator options, but since if shallow is not the default in 2.0 and
because I use decorated classes and because I don't like to have to think
too much about basic decisions, I probably will ever only use one (rather
than having to care if I am passing a value, struct, class instance)...like
observable(raw([])) so maybe having mobservable inspect to figure out what
it is.

I'm not sure about the other asX variants since I haven't had a use case
yet...maybe they should be creation helpers instead of asX? eg.
m.structArray(values)

—
Reply to this email directly or view it on GitHub
#67 (comment)
.

[mweststrate]

Btw would you mind if I merge your branch already back? The tests are very useful!

[mweststrate]

@kmalakoff Btw knockback looks fancy! Didn't know it. I did take a lot of inspiration from knockout btw, it actually inspired me to build mobservable :)

[kmalakoff]

I'm back online...

I've created a pull request for the merge.

(1) Regarding: observable({root: null}) vs {root: observable(null)}...makes sense and I now understand better about the differences. There is just something that maybe seems unintuitive to me. It could be that because mobservable recursively applies observables and knockout does not.

In knockout, you would do something like:

// observable
var store = ko.observable({
  root: null
});
store().root; // get
store(assign(store.peek(), {root: []})); // set

// view model
var store = {
  root: ko.observable(null)
};
store.root(); // get
store.root([]); // set

So you probably wouldn't use the observable approach for a store since you want to modify the whole store at once, but each element individually. Approaching mobservable with this frame of reference, my intuition is that I shouldn't be using the observable pattern, but the view model pattern and would expect this syntax to work, eg. mobservable uses set / get magic on root instead of root() / root(value).

I sort of expect that the following is equivalent:

// view model - plain old object
var store = {
  root: observable(null)
};
store.root; // get - instead needs to be store.root();
store.root = []; // set - instead needs to be store.root([]);

// view model - class
class Store {
  @observable root = [];
}
var store = new Store();
store.root; // get
store.root = []; // set

I think my answer is to not use observable({root: null}) nor {root: observable(null)} syntax, but only stick with the class syntax since it intuitively makes sense to me because I think of mobservable's observables like knockout observables but with get / set magic. I think it just an incorrect mental model on my side biased by Knockout...in my head, if view model - plain old object worked like I wrote above, both versions would be equivalent. Probably just me and not something that get / set was meant for...

It could also be that observable conceptually does too much (I think of it as an atom, not a molecule). Maybe there needs to be separate APIs in mobservable: observable vs view or view model or store (the object formulation of observable) to mentally keep this clear? (eg. rename your existing view to computed and observable with object to something explaining its molecular nature and related to the class syntax)

---

1. return state.name + state.todos.map(transformTodo).join(","); now makes sense to me.

I think it was the name of the transform transformState that was throwing me off since it sounded like it was going to transform the state into a new or updated representation. What about calling it stateSummaryTransform in the tests? eg. it transforms the state into a summary representation? (then you would rename mapped -> summary and maybe transformController -> summaryController or summaryResult.

2. I like the observable getter and will use that myself. In non-ES6 / typescript, what would be a good API to keep a transform in sync with an observable result holder without the root and autorun syntax?

Would something like this work (mind you, the class syntax is more familiar to me):

var store = observable({
  root: null,
  renderedNodes: function() { return this.root ? this.root.map(transformNode) : []; }
});

// or
var store = observable({
  root: null,
  renderedNodes: []
});
store.renderedNodes.sync(function() { return store.root ? store.root.map(transformNode) : []; });

---

On the asStructure, asFlat...like you say, I'm sure there is a better way. You can probably add to your API without doing a semantic version upgrade given the non-breaking nature of it. Just what the API is, I'm not sure, but I know that I prefer not to have to decide between three variations...try to keep it to one or two choices max for observable or if it ends up needing three variations, think of a way that it takes no thought at all to know which one to use when (eg. use a mental test like...the first time to see this feature / variation being used in code or an example, do you know what it does if you know how an observable works and without referring to the documentation).

---

As for my optimization problem (#68), let me know if you agree that initial setup vs updating an observable having different code paths makes sense. Poor construction performance would be the only blocker for me upgrading to transform right now!

[kmalakoff]

I should probably say that I realize some these ideas may be extreme, based on a different mental model than you have in mind, based on poor understanding of all the use cases or ES6 language features, etc so definitely, don't worry about saying you are not going to take your library in any of these directions.

I just wanted to share my impressions (and perhaps biases and misunderstandings!) since I have your ears and these were the sorts of things that came to mind when trying to make a leap from knockout to mobservable so others may have the same thoughts trying to translate concepts.

I think this library is totally awesome!

[mweststrate]

No problem! Saddly I'm a occupied today with other stuff that popped up and
required immediate attention, but I definitely agree that some parts of the
api would benefit from a revamp (and better docs and a nicer website) :)

On Thu, Dec 10, 2015 at 8:11 PM, Kevin Malakoff notifications@github.com
wrote:

I should probably say that I realize some these ideas may be extreme,
based on a different mental model than you have in mind, based on poor
understanding of all the use cases or ES6 language features, etc so
definitely, don't worry about saying you are not going to take your library
in any of these directions...

I just wanted to share my impressions (and perhaps biases and
misunderstandings!) since I have your ears and these were the sorts of
things that came to mind when trying to make a leap from knockout to
mobservable...I think this library is totally awesome!

—
Reply to this email directly or view it on GitHub
#67 (comment)
.

[kmalakoff]

I've been able to update my application (with the array optimizations) and it is working great! This came right at the perfect time.

One thing I noticed on the API, I wanted to pass arguments to m.createTransformer (m.createTransformer((node, filteredChildren) => {}), but got the [mobservable] transformer parameter should be a function that accepts one argument error.

I understand why you would add this warning so only one thing is transformed. To work around this, I just put the arguments on the node itself (it was a filtered version of the children needed for tree manipulation availability), but maybe the arguments should be passed through since in my first time using it, I ran into the need for arguments?

[kmalakoff]

@mweststrate I've been iterating on more filtering and sorting cases, and I'm having a little bit of trouble doing what I need to. It is sort of related to the above (where I wanted to pass arguments to the transformer)...

The problem comes from the fact that sometimes the parent's visibility is determined by it's children and that the children are dynamic based on the filters / search criteria. So first I'm creating the children recursively and then finally the parent.

Here's the pseudocode:

  transformNode(node) {
    let transformedChildren = node.children.map(transformNode).filter(x=>!!x).sort(byName);

    node.transformedChildren = transformedChildren; // the above workaround to pass an argument
    return transformedChildren.length || nodeIsVisible(node) ? return nodeTransform(node) : null // this is the transformer
  }

The first time around, it works, but then if I change a tag or search criteria, the children or node itself may no longer be visible. If I could patch the children in subsequent transformation runs, I could work around it:

  transformNode(node) {
    let transformedChildren = node.children.map(transformNode).filter(x=>!!x).sort(byName);

    node.transformedChildren = transformedChildren; // the above workaround to pass an argument
    let transformedNode = transformedChildren.length || nodeIsVisible(node) ? return nodeTransform(node) : null // this is the transformer
    transformedNode.children = transformedChildren; // CYCLE!
  }

but there is a cycle. So I tried another way which is to make the children of the transformed node a computed / view, but then there is a cycle on the array of transformed nodes since the children cannot looked up until after a full pass through generating all of the transformed nodes (the nodes are created in the middle of the transformation cycle).

I'm going to keep trying to find a way to break the cycles, but if you have any suggestions, please let me know!

Update I ended up finding a work around, but because it isn't as simple of a use case as the tests given the dependencies up and down the tree to determine visibility / hierarchy, I wasn't able to rely on transformations to build the tree, but on the nodes. I ended up using transformations to just maintain a list of displayNodes and used the sort of hacks that I've been trying to avoid (the immediates, untrackeds, a variable on the store telling me that it is mid-processing, etc):

class DisplayNode {
  @observable children =[];
  @observable requeuedAt = null;

  constructor() {

    autorun(() => {
       if (store.inUpdate) return immediate(() => this.requeuedAt = new Date());

       // lookup children
       let allDisplayNodes = store.displayNodes;
       let children = this.node.children.map(c=>find(allDisplayNodes, x=>x.node ===c)).filter(x=>!!x)

       immediate(() => untracked(() => this.children.replace(children)));
    });
  }
}

Maybe there is a more elegant way to do this? I feel like you mentioned earlier about doing things fully reactive is probably the solution, but I can't see the way to do it like that at the moment.

[mweststrate]

Hi @kmalakoff,

I won't have much time until tomorrow evening, so here a short answer. Didn't look into your example in detail yet, but from the description it seems that you try to achieve exactly the same as I did in this test: https://github.com/mweststrate/mobservable/blob/feature/transform/test/perf/transform-perf.js#L38,

is that right? Would the approach from that setup work?

Good point about the additional arguments. Maybe those additonal arguments should be stored then in the memoization cached and passed into the function on subsequent calls (until the same transform is called with other arguments)? I think that would work fine, I just have to make sure it isn't confusing in some way.

[kmalakoff]

@mweststrate thank you for pointing me to this. I woke up with the same thought...the children should be transformed in the display node themselves rather than in the store and pass in. I'm going to try refactoring again! Can't wait until I nail transformation since it seems like such good way to do things.

[kmalakoff]

@mweststrate these transformations are working great for reducing my reliance on the "autorun and setting observable property values" anti-pattern, and managing life cycles (I'm using observable getters and transformations to ensure destroy is called on dynamically created class instances).

It is very satisfying to remove all of those hacks and to experiment with lifecycle-aware computed properties!

I'm still trying to wrap my head around the transformation API though, but will report back when I've narrowed it down, but I'm finding destroy is being called when I'm not expecting it.

Is there a constraint where one node can only have one transformation on it at a time? I've started running multiple transformations on the same nodes which could explain the behavior I'm seeing, but not totally sure. Need to investigate further.

[kmalakoff]

I haven't yet looked into the problem above. I spent most of the day tracking down other bugs, but will try to write tests for the above.

I still have a bug which might be related to the @observable get filteredProjects() naive approach above as I am a bit new to decorators and ES6 get / set syntax, but as a work around, I went back to the autosync approach.

One thing that I did need to do was wrap a transform in an untracked because it was interfering somehow with the dependency tracking (without wrapping the transformer in untracked, other dependencies in the autorun no longer triggered / seemed registered):

    this.autorun(() => {
      let children;
      untracked(() => {
        children = this.node.children.map(this.nodeTransformModuleToDisplay);
      });
      ...
    });

I'm not sure if anything comes to mind...

By the way, it is working "well enough" for me for now (with some work arounds like these) to move back to feature development! I'll try to interleave writing tests and reporting bugs over the coming week so no urgency to respond to me.

I really appreciate the time and effort you have put into all of this. A huge thank you!

If you want to discuss any of my usability feedback or bounce any ideas off of me, feel free use the email address on my Github profile or to discuss on a Github issue.

[mweststrate]

@kmalakoff "Don't use autorun to transform state into state" or "Automatically updating application state is an anti-pattern. Derive data instead" should be really mantra's. Its often how we programmers tend to think :) I think I should blog a bit about that.

Funny enough I exactly run in to the same issue a few days ago, autorun should always be untracked. Thinking about that, I figured that autorun should always run after all other derivations have completed and in complete isolation of any current computations. So I'm refactoring / rewritting / optimizing a bit while I'm adding that (scope creep, definitely). So I'm really interested to see whether, when I've finished that, it makes a difference for your use case.

Any way, thanks a lot for your enthusiasm, patience and good suggestions!

[mweststrate]

Just rewrote history state tracking in the mobservable-reactive2015-demo repo using createTransformer. The performance gains (both in mem and cpu) are massive due to the structural sharing and the composed derivations when having 10.000+ items:

mobxjs/mobx-reactive2015-demo@dea834a

[kmalakoff]

That's awesome to hear!

I wonder if a comparison of before and after (eg. either just a migration or a migration plus some performance numbers) plus an explanation of the reasons why would be a good addition to the upcoming docs or for a blob post with the release of 1.2.0?

[mweststrate]

I'm thinking about writing a blog post about createTransformer indeed. Numbers are a bit hard to give in general as using createTransformer is in many cases an order of magnitude faster than keeping graphs in sync with each other manually, unless you hugely complicate your own code by optimizing every possible mutation path.

For the reactive demo the minimum frame rate quadrupled and the cost of serializing the state went from 9% to 2% when drawing 10.000 boxes and arrows.

Anyway, the feature has been published as 1.2.0 and is documented here: http://mweststrate.github.io/mobservable/refguide/create-transformer.html
(feel free to amend)