Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queues for error-free and traceable code #3507

Closed
justinbmeyer opened this Issue Aug 26, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@justinbmeyer
Copy link
Contributor

commented Aug 26, 2017

tldr; We propose a queueing system that will:

  • make it easier to understand how your application works and CanJS works
  • make it easier to integrate with other technologies (like streams)
  • fix several difficult CanJS bugs

This was discussed at a contributors meeting (2:20)

The problems and solutions

The following 3 sections discuss problems with CanJS and how Queues will address them:

Understanding what caused what

Currently, it can be difficult to understand how your application behaves, especially with respect to what mutations caused what to happen. For example:

var map = new DefineMap({first: "Justin", last: "Meyer", fullName: ""})
var fullName = compute(() => {  return map.first + map.last })
fullName.on("change", function(ev, newValue){
  map.fullName = newValue;
});
map.on("fullName", function(){
  //  debugger
});
map.first = "Ramiya"

How can we tell that the setting of map.fullName was ultimately caused by setting
map.first? The answer - A traceable queuing system!

If someone uncommented the debugger and ran require("can-queues").logStack(), it will print the known tasks that ran up to the current task:

MUTATE map:1 dispatching 'fullName' event
MUTATE compute:2 dispatching 'change' event
DERIVE compute:2 updating value to 'fullName'
NOTIFY map:1 dispatching 'first' event

NOTE: this will only work in development. In dev, tasks will have a name, and a reference to their proceeding task.

Better/Easiser Integration

Currently, integration is "sloppy" with 3rd party libraries. For example, a Kefir stream, to dispatch handlers registered can.onKeyValue has to queue the dispatching like this:

handlers.forEach(function(handler){
  canBatch.queue([handler, stream, [value]]);
});

But for some reason (we know the reason), can-observation doesn't do queue handlers in a batch.

this.handlers.forEach(function(handler){
   handler.call(this.compute, newValue);
}, this);

Having a well defined queue system will make it "well known" how to create observables that dispatch their event handlers in the right way. This will make it easier to integrate new functionality correctly.

Fewer bugs

Having a well defined queue system will also make it less likely we will create bugs. Because the queuing system has been poorly defined, it's easy to not dispatch handlers in the right way. If handlers are not called in the right way, bad bugs can happen. Things like computes not updating, or updating too many times, or updating at the wrong times.

There are several issues this will help with:

Technical Specifics

At its core, we need different task queues that are used to run callback functions (event handlers) at appropriate times.

The queues should be:

  • notify - tasks that notify objects that derive a value from another value
  • derive - tasks that derive a value from other values (can-observation)
  • dom - tasks that update the DOM from other values (used by can-view-live to ensure it happens after state has settled, before USER sees these events)
  • mutate - tasks that may mutate other values (user event handlers)

The derive queue should be prioritize-able.

If anything gets added to the notify queue, other queues are paused while the notify queue is flushed.

Example

Lets see a rough example of how callbacks are queued up. The following sets up relationships between computes:

grandChild1 = compute()
grandChild2 = compute()

derivedChild = compute( grandChild1 + grandChild2 )

writableChild = compute()
grandChild1.on( grandChild1Handler() => { writableChild(NEW VALUE)  } )

root = compute( derivedChild + writableChild )

root.on( rootHandler )
writableChild.on( writableChildHandler )

This can be represented as follows:

cursor_and_batching_pptx

If the following mutations happen:

batch.start()
grandChild1( NEW_VALUE )
grandChild2( NEW_VALUE )
batch.stop();

The following handlers will be queued:

Notify Derive Mutate
grandChild1 callback grandChild1Handler - b#1
grandChild2 callback

When batch.stop() is called, the Notify handlers will begin to be executed.

When grandChild1's notify callbacks are run, this will add derivedChild's update to the Derive queue:

Notify Derive Mutate
grandChild2 callback derivedChild update grandChild1Handler - b#1

grandChild2 callback will be called, which will do nothing because derivedChild is already queued. Then derivedChild's update will be run, which will queue the update root:

Notify Derive Mutate
root update grandChild1Handler - b#1

The root update will add the rootHandler:

Notify Derive Mutate
grandChild1Handler - b#1
rootHandler - b#1

Then the grandChild1Handler callback will run as BATCH 1, which will end up adding the Notify and Mutate callbacks. Note that because a new notify is going to be started, writableChildHandler will need to present itself as part of another batch of changes.

Notify Derive Mutate
writableChild rootHandler - b#1
writableChildHandler - b#2

The Notify queue is always run immediately. So writableChild notify is going to be called, which will add a root update:

Notify Derive Mutate
rootUpdate rootHandler - b#1
writableChildHandler - b#2

When rootUpdate runs, it will add another rootHandler to mutate:

Notify Derive Mutate
rootHandler - b#1
writableChildHandler - b#2
rootHandler - b#2

These will all then be run. Notice that rootHandler is going to be called multiple times.

Notes

  • It's possible we don't need a notify queue. When a change happens, all "notify" handlers could probably be immediately fired. It's really derive that is "waiting" to be kicked off by a start/stop.
    • We need to make sure every compute is notified that it should begin re-evaluating before the 1st one re-evaluates. This likely means we need a notify queue to run to completion before the notify queue starts.
  • We should not have any additional notify tasks once the Notify queue is started until the first mutate task is performed. If we DO have a notify task added, something went wrong and we can warn / throw an error.
  • It might be useful to be able to add some tracing here. Perhaps tasks can have more meta information in development, allowing them to track the tasks in other queues that lead to them.

APIs

onKeyValue( key, handler, queue )

I think we should support some type of argument to onKeyValue and onValue that describes when we expect callbacks to be called back:

map.onKeyValue( "name", observation.dependencyUpdated, "notify" )

We might even want to pass the queue as the 3rd argument.

map.onKeyValue( "name", observation.dependencyUpdated, mutationQueue )

can-queue

can-queue would be a basic queing system like the one in can-event/batch

We will need to be able to queue tasks to it like:

mutationQueue.queue( [function, this, arguments ])

And flush the queue to run all of its tasks:

mutationQueue.flush()

We will need the ending of one queue to kick off the start of another queue. For example, once all DOM updates have completed, we will want to begin flushing the mutationQueue.

Also, we'll need some sort of batch number if we want to be backwards compatible.

can-priority-queue

The priority queue will be similar to the updateAndNotify loop in can-observation

It should look like a normal task queue, except:

  1. Tasks needs to be prioritized first on depth then on the order of being added to the queue. This is because we want to reevaluate computes controlling the "root" of the DOM before the children.
  2. We need to be able to remove tasks. This is because we might need to "force" a queued update to run early because some deep child might be updating. (a root compute might be re-calculating but a super deep child of it might need to be run first).

For example, an observation might identify (through Observation.add) to listen to a user's age property and do:

canReflect.onKeyValue(user, "age", this.onDependencyChange,"notify");

In onDependencyChange, it might call:

// removeTask somehow lets you remove this task
var removeTask = derivePriorityQueue.add( [
  this.update, this
], {priority: 0})

When the derivePriorityQueue begins processing, it will run tasks based on their priority. Once complete, it should kick off the DOM queue, and then finally the MUTATION queue.

@justinbmeyer justinbmeyer changed the title Derive, DOM, and Mutate Queues Notify, Derive, DOM, and Mutate Queues Aug 26, 2017

@justinbmeyer

This comment has been minimized.

Copy link
Contributor Author

commented Aug 28, 2017

Like Ember, it would probably be good to have a teardown queue too. This will be for temporarily bound things to unbind themselves.

@justinbmeyer justinbmeyer changed the title Notify, Derive, DOM, and Mutate Queues Queues for error free and traceable code Sep 15, 2017

@justinbmeyer

This comment has been minimized.

Copy link
Contributor Author

commented Sep 15, 2017

@andrejewski pinged me about .log() and supporting filtering. Some thoughts about that:

Say someone had some code similar to what's above, but I've added a fullName2 compute and someone listening on "notify" on the map's fullName:

var map = new DefineMap({first: "Justin", last: "Meyer", fullName: ""}); //map:1

var fullName = compute(() => {  return map.first + map.last }); // compute:2
var fullName2 = compute(() => {  return map.first + map.last });  // extra compute:3

fullName.on("change", function(ev, newValue){
  map.fullName = newValue;
});
fullName2.on("change", function(){}); // listening to extra compute

map.on("fullName", function(){
  queues.logStack()
});
canReflect.onKeyValue(map, "fullName", function(){}, "notify"); // Listen on notify

queues.log();

map.first = "Ramiya"

queues.logStack() would show what tasks lead to the current (or last) task being executed:

MUTATE map:1 dispatching 'fullName' event
MUTATE compute:2 dispatching 'change' event
DERIVE compute:2 updating value to 'Ramiya Meyer'
NOTIFY map:1 dispatching 'first' event

queues.log would show everything running in the order of execution as it happened:

NOTIFY map:1 dispatching 'first' event
DERIVE compute:2 updating value to 'Ramiya Meyer'
DERIVE compute:3 updating value to 'Ramiya Meyer'
MUTATE compute:2 dispatching 'change' event
NOTIFY map:1 dispatching 'fullName' event
MUTATE compute:3 dispatching 'change' event
MUTATE map:1 dispatching 'fullName' event

I think supporting filtering of .log()`, limiting it to a particular object would be great. Two thoughts about this:

  1. Tasks are currently just an array with: [function, this, arguments]. We will have to support some metadata for printing "nice" names. We'll want to include a subject for filtering.
  2. I think we'll want people to be able to stop on a particular task easily, and then call logStack to see what caused a particular task.
@justinbmeyer

This comment has been minimized.

Copy link
Contributor Author

commented Sep 15, 2017

Other notes:

  • queues should maintain the last task to run. This is how we can keep the relationship. Anytime a task is queued, it can look up the last task and remember it as its parent.
  • after the last task of the mutate queue has run, queues should null the last task.

@justinbmeyer justinbmeyer referenced this issue Sep 22, 2017

Closed

Make CanJS easier to debug #3278

6 of 11 tasks complete

@chasenlehara chasenlehara changed the title Queues for error free and traceable code Queues for error-free and traceable code Sep 22, 2017

@matthewp

This comment has been minimized.

Copy link
Contributor

commented Apr 2, 2018

Can this issue be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.