Tracking of user perceived response times

[vlsi]

I'm working on a performance of a web-based enterprise application and typical problem is to measure the time it takes to process button clicks.

The initial thought was "ok, I need to measure UI performance, then I use boomerang".
As I applied boomerang, it turned out to be something unrelated to performance validation.
None of the user clicks/actions result in just a page reload.

The issues (see below) can be solved to a certain degree with plugins, however it looks like blocking all the default boomerang beaconing behavior and rolling a new one.
I've been using boomerang.js for 2 months for measuring the response times, however I had to roll my own set of plugins.

Can you please clarify if you feel boomerang will/need/should support tracking of user perceived response times?

I understand that hosted applications and open-to-internet applications might have different requirements (e.g. in hosted enterprise app you have access.logs, thus you do not need resource timing beacons from client). However, it looks like current boomerang approach makes little sense for enterprise apps.

Here is the list of issues I ran into as I tried boomerang:

1. Boomerang is page-level based. For instance, it "does not work" for single-page applications when the page is loaded just once, and further actions are performed without page reloads.
   I do not want per-ajax tracking (RT.startTimer/done). I want per end-user action tracking.
2. Boomerang does not glue multiple page transitions under a single "user activity". For instance, a single button click might perform some ajax calls, and then perform redirect to another page. Boomerang captures this redirect as a "new page".
3. Boomerang does not support frames/iframes. E.g. a click in main window might result in children iframe being created. I expect to get overall timing for the whole activity.
4. Boomerang does not track/capture page rendering activities. Big pages often take much time to process at javascript/css/html paints. If your SLAs are "4 seconds for full page render", you need take rendering into consideration and measure it.
5. Boomerang does not figure out the action user performs (e.g. the name of the button that was clicked, etc)

[bluesmoon]

1. boomerang does actually work for single page apps. You use the requestStart and responseEnd methods to time in-page user events.

For example, when the action is initiated by the user, you would call this:

if (typeof BOOMR !== "undefined") {
    var timer = BOOMR.requestStart("some-tag-name");
}

Then, when the action is completed, you will call this code:

if (timer) {
    timer.loaded();
}

which will cause a beacon to be fired.

2. you're right that the default version of boomerang does not do this, however we've been experimenting for some time now with tying actions to a session and if there's interest, I can merge this into the primary product.

3. If the child iframe is also instrumented with boomerang, then should get a separate beacon from the iframe. If not, then you could still use the API as mentioned in (1) to measure from click to iframe.onload/onerror (although I've had mixed success with whether an iframe's onload event will fire or not, and when).

4. This is true. There's no way to tell from JavaScript (or even from within the browser) when everything has been drawn to screen. What generally happens is that the browser will execute JavaScript, and send rendering information to the GPU (or video memory), and at that point it's a black box. There is no feedback on when those bytes make it to the display device. The best that we can do is measure when JavaScript has finished executing, which again means instrumenting using the API mentioned in (1) since this would be different for different sites.

5. I can see how this would be useful, and it's doable via the API, but I think it could be made better. Perhaps if we capture clicks on buttons as well and automatically add meta data about that button to the beacon.

One question I do have is, are you okay with firing a beacon on every action (this is more reliable), or would you prefer if beacons were batched and sent periodically (this is likely to fail if the user closes the page).

[vlsi]

1. boomerang does actually work for single page apps.
   For example, when the action is initiated by the user, you would call this:

1.1) I am not sure this is often the case. I expect most of the applications are written without boomerang in mind. In other words, if you "just add boomerang" to the existing app, it does not "just work".

You know, there are some signs of notion of the url of the link / form. However it is very limited and it is not extensible.

Then, when the action is completed, you will call this code:

1.2) Sometimes the action is completed in a different module. For instance, "shop" button in a Customer Summary module starts the interation, and the interaction completes in the Shopping module. You might have hard time passing boomr variables between development teams.

however we've been experimenting for some time now with tying actions to a session and if there's interest, I can merge this into the primary product.

I am going to refine my set of plugins (I do not support multiframe pages very well yet), and I do not find requestStart API, RT plugin, etc boomerang API very useful for that purpose.

I am interested if there is some generic solution. For now I have nothing to contribute to RT plugin as it just does not suit me.

3. If the child iframe is also instrumented with boomerang, then should get a separate beacon from the iframe. If not, then you could still use the API as mentioned in (1) to measure from click to iframe.onload/onerror (although I've had mixed success with whether an iframe's onload event will fire or not, and when).

My aim is to have a single beacon for a single user interaction. Sub-beacons are fine (as in "nice to have") provided there is a global notion of the overall event. I do not want to develop extended log processing tools that will use artificial intelligence to deduce which beacons should be combined to get the response time of a "shop" button. I just want to see it as a single entry.

4. Boomerang does not track/capture page rendering activities.

5. This is true. There's no way to tell from JavaScript (or even from within the browser) when everything has been drawn to screen.

You tell a bit different story. I mean: boomerang could do much better job for capturing paint/document mutation events. MutationObserver, requestAnimationFrame, and setTimeout can detect document mutations (and paintings) quite well.

The best that we can do is measure when JavaScript has finished executing

This is false. See above.

5. and it's doable via the API, but I think it could be made better.

The extensibility part of RT plugin is a bit poor.

One question I do have is, are you okay with firing a beacon on every action (this is more reliable), or would you prefer if beacons were batched and sent periodically (this is likely to fail if the user closes the page).

Currently I fire a beacon for each click. The clickstream is not high as I am running the load tests: real browsers, Selenium, limited set of load boxes.
Beacon batching is "nice to have", however I believe the primary goal is to capture the required information. The batching can be added afterwards.

sent periodically (this is likely to fail if the user closes the page)

For instance, when performing a load test, you control the environment and the load script just does not close the window unexpectedly.

localStorage can be used to buffer beacons, so they get sent after user reconnects.

[andreas-marschke]

1.1) I am not sure this is often the case. I expect most of the applications are written without
boomerang in mind. In other words, if you "just add boomerang" to the existing app, it does not "just
work".

You know, there are some signs of notion of the url of the link / form. However it is very limited and it
is not extensible.

Boomerang (in my opinion) has been kept and designed in a general purpose way in order to meet the needs of a broader user base. Of course if you want to do more involved things with boomerang you will have to put effort into it. If you are planning to implement metrics and analysis into a internal tool on an enterprise-scale, yes you will have to reach consensus across teams to get where you want to go.

Vanilla Boomerang is not a stop-gap measure or drop-in replacement for Google-Analytics. You will need to integrate it properly if you want to have a enhanced look into your application.

1.2) Sometimes the action is completed in a different module. For instance, "shop" button in a Customer Summary module starts the interation, and the interaction completes in the Shopping module. You might have hard time passing boomr variables between development teams.

This again comes down to team-collaboration and communication across teams. If you are from the engineering or analytics/metrics team and want to have data from a specific teams module you will have to communicate that.

However if we're talking about an SPA it should not be as problematic as one might think as most of Boomerang and here specifically the BOOMR Object is hooked into the window object of the browser.

This way accessing variables,subscriptions,events,functions of the BOOMR Object can be accessed by any of the modules.

Functionality to measure specific non-DOM-Element related events is not yet implemented in boomerang but I'd like to suggest this as a new feature for the future:

@bluesmoon How about letting users add subscribable events to boomerang using BOOMR.utils?
i.e:

BOOMR.utils.addCustomEvent({
    "object_change" : function(data) {
        console.log("Event triggered: ", data);
    }
});

Where addCustomEvent() would append "object_change" to impl.events and push the function onto the stack of events subscribed.

That way more subscriptions can be added with:

BOOMR.subscribe("object_change",function() { /* foo bar baz */});

If you want I can see if I can implement this?

You tell a bit different story. I mean: boomerang could do much better job for capturing
paint/document mutation events. MutationObserver, requestAnimationFrame, and setTimeout can
detect document mutations (and paintings) quite well.
...
My aim is to have a single beacon for a single user interaction. Sub-beacons are fine (as in "nice to have") provided there is a global notion of the overall event. I do not want to develop extended log processing tools that will use artificial intelligence to deduce which beacons should be combined to get the response time of a "shop" button. I just want to see it as a single entry.

I'll have to agree that the wording of the boomerang documentation isn't necessarily the best in that regard. Boomerang is (by design and mechanics) mostly focused on the timing aspects from the initial GET Request of a page by a browser to the point where everything is rendered and the page_ready event is fired. Everything else in between that is handled by plugins and the like. Boomerang itself is - as mentioned before - NOT a strict replacement for Google-Analytics. And yes one has to create your own data analysis for what boomerang sends back to the server.

Currently I fire a beacon for each click. The clickstream is not high as I am running the load tests: real browsers, Selenium, limited set of load boxes.
Beacon batching is "nice to have", however I believe the primary goal is to capture the required information. The batching can be added afterwards.

Rudimentary click-tracking has landed in bluesmoon/boomerang.git quite recently. Maybe you can check it out.

For instance, when performing a load test, you control the environment and the load script just does not close the window unexpectedly.
localStorage can be used to buffer beacons, so they get sent after user reconnects.

This goes with the assumption that theres only a limited set of very modern browsers. That too isn't always the case unless you know your environment. Boomerang tries to be as general in this regard as possible and work with most audiences (even those that run IE6)

[nicjansma]

Thanks @vlsi for your feedback. Since there haven't been any additional comments on this issue, we're closing it for now.

Note SOASTA has contributed quite a bit of updates for monitoring SPAs in the spa, angular and other plugins. #68