investigate network events without timing data

[vigneshshanmugam]

Some of the network events are not having the timing data and have them as null. One of the immediate thing to notice is that these requests are not receiving any loadingFinished or loadingFailed event.

The UI currently shows empty bar without any data on this request. Need more investigation on the agent side, to figure out what is happening.

await page.goto('https://www.mlb.com/dodgers');

[dominiqueclarke]

Specifically for inflight requests that have no timing data at all (either all the timing data, including total is -1 or the timings key is entirely omitted) this is the user experience on hover. There is no bar and there is no tooltip, because there is no data to display in the tooltip. (See the bottom item)

[vigneshshanmugam]

The issue happens mostly because our runner waits only till the load event which typically means all of our inflight requests ( tracking requests mostly will be cut off as they are sent only after load event in lots of cases). We would loose out on all of the requests as our browser session would be closed and we will never receive finished events.

Solution 1

1. Educate the users that this is a common scenario and provide some examples on how users can wait for all events in a single step which effectively increases step duration and other consequences based on that.

await driver.page.waitForLoadState('networkidle');

2. In the UI, we create a new colour code for the inflight/dropped requests and add draw them with startTime - till the record endtime.

Cons:

• For a heavy page like https://www.mlb.com/dodgers, user would need to be aware of this issue and would not be obvious from the start.
• Navigation timeout would happen most of the time for heavy pages which results in failed run.

Solution 2

We already talked a bit about introducing timeout for journeys and steps which is a great feature in most of the test runners #133

1. We expand on that scope and allow a configurable navigation timeout option to consider if it's present or we use our previous load event as our signal.

npx elastic-synthetics suites --timeout 60s

2. The above timeout would replace the existing playwright default timeout of 30s and get applied for all navigations inside journeys and steps.

The advantage with this pattern is that, we don't need to overwhelm users with number of possibilities when it comes to capturing more requests in the waterfall.

[hmdhk]

I would add another option to those.

Solution 3

We can add a default waiting time for network events and capture as many events as possible but add a post-processing phase once the journey is finished to adjust the end time of each step (or mark network events as truncated). This post-processing is done in the agent and it can include both the time that we recorded for the end of the step as well as network events that were started during that step, the following pseudocode can be used:

• For each step find the network events that were started during the step i.e. requestSentTime > step.start && requestSentTime < step.end (note that this doesn't include the currentStep property)
• Adjust the step end to the maximum of step.end and the maximum end time of the network events from previous step
• Alternatively, we can mark network events that end after the step.end as "truncated" and show them in the step they were started in up to the step they end in (according to their timing). APM UI also has a design for truncated spans that can be used here (with triangle shape at the end of the bar chart)