Service workers missing from 12/15 response bodies

[rviscomi]

For example, this test includes a request for sw.js. It's not in the 12/15 response bodies table, despite the same resource being available in the 12/1 table.

This was originally reported by @jeffposnick in a query similar to this:

SELECT
  DISTINCT url
FROM
  `httparchive.har.2017_12_01_chrome_requests_bodies`
WHERE
  body LIKE '%new self.Workbox%' OR
  body LIKE '%new Workbox%'

This yields 74 results, while a query over the 12_15 table only yields 1 result (the workbox website HTML).

@pmeenan did anything change on the WPT side? This seems like a regression of #113.

[pmeenan]

Looks like it's mostly from Chrome 63 and 2 separate issues.

#1 - 63 started logging some phantom network requests which WPT filtered out but that also filtered out the SW requests (only from dev tools, the netlog had them as well so the requests were still logged but the request ID from devtools was lost and that is used to match the response body. I fixed that which should help.

#2 - We aren't reliably seeing the request for the worker js in dev tools. With the change to plznavigate in 63 it may have broken something there (re-opened the devtools issue and they are looking into it).

[mfalken]

Just adding a "Me too!" comment... I work on Chrome's SW implementation and just now tried to use HTTPArchive to investigate how SWs are used in the wild. Would be good to have this working again. On the DevTools issue it looks like there was some clue about off-main-thread-networking?

[pmeenan]

I'm planning to add a safety net to re-fetch any missing bodies (using the same request headers). That should fill any missing bodies from dev tools.

…

________________________________ From: Matt Falkenhagen <notifications@github.com> Sent: Wednesday, January 31, 2018 8:09:21 PM To: HTTPArchive/httparchive Cc: Patrick Meenan; Mention Subject: Re: [HTTPArchive/httparchive] Service workers missing from 12/15 response bodies (#141) Just adding a "Me too!" comment... I work on Chrome's SW implementation and just now tried to use HTTPArchive to investigate how SWs are used in the wild. Would be good to have this working again. On the DevTools issue it looks like there was some clue about off-main-thread-networking? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#141 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAbHBa1L-1eGsoMxvQCBMpmGJV22J0wPks5tQQ7BgaJpZM4RSV8j>.

[pmeenan]

This should be fixed now. WebPageTest will backfill any missing response bodies that it couldn't get from the browser by re-fetching them. Not elegant but works. I'll leave it open until we can verify with the Feb 1 crawl (it will be missing for the 1/2 day or so that has been running so far but that's less than 10% of the crawl)

[mfalken]

Thank you!

[rviscomi]

SELECT
  DISTINCT url
FROM
  `httparchive.har.2018_02_01_chrome_requests_bodies`
WHERE
  body LIKE '%new self.Workbox%' OR
  body LIKE '%new Workbox%'

Testing against the Feb 1 crawl shows 103 results, so it seems like this has been fixed. Thanks @pmeenan!

@jeffposnick feel free to reopen if you're still seeing issues.

[jeffposnick]

Looks good! Thanks!