Usage of serviceworkers to cache resources

[SvanteRichter]

I posted this as a comment in #1217 but I figured it deserved its own issue.

I was looking at the size comparisons and it seems like anansi uses a service-worker to precache most of the app.

This makes it seem like the full size of it is just the ~1kb for the loader that loads the serviceworker, but on a fresh load there is another ~250kb loaded.

Here is a comparison of the startup metrics with the serviceworker and without:

It seems like anansi is the only testsuite using serviceworkers, at least in code within this repo.

I'm not sure if there is any guidelines on how to use serviceworkers in this test, but if it is not measured properly (i.e the full, non-cached load) then the size comparison becomes pretty useless since every framework will be reporting a size for something like navigator.serviceWorker.register('sw.js'); and no framework or app code.

I'm a bit biased since I care about the size metrics for my own framework, but my suggestion is either clearing the serviceworkers before each test run, disallow using serviceworkers or disabling serviceworkers in the tests entirely.

[SvanteRichter]

Feedback/thoughts from @saru-tora would be very welcome!

[SvanteRichter]

@krausest Any thoughts on this? In my opinion it basically invalidates the "total kilobyte weight" metric, but if it is desired then a readme note similar to "preload the glyphicon" might be useful.

[krausest]

I had hoped I fixed it accidentally by running lighthouse only once anymore, but it still reports 142 KBs.
Actually that's strange since chrome is started with caching disabled and I can see that lighthouse requests
anansi/js-framework-wasm/sw.js,
anansi/js-framework-wasm/pkg/js_framework_wasm.js and
anansi/js-framework-wasm/pkg/js_framework_wasm_bg.wasm

But it seems like it doesn't count them. Will investigate...

[krausest]

Hmmmm - I tried if I can log the response sizes in the server (then maybe we could measure both uncompressed and compressed). But I failed both for fastify and for the old express server to do so.
Currently I'm getting:

/keyed/anansi/index.html 2385
/currentStyle.css 88
/keyed/anansi/js-framework-wasm/main.js 1571
/bootstrap/dist/css/bootstrap.min.css 121260
/main.css 364
/bootstrap/dist/fonts/glyphicons-halflings-regular.woff2 18028
/keyed/anansi/js-framework-wasm/sw.js 0
/bootstrap/dist/css/bootstrap.min.css.map 54416
/keyed/anansi/js-framework-wasm/main.js 1571
/keyed/anansi/js-framework-wasm/pkg/js_framework_wasm.js 21879
/keyed/anansi/js-framework-wasm/pkg/js_framework_wasm_bg.wasm 235709
/favicon.ico 150

I have no idea why the service worker returns a size of 0 (with express-response-size). But I get the same effect for fastify with the onSend hook (Response is an empty string though it's actually 1739 bytes).
Edit: Maybe there‘s something I didn‘t look at: I should check the status codes and report back. Maybe sw.js has 304. I‘ll check tomorrow.
@nakrovati Any ideas?

[krausest]

Yes, it was indeed a 304. Seems like chrome caches the service worker even when Application > Storage > Clear site data is invoked.

I tried three approaches to calculate the size of the frameworks:

1. Use performance.getEntriesByType('resource'): I couldn't get any information for WASM files this way
2. Use tracing in chrome. I didn't find any information about the size in the chrome trace (I enabled all categories from https://chromium.googlesource.com/chromium/src/+/main/base/trace_event/builtin_categories.h, but didn't find anything).
3. Let the server measure the ressources. I hacked some compression into the fastify server. The challenge was that I also wanted to measure the compressed size (which I didn't get working with the compress module). So I hard coded a brotli compression.

Please let me know if you have any suggestions about step 1+2.

The results of step 3 looks like that:

onSend: /frameworks/keyed/anansi/index.html as stream with uncompressed size 2385 compressed 498 sum uncompressed 2385, compressed 498
onSend: /frameworks/keyed/anansi/js-framework-wasm/main.js as stream with uncompressed size 1571 compressed 511 sum uncompressed 3956, compressed 1009
onSend: /frameworks/keyed/anansi/js-framework-wasm/sw.js as stream with uncompressed size 1739 compressed 494 sum uncompressed 5695, compressed 1503
onSend: /frameworks/keyed/anansi/js-framework-wasm/pkg/js_framework_wasm.js as stream with uncompressed size 21879 compressed 3835 sum uncompressed 27574, compressed 5338
onSend: /frameworks/keyed/anansi/js-framework-wasm/pkg/js_framework_wasm_bg.wasm as stream with uncompressed size 235709 compressed 69909 sum uncompressed 263283, compressed 75247

I decided to filter out all /css resources. The size is smaller than the one reported by chrome, since the headers are missing. I tried to emulate the headers, but some headers (like Date) are not present so it didn't match exactly either.

There's a new callback http://localhost:8080/getSize that returns {"size_uncompressed":263283,"size_compressed":75247}

The size is reset whenever index.html is called. The idea is to fetch this information from the benchmark driver and report both data in the results table. This would also be a good solution for #1377, #1041

The fastify server code suffered from this feature. I only got it working with a hack in a onSend-hook.

Looking forward to hearing you opinions! Does that sound like a reasonable approach?

[SvanteRichter]

That sounds like a nice solution, especially since it also solves #1377 and #1041! My initial idea was to just disallow service-workers, but this seems more reasonable.

[krausest]

Here's a first preview. I'm looking forward to getting feedback.
https://krausest.github.io/js-framework-benchmark/2023/table_chrome_119_sizes.html

The numbers are in kBs (I'll add that info to the table). If you want to check here's how to do:

• The numbers should equal the size of all files except the css directory (including serviceworkers etc.) without http-headers. It should equal the content-length of those files.
• If compression is enabled via http://localhost:8080/enableCompression the server prints size information for both brotli compression and uncompressed transfers. Use /disableCompression to disable the measurement.

Anything surprising or something that sounds fishy in the table?

[SvanteRichter]

Thanks! misojs looks a bit weird when comparing the total kilobyte weight with the new numbers, any idea as to why?

I'd also think that if the total kilobyte weight metric is kept around it'd be good for some clarification as to what it is meant to represent, since it is neither a completely cold load or a load with a warmed cache.

[krausest]

Here's the output for misojs:

onSend: /frameworks/keyed/misojs//index.html as stream with uncompressed size 440 (not compressed since below threshold) sum uncompressed 440, compressed 440
onSend: /frameworks/keyed/misojs//delegate.js as stream with uncompressed size 4130 compressed 1244 sum uncompressed 4570, compressed 1684
onSend: /frameworks/keyed/misojs//diff.js as stream with uncompressed size 12814 compressed 2835 sum uncompressed 17384, compressed 4519
onSend: /frameworks/keyed/misojs//main.js as stream with uncompressed size 6922 compressed 1957 sum uncompressed 24306, compressed 6476

When comparing with the network tab this looks ok. I have no idea why lighthouse doesn't show that difference. I plan to remove the total kilobytes from the table.
Maybe I'll even completely remove lighthouse. It always stood out in that I can't say anything about why the numbers are what they are.

[krausest]

The proposed solution didn't get negative feedback so I'm closing this issue.