Benchmark requirements

[kyliau]

Brainstorm general requirements for all benchmarks (this is intended to be a discussion for now, no decision has been made yet).

1. Build step
   A build step is probably OK, in particular if using open-source toolchains. Many of today's frameworks require or encourage one.
   Rationale: Our options are very limited if we don't allow a build step. Almost all meta frameworks like Next.js, Angular, and Nuxt provide an official way to build their projects, and often it's the only way to optimize & bundle applications built with those frameworks.
   In Grand Prix, the final output of the build is checked into the repository, but we should check in the sources (package.json, script / command to build the project, lock file to pin dependencies) too. We should probably not check in the node_modules, as is currently done in GrandPrix.
   Our options are: source only (build during CI / release), source + output, or output only (what's in the repo is what gets deployed)
   I vote for source + output since checking in the source allows everyone to reproduce the build artifacts locally, and having output in the source control makes it easy to inspect the code we deploy.

2. Dedicated server for benchmark
   A dedicated server for just one workload is likely not an option.
   Rationale: If each benchmark requires its own server, then we have to maintain and deploy multiple servers. Perhaps one server to host multiple applications is ok? Server runtime is mostly likely gonna be NodeJS, given how ubiquitous it is with popular frameworks. For example, features like incremental static regeneration (ISR), or image optimization at request time in Next.js requires a server.

3. Third-party API / services
   At least for now, we probably don't want to rely on any third-party services or REST APIs for the benchmarks as this can influence determinism/score in different ways. Benchmarks should be as hermetic as possible, and most if not all dependencies should be under our direct control.

4. Third-party resources
   As an extension to requirement (3), is it ok to host resources like scripts, images, fonts on third-party origins like CDNs?
   On the one hand, we like to have as much control as possible over the benchmarks. But on the other hand, loading resources from CDNs is more representative of the real world use case, particularly for ubiquitous resources like google fonts, google analytics, twitter snippets, facebook button, etc. It's important to note that some CDNs do more than serving static content. Cloudinary for example optimizes image on the fly, serves the appropriate image based on device screen size, etc.

[rniwa]

I agree we should check in both source & output. Checking in all dependencies is important because we have a process to review each & every library / framework being used for licensing purposes. Without a specific version of each module being checked in, this validation becomes harder.

I don't think we want to load google analytics, etc... into our benchmark regardless of how ubiquitous their use might be. Nor do I think we want to use third party CDN even if they're commonly used. In particular, we have needs to be able to run everything locally in our lab environment (i.e. no internet connection).

[addyosmani]

Thanks for continuing our early discussion on requirements here, Keen! I unsurprisingly agree with 1-3 :)

I don't think we want to load google analytics, etc... into our benchmark regardless of how ubiquitous their use might be. Nor do I think we want to use third party CDN even if they're commonly used. In particular, we have needs to be able to run everything locally in our lab environment (i.e. no internet connection).

I concur with @rniwa. One of the nice benefits of not relying on third-party hosts for web benchmarking resources is that we can do hermetic testing in an environment that isn't influenced by external factors (network connection latency, bandwidth, server load etc). This is quite important for helping us reduce noise.

[rniwa]

Right. Just having a network connection open introduces a lot of noise in a benchmark because the kernel still has to service whatever packets the device receives.

I also don't think we want to scope creep Speedometer too much. Things like page load time are better measured with a more tailored test suite. In particular, actual page time is highly sensitive to things like SSL handshakes, the number of TCP connections, file cache strategies, speculative prefetching / DNS resolution strategies, etc... yet I don't think we want to emulate these things in this benchmark.

In fact, I suggest we categorically exclude these metrics as out of scope for this benchmark in order to achieve a high signal to noise ratio in the core purpose of this benchmark: measuring web app responsiveness once loading has finished.

There are some gray zones like HTML, CSS, JS parsing time. We never tried to measure these in speedometer v1 or v2 but we could consider trying to measure them in v3 if we're going to serve content from a HTTP/HTTPS server.

[bgrins]

Agreed on no third party runtime dependencies - if there are cases where we needed to simulate this I could imagine saving the network response as a file in the benchmark and fetching it directly.

Dedicated server for benchmark
Rationale: If each benchmark requires its own server, then we have to maintain and deploy multiple servers. Perhaps one server to host multiple applications is ok? Server runtime is mostly likely gonna be NodeJS, given how ubiquitous it is with popular frameworks. For example, features like incremental static regeneration (ISR), or image optimization at request time in Next.js requires a server.

I want to dig into this requirement a bit more. I wonder if it'll be hard to control for different environments (each browser's CI system, any external perf test tooling, a webpage based runner) to make sure that Node itself and any dependent frameworks are running in a consistent way that don't affect test results too much. Is it important for these use cases that we're actually running i.e. Next.js live during the run (as opposed to generating responses and delivering via plain files or from a Service Worker similar to what I mention above for third party resources)?

[rniwa]

I wonder if it'll be hard to control for different environments (each browser's CI system, any external perf test tooling, a webpage based runner) to make sure that Node itself and any dependent frameworks are running in a consistent way that don't affect test results too much.

Yeah, I'd prefer if we had completely static content like v1 & v2. That would certainly make things easier on our end.

[kyliau]

SGTM, having static-only content would make the tests hermetic and eliminate potential bias due to unaccounted factors. @bgrins idea of recording and replaying responses either via plain files or Service Worker sounds interesting. I don't see a workload that requires this for now, but we could revisit this idea when/if the need arises.

[rniwa]

I think this has been largely answered by the group at this point? Is there anything else left to discuss?

[bgrins]

I think this can be closed. There's been some good discussion here that we can refer back to but I don't think there's any action needed for this issue.