Lazy loading of resource set items for browser based tests

[augustl]

The current process is to serialize the entire resource set to JSON from the buster-test end, and ship it to the buster-server over HTTP.

This issue tracks the progress of rewriting Buster (specifically, ramp and ramp-resources) to have a lazy approach instead.

The downsides of the current approach is:

• We have to read the entire file from disk, encode it into a base64 string so it can be further encoded into JSON, and deserialize on the server side. If we load lazily, the best case is that the buster-server reads the file directly on-demand if buster-test and buster-server runs on the same machine. The worst case is to load the file on-demand from buster-test to buster-server directly over a TCP connection, with no encoding and decoding.
• Loading more files than actually needed when using buster-amd. For example, in a hybrid node and browser project you aren't requiring the node files with AMD, so serializing and deserializing the node files is unneeded work. This can also be a problem if your node files are causing syntax errors in our syntax error checker, such as node.js files with a shebang. See Config: syntax error from "sources" with hash-bang #362

The downsides of a lazy approach is:

• Fail late. We won't be able to do things like syntax checking and compilation (coffeescript, ...) until the browser actually requests the file, so we have to start the test run before running the validations.

[augustl]

This will probably require a 2.0 release of ramp and ramp-resources. I don't see why we would keep the old serialization API, both buster-static and buster-server can use the imagined lazy streaming API.

It makes sense to keep the API surface of ramp resources mostly intact for the things various extensions uses it for. I want to make it immutable, so some minor changes may be required. A mutable builder API could help mitigate this, so the only change you need to make is to finalize the builder at the end.

[GCheung55]

Lazy-loading of resources could also affect the buster-istanbul extension, which processes each resource that is sent to the browser to capture code coverage.

[augustl]

Good point!

The idea is to still allow for a content function that does whatever. But instead of invoking this and serializing every resource in advance, buster-server asks buster-test for the files it actually loads, on-demand. That way, the function will run just like before, inside the buster-test process, so it should work just fine.

[dkl-ppi]

@augustl, is the issue #399 related to this issue?