test(gatsby): Add a memory test suite command to the memory benchmark

[imjoshin]

Description

This PR adds a test suite capability to the benchmarking tool we created previously.

This allows us to measure the impact of changes we make to our build process. Notably, it gives us data showing that our efforts in recent memory improvements have a positive impact. In the future, we'll be able to use this to test other improvements.

This CSVs output from these test suites allow us to create graphs like this:

In the future, we may be able to use this to add smoke tests to our CI. 👀

For more information, check out the README.

Examples:

$ yarn test --memory 16g --num-nodes 1000 --node-size 1m

yarn run v1.21.0
$ node scripts/test.js --memory 16g --num-nodes 1000 --node-size 1m
$ ./scripts/docker-get-id

Starting container with 16g memory.
$ DOCKER_MEMORY_LIMIT=16g ./scripts/docker-start
$ ./scripts/docker-get-id

Starting test using container dcdb6681da50.
 - clearing cache
$ docker exec  dcdb6681da50 rm -rf .cache
 - running build with 1000 nodes of size 1m
$ docker exec -e BUILD_NUM_NODES=1000 -e BUILD_STRING_NODE_SIZE=1m -e NUM_KEYS_IN_LARGE_SIZE_OBJ=1 dcdb6681da50 yarn gatsby buildFinished test in 43.81s

Success!

Stopping container dcdb6681da50
$ ./scripts/docker-stop
Done in 49.53s.

$  yarn test-suite
...
Running test #14 for 4g mem, 1000 nodes of 1m
$ yarn test --memory 4g --num-nodes 1000 --node-size 1m
Failed with exit code 134 after 15.63s.

Running test #15 for 4g mem, 1000 nodes of 1m
$ yarn test --memory 4g --num-nodes 1000 --node-size 1m
Failed with exit code 134 after 16.05s.

All tests failed for 4g memory, 1000 nodes.

Running test #0 for 8g mem, 100 nodes of 1m
$ yarn test --memory 8g --num-nodes 100 --node-size 1m
Built after 18.55s!

Running test #1 for 8g mem, 100 nodes of 1m
$ yarn test --memory 8g --num-nodes 100 --node-size 1m
Built after 18.51s!
...

Documentation

Updated the README!

[pieh]

Awesome!

[pieh]

Not blocking, just future opportunity:

Maybe we can just install jemalloc always in image and then have --use-jemalloc as toggle for cli (we could set LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so env var in test runner script to enable it), this could limit how often we have to (re)build image when testing

[imjoshin]

Good call. I think if we revisit this, that's a good improvement.

[pieh]

How often does that happen? We shouldn't be getting flakes like that :S

[imjoshin]

Maybe 1/5 times? Fairly often... I tried to debug it for ~30 minutes and couldn't find it. It has nothing to do with our build, but the docker exec. Still very confused by it.

I've accounted for this in our timings, so it only pulls the timing for the latest build. It'll slow the test run down a tiny bit, but doesn't affect much else. Just annoying. 😦