Performance tests

[jmgasper]

Overview

The current pxScene / Spark app has a test suite that can be run as follows:

1. Launch the app
2. Enter a URL like this, changing the file path to the file on your local machine.

https://px-apps.sys.comcast.net/pxscene-samples/examples/px-reference/test-run/testRunner_v5.js?tests=file:/Users/justingasper1/Documents/Git/pxCore-tc/tests/pxScene2d/testRunner/tests.json

Description

The test suite above tests compatibility, ensuring that functionality works as expected and renders properly. What it doesn't test is overall performance, including things like:

• Image load times
• Memory usage
• CPU usage

What we want to start to build is a test suite that can be run to compare the overall performance of different builds of the app to each other.

Base test requirements

For this challenge, we will first build a base test object that can be used to wrap results of any test we want, calculating:

• Image load times
• Memory usage
• CPU usage

At the end of a test will be a summary shown to the user listing:

• The average image load time (in milliseconds)
• The maximum memory used (in megabytes)
• A simple graph of the overall CPU usage, with the high CPU usage % also listed.

Individual tests

These individual tests are required to be implemented as extensions of the base test. We want to be able to run these tests and have them run / loading in pxScene and then the results from the base test wrapper will be displayed to the user. Note that these examples may have to be adapted a bit, so feel free to download them, modify them, and include them in your submission. Don't expect to run them from the server. For instance, the gallery example loops, but we only want it to run once or twice to generate the performance test results.

• http://pxscene.org/examples/px-reference/gallery/gallery.js
• http://pxscene.org/examples/px-reference/gallery/picturepile.js
• http://pxscene.org/examples/px-reference/gallery/fancy.js
• https://px-apps.sys.comcast.net/pxscene-samples/examples/px-reference/gallery/coverflowtest_v2.js

For the coverflow test, please implement an auto-scroll for the test. Don't expect input from the user.

You will submit at least 5 test files:

• base-performance-test.js
• gallery-test.js
• picturepile-test.js
• fancy-test.js
• coverflow_v2-test.js

New APIs

You will likely need to add new APIs to the C++ and JS code to be able to support monitoring performance of the app. The client is very sensitive to changes to the C++ and JS code, so you must document every change very, very clearly. Please include inline comments for every change, and also include a document describing every C++ and JS change to the core pxScene layer. It's a lot of work, but our changes won't be accepted without this.

Ideally we won't need to change any C++ / JS code in the existing app, so please investigate that option first.

No new libraries can be added to the app.

Any changes to the C++ have to be able to be turned on via the cmake options. The default is that any new features / APIs will be disabled by default, not changing the current functionality of the default build. We can then choose to turn the function on for debug / test builds.

Flow

The goal is to be able to take a test file, like a new gallery-test.js file, load it in pxScene on various platforms, including:

• Mac
• Linux
• Windows
• Raspberry Pi

And compare the performance numbers directly between each platform.

We will start to expand the tests to include more functionality after this initial version, based on client feedback.