Make a Node-based CT client

[zepumph]

Instead of relying on continuous-loop, we can set up a puppeteer client and have much more control over rerunning tests, getting the console output, and handling errors. @jonathanolson will take the lead, and this is coming from https://github.com/phetsims/special-ops/issues/234 and #173

[jonathanolson]

Initial commit in f4b952e

[zepumph]

Taking a look at this now and running some local testing with it.

[zepumph]

This is looking great! Thanks @jonathanolson.

• we fixed a bug above where playwright support meant that we lost the "this" of the page.
• We need to support id checking
• It would be nice to get some of these onto CT to see how they go.
• Perhaps this would be a nice time to use pm2 cluster mode where we can just have 50 of these all running a forever loop wrapping runNextTest. Though it may clog up the pm2 list to an unacceptable level.

[zepumph]

I should test runNextTest with some options like id and browserCreator (for firefox).

[zepumph]

Things seem to be running well on sparky. I supported cluster mode, and it is working seemlessly from what I can tell:

I don't know if we want to have 100 of these though. It seems a bit messy. @jonathanolson thoughts?

[zepumph]

• One error I'm seeing out of testing on sparky:

 You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
5|ct-node- | Error: Did not get next-test message in 400000ms
5|ct-node- |     at /data/share/phet/continuous-testing/ct-node-client/aqua/js/node-client/runTest.js:146:19

[zepumph]

Here is what one CT error looks like from this new client:

molarity : interactive-description-fuzzBoard : unbuilt
http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/molarity/molarity_en.html?continuousTest=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D&brand=phet&ea&fuzzBoard&supportsInteractiveDescription=true
Query: brand=phet&ea&fuzzBoard&supportsInteractiveDescription=true
Uncaught Error: Assertion failed: no item found that has focus
Error: Assertion failed: no item found that has focus
at window.assertions.assertFunction (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/assert/js/assert.js:28:13)
at assert (ComboBoxListBox.ts:311:14)
at getFocusedItemNode (ComboBoxListBox.ts:238:75)
at inputEvent (Input.ts:1899:69)
at dispatchToListeners (Input.ts:1939:11)
at dispatchToTargets (Input.ts:1851:9)
at dispatchEvent (Input.ts:1119:11)
at dispatchPDOMEvent (Input.ts:800:20)
at apply (PhetioAction.ts:158:16)
at execute (Input.ts:1597:23)
[URL] http://127.0.0.1/continuous-testing/aqua/html/sim-test.html?url=..%2F..%2Fct-snapshots%2F1683744638744%2Fmolarity%2Fmolarity_en.html&simQueryParameters=brand%3Dphet%26ea%26fuzzBoard%26supportsInteractiveDescription%3Dtrue&duration=40000&testInfo=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D
[NAVIGATED] http://127.0.0.1/continuous-testing/aqua/html/sim-test.html?url=..%2F..%2Fct-snapshots%2F1683744638744%2Fmolarity%2Fmolarity_en.html&simQueryParameters=brand%3Dphet%26ea%26fuzzBoard%26supportsInteractiveDescription%3Dtrue&duration=40000&testInfo=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D
[NAVIGATED] about:blank
[NAVIGATED] http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/molarity/molarity_en.html?continuousTest=%7B%22test%22%3A%5B%22molarity%22%2C%22interactive-description-fuzzBoard%22%2C%22unbuilt%22%5D%2C%22snapshotName%22%3A%22snapshot-1683744638744%22%2C%22timestamp%22%3A1683745232137%7D&brand=phet&ea&fuzzBoard&supportsInteractiveDescription=true
[CONSOLE] enabling assert
[CONSOLE] continuous-test-load
[CONSOLE] Assertion failed: no item found that has focus
[PAGE ERROR] Error: Error: Assertion failed: no item found that has focus
at window.assertions.assertFunction (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/assert/js/assert.js:28:13)
at ComboBoxListBox.getFocusedItemNode (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/sun/js/ComboBoxListBox.js:251:15)
at Object.keydown (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/sun/js/ComboBoxListBox.js:185:75)
at Input.dispatchToListeners (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1714:25)
at Input.dispatchToTargets (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1743:12)
at Input.dispatchEvent (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1674:10)
at Input.dispatchPDOMEvent (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:998:12)
at Input.keydownAction.PhetioAction.phetioPlayback (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:708:21)
at PhetioAction.execute (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/tandem/js/PhetioAction.js:127:17)
at Input.keyDown (http://127.0.0.1/continuous-testing/ct-snapshots/1683744638744/chipper/dist/js/scenery/js/input/Input.js:1449:24)
[CONSOLE] continuous-test-error

id: "Sparky Node Puppeteer"
Snapshot from 5/10/2023, 12:50:38 PM

[zepumph]

• launchArgs need to be moved over for the puppeteer node client:
  

  aqua/js/server/puppeteerCTClient.js

  Lines 39 to 54 in 146014f

  	// With this flag, temp files are written to /tmp/ on bayes, which caused https://github.com/phetsims/aqua/issues/145
  	// /dev/shm/ is much bigger
  	ignoreDefaultArgs: [ '--disable-dev-shm-usage' ],
  	// Command line arguments passed to the chrome instance,
  	args: [
  	'--enable-precise-memory-info',
  	// To prevent filling up `/tmp`, see https://github.com/phetsims/aqua/issues/145
  	`--user-data-dir=${process.cwd()}/../tmp/puppeteerUserData/`,
  	// Fork child processes directly to prevent orphaned chrome instances from lingering on sparky, https://github.com/phetsims/aqua/issues/150#issuecomment-1170140994
  	'--no-zygote',
  	'--no-sandbox'
  	]

[zepumph]

We want to have this new client be the way of the future, let's phase out the old pattern

[zepumph]

Oops, I forgot about all the great puppeteer options we still need.

[zepumph]

Ok, we now have:

bayes:
16 node puppeteer clients

sparky:
5 browser firefox,
5 browser puppeteer
65 node puppeteer
25 node firefox.

I'll check in tomorrow.

[zepumph]

• Check the logs and see if firefox is causing any trouble.

[zepumph]

It has been helpful to have objects like this for testing:

  // testInfo = {
  //   'snapshotName': 'snapshot-1684173607587',
  //   'test': [ 'geometric-optics', 'phet-io-wrappers-tests', 'assert' ],
  //   'url': 'qunit-test.html?url=..%2F..%2Fct-snapshots%2F1684173607587%2Fphet-io-wrappers%2Fphet-io-wrappers-tests.html%3Fsim%3Dgeometric-optics%26phetioDebug%3Dtrue%26phetioWrapperDebug%3Dtrue',
  //   'timestamp': 1684172635047
  // };
  testInfo = {
    'snapshotName': 'snapshot-1684173607587',
    'test': [ 'geometric-optics', 'phet-io-wrappers-tests', 'assert' ],
    'url': '../../ct-snapshots/1684173607587/phet-io-wrappers/phet-io-wrappers-tests.html?sim=geometric-optics&phetioDebug=true&phetioWrapperDebug=true',
    'timestamp': 1684172635047
  };

[zepumph]

Ok. I'm feeling good about this issue. Over the weekend tests were running very well, save a couple of small problems.

I found that CT snapshots weren't getting transpiled and so the phet-io-wrappers tests were breaking, but e143df1 solved that.

@jonathanolson and I also updated some of the criteria for keeping snapshots, keeping the last 20 instead of 3 now, so that we can reproduce/test problems easier. Space isn't an issue on Sparky for that, so we aren't worried.

I think we are ready to close this issue. The rest will be handled by side issues. Thanks @jonathanolson for getting this to such a nice spot.

[phet-dev]

Reopening because there is a TODO marked for this issue.