CT-style testing for release branches

[jonathanolson]

I've added some checks in perennial to detect errors in release branches. It seems like this could get tested regularly as part of CT, and be displayed in the interface.

We wouldn't want to burn too much CPU or space on this, but separate check-outs per sim have an advantage (to verify only those dependencies are needed).

I'll discuss sometime with @jbphet (subgroup).

[zepumph]

Noting thoughts from #203:

From discussion with @jonathanolson related to phetsims/chipper#983 and #202.

We believe it would save a lot of time with maintenance if we had a tool on the server that could tell us the current status and errors with our release branches. Here is our first idea:

• This would be a new tool, since it doesn't fit well into the idea of snapshots.

• We want these tests:

  • ReleaseBranch.getStatus()
  • build/lint/tsc (noting here that lint could potentially because annoying to maintain, but let's start with it in.
  • Fuzz the built sim. This may lead to a lot of noise, but it would be good to see to start. At its best it could help us see when old releases break on modern browsers. At its worst it could just show us fuzz errors that we have already deemed unimportant and not blocking publication.

• The server could have data that could integrate into the current CT report (kinda like CTQ does).

I'm not going to work on this right away, but I'm glad to have an idea, and to be able to crank away at it on the back burner.

[jonathanolson]

I'd like to point you to some release branch checks (growing list of automated tests that I'm running for release branches).

[zepumph]

Sounds excellent. Noting here that I'm not working on this actively, I just noticed the duplicate issue today in the aqua repo. Feel free to list them here though! I bet that future work on CT to support development branches could be related to this, so it would be good to have an idea about what you would like to see.