Add profile information to the visual test files (#3270)

* Update test files
* Fix apollo test
* Cleanup test files
* Include information about the profile being loaded
* Add documentation about the visual tests

src/engine/openspaceengine.cpp

@@ -422,6 +422,7 @@ void OpenSpaceEngine::initialize() {
     }
 
     // Load the profile
+    LINFO(std::format("Loading profile '{}'", profile));
     *global::profile = Profile(profile);
 
     // Set up asset loader

tests/visual/README.md

@@ -0,0 +1,76 @@
+# Visual Test Specification
+All `.ostest` files in this folders specify visual image tests that are automatically run to ensure that changes in OpenSpace do not negatively impact the rendered results. The tested results are available at https://regression.openspaceproject.com.
+
+The files are organized by folders, which are used as the "group" name for the test and filename of each test (without `.ostest` extension) is used as the name of the test. In general the first folder should name a profile that is being tested by the files within or a specific use-case, such as testing all renderables. Additional subfolders can be used as required within each top-level folder.
+
+## Test Structure
+Each test must have a `screenshot` instruction as the last entry, which causes an image to be created that is used as the end result of the test. Only exactly one `screenshot` instruction per test is currently supported. Each `.ostest` file is a JSON file with two top-level keys: `profile` provides the name of the profile that should be loaded before running these test instructions, and `commands` is a list of instructions that should be executed after the profile was loaded. All instructions must have a `type` key to determine which type of instruction it is and most have a `value` key that determines the parameters for that instruction.
+
+### Best practices
+  - By default, all tests always start paused, MRF caching is enabled, and the user interface and dashboard items are disabled
+  - All test should start with in instruction to set a specific time to improve reproducibility
+  - The few instructions there are per test, the better
+  - Adding `wait` instructions to ensure OpenSpace has time to load dynamic datasets increases reliability, but too many `wait`s will slow-down the over all testing
+  - Avoid `recording` and use `navigationstate` and `time` instead
+  - Avoid `script` if possible and use dedicated instructions when they exist. If we see the same `script` instruction used in many tests, they can be upgraded to a first-class instruction at later stage
+
+### Instructions
+  - `action`: Triggers an action that must already be defined in the profile or that was defined previously in this test. The provided value must be a string that is the identifier of the action that should be triggered.
+
+    Example: `{ "type": "action", "value": "os.FadeDownTrails" }`
+
+    Script Equivalent: `openspace.action.triggerAction`
+
+  - `deltatime`: Instantly changes the delta time in OpenSpace to the provided value. The provided value must be a number that is the delta time in seconds per realtime second that the engine should be set to.
+
+    Example: `{ "type": "deltatime", "value": 10 }`
+
+    Script Equivalent: `openspace.time.setDeltaTime`
+
+  - `navigationstate`: Sets the camera to the provided navigation state. The provided value must be an object that must contain at least a `anchor` and `position` and may optionally contain an `aim`, `referenceFrame`, `up`, `yaw`, `pitch`, and `timestamp`. All these values are then used to instantaneously set the position of the camera.
+
+    Example: `{ "type": "navigationstate", "value": { "anchor": "Juno", "pitch": -0.0165756, "position": [ -22.49081, 1.191533, 26.35740 ], "up": [ 0.0288083, 0.999373, -0.0205962 ], "yaw": 0.152454 } }`
+
+    Script Equivalent: `openspace.navigation.setNavigationState`
+
+  - `pause`: Determines whether the in-game clock should be paused or resumed. The provided value must be a boolean that is the clock state after the instruction
+
+    Example: `{ "type": "pause", "value": false }`
+
+    Script Equivalent: `openspace.time.setPause`
+
+  - `property`: Sets a specific property or group of properties to the specified value. This change is instantaneous. The provided value must contain a `property` key that is the identifier or regex for the property that should be set and a `value` key that is the new value for the property. The type of the `value` must be correct for the matched `property`.
+
+    Example: `{ "type": "property", "value": { "property": "Scene.Constellations.Renderable.Enabled", "value": true } }`
+
+    Script Equivalent: `openspace.setPropertyValue`
+
+  - `recording`: Triggers the playback of a session recording. The provided value is the name of the session recording file that should be played.
+
+    Example: `{ "type": "recording", "value": "solarsystem.osrec" }`
+
+    Script Equivalent: `openspace.sessionRecording.startPlayback`
+
+  - `screenshot`: Takes a screenshot of the application. At the moment, there can be only exactly one instruction of this type and it should be the last instruction in the test. This instruction also does not take any parameters
+
+    Example: `{ "type": "screenshot" }`
+
+    Script Equivalent: `openspace.takeScreenshot`
+
+  - `script`: Executes the script that is passed in as a value. That value must be a string that is the Lua script that is executed directly.
+
+    Example: `{ "type": "script", "value": "openspace.printError("Hello world") }`
+
+    Script Equivalent: `value`
+
+  - `time`: Sets the in-game time to the provided value. The value can be either a string, in which case it needs to be a valid date-time string, or a number, in which case it represents the number of seconds past the J2000 epoch.
+
+    Example: `{ "type": "time", "value": "2016-07-01T00:00:01.00" }`
+
+    Script Equivalent: `openspace.time.setTime`
+
+  - `wait`: Causes the test to wait for the specified number of seconds. Note that the OpenSpace testing instance is still running in the background and is, for example continuing to load dynamic content while the test is waiting.
+
+    Example: `{ "type": "wait", "value": 2 }`
+
+    Script Equivalent: none

tests/visual/apollo/11/landing.ostest

@@ -0,0 +1,20 @@
+{
+  "profile": "apollo",
+  "commands": [
+    { "type": "action", "value": "os.apollo11.setup.LandingSite" },
+    { "type": "wait", "value": 10 },
+    { "type": "time", "value": "1969-07-20T20:15:50.00" },
+    {
+      "type": "navigationstate",
+      "value": {
+        "anchor": "Apollo11LemPosition",
+        "pitch": 0.979095E-1,
+        "position": [ -2.457687, 25.04798, 3.804939 ],
+        "up": [ -0.682505, 0.0438584, -0.729564 ],
+        "yaw": 0.0435056
+      }
+    },
+    { "type": "wait", "value": 45 },
+    { "type": "screenshot" }
+  ]
+}

tests/visual/apollo/11/landingsite.ostest

@@ -0,0 +1,30 @@
+{
+  "profile": "apollo",
+  "commands": [
+    { "type": "action", "value": "os.apollo11.setup.LandingSite" },
+    { "type": "wait", "value": 10 },
+    {
+      "type": "navigationstate",
+      "value": {
+        "anchor": "Apollo11LemPosition",
+        "pitch": -0.275871E-1,
+        "position": [ -329.1966, 726.2499, -767.1269 ],
+        "referenceFrame": "Root",
+        "up": [ -0.833341, -0.532887, -0.146881 ],
+        "yaw": -0.000141130
+      }
+    },
+    { "type": "wait", "value": 10 },
+    {
+      "type": "navigationstate",
+      "value": {
+        "anchor": "Apollo11LemPosition",
+        "position": [ 6.249130, 25.76497, -14.35499 ],
+        "up": [ -0.860897, -0.0718231, -0.503684 ],
+        "yaw": -0.000141143
+      }
+    },
+    { "type": "wait", "value": 45 },
+    { "type": "screenshot" }
+  ]
+}

tests/visual/apollo/11/orbits.ostest

@@ -0,0 +1,19 @@
+{
+  "profile": "apollo",
+  "commands": [
+    { "type": "action", "value": "os.apollo11.setup.LandingSite" },
+    { "type": "wait", "value": 10 },
+    {
+      "type": "navigationstate",
+      "value": {
+        "anchor": "Apollo11LemPosition",
+        "pitch": 0.00776040,
+        "position": [ -711349.3, 2688364, 539334.1 ],
+        "up": [ -0.382741, 0.0833124, -0.920091 ],
+        "yaw": 0.00288877
+      }
+    },
+    { "type": "wait", "value": 30 },
+    { "type": "screenshot" }
+  ]
+}

tests/visual/apollo/17/landingsite.ostest

@@ -0,0 +1,29 @@
+{
+  "profile": "apollo",
+  "commands": [
+    { "type": "action", "value": "os.apollo17.setup.LandingSite" },
+    {
+      "type": "navigationstate",
+      "value": {
+        "anchor": "Moon",
+        "pitch": 0.191313,
+        "position": [ 1403983, 835896.1, 600562.5 ],
+        "up": [ -0.316846, -0.141630, 0.937843 ],
+        "yaw": -0.143901
+      }
+    },
+    { "type": "wait", "value": 10 },
+    {
+      "type": "navigationstate",
+      "value": {
+        "anchor": "Moon",
+        "pitch": 0.953555,
+        "position": [ 1399797, 833416.1, 598828.3 ],
+        "up": [ -0.381687, -0.0228442, 0.924009 ],
+        "yaw": -0.0789410
+      }
+    },
+    { "type": "wait", "value": 45 },
+    { "type": "screenshot" }
+  ]
+}

tests/visual/apollo/8/pre-earthrise.ostest

@@ -0,0 +1,8 @@
+{
+  "profile": "apollo",
+  "commands": [
+    { "type": "action", "value": "os.apollo8.SetupEarthrise" },
+    { "type": "wait", "value": 60 },
+    { "type": "screenshot" }
+  ]
+}