Variables in test cases for Automation Runtime are inherited across test cases

[chryzsh]

Describe the bug
When running multiple test cases at once with the Automation Runtime on Windows, any variable name that are used across multiple test cases are applied to all test cases in the runtime. Thus, the variable of individual test cases are not respected. This is a problem because a lot of the automation for test cases include commonly used variable names like "path", "url" and similar.
From what I can tell this happens individually of executor, so I can confirm the bug is present on both Powershell, CMD and bash. I have tested it on Windows, Linux and Mac.

To Reproduce
Steps to reproduce the behavior:
Preparation:
Go to Assessments, select a Campaign and create two Test Cases. For each of them:
Click on Configure under Automation & Logging
Verify that the test case has an automation configuration, or create one. Use the same variable name for both cases, but different values.
Save
In Campaign Actions click Build Automation Runtime. This will prep and download a ZIP-file containg the binary to be executed
Extract the zip file to a folder.
Execution:
Open a Console window and navigate to the folder.
Execute the automation runtime binary.
Inspect the STDOUT in the console window and/or JSON file which is written to test that the first environment variable is inherited to future test cases when they are executed.

Expected behavior
I expected the variables set on each test case to only be used on that test case, and not inherited to future test cases in the same runtime binary.

Screenshots
To provide a minimal and clear example, I've here created two test cases : Test 1 and Test 2. Both are identical, but in the automation I have configured Test 1 to use the variable url set to http://test1.xyz and Test 2 the same variable name but value http://test2.xyz.



Here, both test cases are executed, and the variable from the first executed test case, here "Test Case 1" and its url variable is applied to Test Case 2 because the variable name is the same. So Test Case 2 does not get executed with the correct url set in the automation configuration.

I replicated this test by changing the executor to Bash and testing the binary on Linux and Mac. The issue is the same there:

Desktop (please complete the following information):
OS: Windows 10
Browser: Chrome
Version: ce-8.7.1

Additional context
A possible workaround is to generate individual test cases. Another workaround is to use individual variable names for all test cases. Neither are very scalable solutions.

[SRAPSpencer]

Thanks for reporting, appreciate the detailed reporting.

We've confirmed this is an issue. No ETA on a fix right now.

[thebleucheese]

Fixed in VECTR 8.8.0

[chryzsh]

Sorry to say this still happens btw. Just tested it on Version 9.4. Not sure if the fix ever worked or if its a regression since the time of the fix.

[doodleincode]

Thanks for the update. Sorry about the continued issues with this. We'll investigate and provide an update.