bug: noReset and enforceFreshSimulatorCreation on xcuidriver do not act as expected with simulators

[mythsunwind]

Do I have the most recent component updates?

• I use the most recent available driver/plugin and server versions

Is the component officially supported by the Appium team?

• I have verified the component repository is present under the Appium organization in GitHub

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

The documentation of xcuidriver explains the capabiltiyenforceFreshSimulatorCreation like this: "Creates a new simulator in session creation and deletes it in session deletion. Defaults to false."

Steps:

1. Run a session with simulator and the capabilties appium:noReset = false and appium:enforceFreshSimulatorCreation = false. You can find the output in the gist.

Current Behaviour:
The simulator is deleted after the session ended (see log) even though enforceFreshSimulatorCreation is set to false

It seems this is caused by a wrong implementation of noReset in xcuitest driver here: https://github.com/appium/appium-xcuitest-driver/blob/master/lib/driver.js#L1008 (because i see no check against enforceFreshSimulatorCreation in this code).

Or by the "TEST EXECUTE INTERRUPTED" after the test finished successfully (see gist).

Expected Behavior

• The app is uninstalled or reinstalled on session start
• The simulator is not deleted after the run

Minimal Reproducible Example

See gist

Environment

• Operating system: macOS
• Appium server version (output of appium --version): v2.0.0-rc.3
• Appium driver(s) and their version(s): xcuitest v4.32.10
• Appium plugin(s) and their version(s): none
• Node.js version (output of node --version): v18.14.0
• npm version (output of npm --version): 9.3.1
• Last component(s) version which did not exhibit the problem: could not find one. tried with v2.0.0-beta.56 and xcuitest v4.14.0 and they seem to be also affected
• Platform and version under test: xcode 14.3.1
• Real device or emulator/simulator: simulator on 16.4 runtime

Link to Appium Logs

https://gist.github.com/mythsunwind/bb8e27cee5d96786dd18606401f673a8

Futher Information

I experience another strange behaviour regarding noReset: If noReset is set once to true and then to false again the simulator is not deleted until i delete it manually. After that the simulator is deleted every time after the test. i cannot explain this behaviour.

[mykola-mokhnach]

What is your initial intention there? Do you want a new simulator to be created every time a new test is started? I assume it takes a lot of time and it would make sense to simply provide platformVersion capability in order to use a precreated device.

[mykola-mokhnach]

Also the fact that setting enforceFreshSimulatorCreation to false does not change the actual behavior is expected. It only affects things being set to true (e.g. even if a test could reuse an existing device it would still create a new one). In the above case the driver was not able to match any existng sims, so it had to create a new one (which has happened) or to fail.

[mythsunwind]

What is your initial intention there? Do you want a new simulator to be created every time a new test is started? I assume it takes a lot of time and it would make sense to simply provide platformVersion capability in order to use a precreated device.

My intention is to NOT create a new simulator every time. But unfortunately setting appium:noReset = false and appium:enforceFreshSimulatorCreation = false unfortunately does sometimes create a new simulator every time right now due to a potential bug.

I currently can reproduce the behaviour with the step:

1. Delete the simulator (called appiumTest...) with simutil
2. Run test with noReset = false

[mythsunwind]

Also the fact that setting enforceFreshSimulatorCreation to false does not change the actual behavior is expected. It only affects things being set to true (e.g. even if a test could reuse an existing device it would still create a new one). In the above case the driver was not able to match any existng sims, so it had to create a new one (which has happened) or to fail.

The problem is not that it creates a new one in the beginning but delete the simulator in the end of the test which results in creating a new one again next time.

[mykola-mokhnach]

the driver has a concept of life cycle and thus always assumes that if the simulator was created by it then it must also be deleted at the end to complete the life cycle. So if you want to reuse the simulator it would make sense to precreate a new one using simctl and then just provide its uuid to capabilities.

[mythsunwind]

the driver has a concept of life cycle and thus always assumes that if the simulator was created by it then it must also be deleted at the end to complete the life cycle. So if you want to reuse the simulator it would make sense to precreate a new one using simctl and then just provide its uuid to capabilities.

Thanks for the clarification. Maybe the documentation at https://appium.github.io/appium-xcuitest-driver/4.32.7/capabilities/ should be updated for this. I think the information for noReset and enforceFreshSimulatorCreation are quite confusing there.

Thanks also for the workaround to create a simulator manually. I will continue with this although I think there is a bug that sometimes the automatically generated simulator (called appiumTest...) is not always deleted which gave me the impression there is a right way to reuse a automatically created simulator. But now (as far as i understood your explanation) there currently is no way to use the function to automatically generated simulator together with a capability that deletes the app but not the WDA or the simulator. Did I understood this correctly?

[mykola-mokhnach]

I assume this part of the code has not been touched for very long time. I'll create a PR to properly align the logic there

[mykola-mokhnach]

I will continue with this although I think there is a bug that sometimes the automatically generated simulator (called appiumTest...) is not always deleted which gave me the impression there is a right way to reuse a automatically created simulator

The idea is that if you want a Simulator to be reused then you should not assign its creation to the driver. This is how the ownership works: if I create a Simulator then this is also my responsibility to delete it later (or to keep it), - if the driver does that then it is also up to the driver to delete/cleanup it whenever needed.

[mykola-mokhnach]

But now (as far as i understood your explanation) there currently is no way to use the function to automatically generated simulator together with a capability that deletes the app but not the WDA or the simulator. Did I understood this correctly?

Based on the above explanation - yes, it is not possible to preserve a simulator that is created automatically by the driver.

[mythsunwind]

I assume this part of the code has not been touched for very long time. I'll create a PR to properly align the logic there

Thank you for the effort, Nick. Would it be possible to reflect this somewhere in the documentation of xcuidriver to help future users?

[mykola-mokhnach]

I have aligned the actual behaviour in the driver code. Not sure what to change in the docs though. Feel free to propose a PR yourself. Driver docs could be found at https://github.com/appium/appium-xcuitest-driver/tree/master/docs

[eglitise]

This information has since been added to the XCUITest documentation: https://appium.github.io/appium-xcuitest-driver/latest/guides/troubleshooting/#simulator-resetting