When integration test does not end properly it leaves minishift running

[agajdosi]

When integration test runner fails - for example on a timeout then minishift.runner.EnsureDeleted() in afterSuite() is not executed and thus the environment is not left clean - minishift instance is running. This can be little impractical in cases of manual use, but is much more in cases of automated tests. In downstream, when nightly tests fail it this manner, then also next nightly tests will fail as minishift start is then not possible.

Would be nice to add check into beforeSuite() to prevent this issue.

[gbraad]

In the BeforeSuite() you would like to detect if a Minishift instance is still running? I believe there is another problem... I would expect that the timeout should trigger a cleanup action, such as an OnTimeout(). I think we should try to keep the integration tests from these startup checks as much as possible,

Who is timing out?

[agajdosi]

@gbraad Often it is because of a network issue. But some fails were also when there were changes with iso building job. It is fine to fail, but then it is wrong to throw away next day's nightly tests. Sometimes I am doing the clean up manually to save the next day, but I should not do that ideally.

I am fine with OnTimeout() if it will work, but I am still curious, what would pre-flight check cost us, or why you want to avoid them? :)

Another possibility would be to have some option to trigger the clean up through some command, so I could do that just on automated tests - but that is the case only if OnTimeout() plan fails.

[gbraad]

With CentOS CI, the node is always handed in a clean state. For nightlies it seems we rerun on the same install? If so, something else needs to ensure the clean state. It feels weird to me that a build and test it runs would have to clean up first. This is rather an action that happens at the end (but I think after the timeout, we are unable to do something). Why not cronbased? Or a depended task? I am trying to understand the setup. Which platforms suffer from this, etc

…

-- Gerard Braad | http://gbraad.nl [ Doing Open Source Matters ]

[agajdosi]

@gbraad It can happen on all platforms. Yes, this problem can be handled in jenkins job directly, which would lead me to writing a solution for each platform. But another perspective is also: if for any reason, developer, user, qe has some broken forgotten instance of minishift from previous test run still running, can be from terminating the process because you are impatient, what is the meaning of letting her start the tests and watch how they fail.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hferentschik]

@agajdosi Is this still an issue?

[agajdosi]

@hferentschik Sorry, missed your comment. This issue is less frequent as it was before, also I added some pre-flight checks to the nightly jobs, but still it is here. However in most cases it is caused by application deployment timeout, so I have created an issue: #1703 which should fix it.

I will then report here whether the issue is minimized or we need to add more timeouts to out steps.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.