LP1964852: Need a way to end a test session, aborting remaining reboot tests

[beliaev-maksim]

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1964852

Summary

Status	Created on	Heat	Importance	Security related
Triaged	2022-03-15 02:01:50	6	High	False

Description

There are probably multiple issues here.
I would like to be able to tell checkbox-cli to abort a test session, canceling any unrun tests, including scheduled reboot tests.

I'm running CDTS locally on the DUT.
I chose ODM-Certification 20.04 Desktop Automated Tests.

I only wanted to run the Cold Reboot stress test. This test generates 100 test jobs, each one does a reboot of the system.

Around reboot #60 or so, the system failed to reboot.

I manually power cycled the machine and quickly ran cdts.checkbox-cli

I am only given the option to either resume the test session, or delete it. I cannot end it.

Then the next reboot happened.

System rebooted fine this time, so once again I run cdts.checkbox-cli. I say to resume the session. The session resumes at Cold Reboot 63, which failed because of a missing log file. I can either rerun the test, mark it as pass/fail or skip it. I don't want to rerun Cold Reboot 63, nor do I want to progress to Cold Reboot 64, but there are no other options.

I thought I could just stay at that menu, and this would effectively pause the tests so I could gather logs, run sosreport, etc.

Then the next reboot happened.

At this point, I don't seem to be able to regain control of the system. Either I let it finish the remaining 40 tests, or I reinstall Ubuntu.

The system doesn't stay up long enough for me to gather logs or other files.

Attachments

No attachments

Tags:
['checkbox', 'hwe-powercycle', 'testcase']

[beliaev-maksim]

This thread was migrated from launchpad.net

https://launchpad.net/~djacobs98 wrote on 2022-03-22 02:27:31:

I observed that a checkbox process gets left behind even after I had tried to stop and delete the current session. This process is responsible for configuring the system to shutdown and reboot.

Manually killing this task, and doing a manual reboot for good measure before starting a new session seems to have worked.

How to reproduce:
Start the cold reboot test.
When the system reboots, quickly log in, and rerun checkbox.
Delete the current session.

Expected result:
This should stop the reboot test.

Actual result:
System still reboots.

You will see a process that talks about reboot in the process list. Killing this process will end the reboot loop.

[kissiel]

The code needed to enable skipping a set of jobs in the queue when the test session is already running would introduce a tremendous amount of complexity, so I'm inclined to reject this ATM.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CHECKBOX-1196.

This message was autogenerated