New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Redesign the migration regression cases #15154
Conversation
Great PR! Please pay attention to the following items before merging: Files matching
This is an automatically generated QA checklist based on modified files |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this need more investigation.
- Regression module have module dependency, how to solve it? If need more module combination for the migration qcow, need investigate whether the redesign is valuable.
- Need investigate whether to split to two jobs is valuable to do it.
- I don't think change the logic of service check is needed. If you want to change it for a specific change please add setting to control.
From my opinion, we just need split the regression test into more small group is enough, do not need so big change
9ced6c5
to
355227e
Compare
If any modules are needed by regression tests, add the module in the migration case would solve it. BTW, the uploaded qcow is registered, so the environment is the same as before.
This is what I did recently, splitting to several jobs would save the time that running the regression tests one by one.
I didn't change the logic of service check. I just put the checking of the upgraded services into the second case.
|
ed33061
to
cf811ab
Compare
We redesign the migration regression cases that spliting a long case into several cases: after migration, publsh a qcow, and then do regression tests with the qcow; for the regression tests, we use INCLUDE_MODULES setting to load the regression test modules. The reason that fix the service check is we check the upgraded services on the second case, it can not check the result in the first job.
cf811ab
to
9228e52
Compare
The PR seems works while base on the solution will cause lost test coverage, and it's benefit can be realize with debug method. I will reserve my opinion, PO & SM can decide whether to use this solution. |
We redesign the migration regression cases that spliting a long case into
several cases: after migration, publsh a qcow, and then do regression
tests with the qcow; for the regression tests, we use INCLUDE_MODULES
setting to load the regression test modules. The reason that fix the
service check is we check the upgraded services on the second case, it can
not check the result in the first job.
For the demo cases:
https://openqa.suse.de/tests/9009427 (migration and publish qcow)
https://openqa.suse.de/tests/9009432 (regression test of ImageMagick)
https://openqa.suse.de/tests/9009435 (regression test of console)
https://openqa.suse.de/tests/9009434 (regression test of gnome)
https://openqa.suse.de/tests/9009428 (regression test of we)
https://openqa.suse.de/tests/9009431 (regression test of wireshark)
https://openqa.suse.de/tests/9009429 (regression test of openldap_to_389ds)
https://openqa.suse.de/tests/9009430 (regression test of migration_features)
https://openqa.suse.de/tests/9009433 (regression test of tomcat)
I updated the service check code, so I run the upgrade service separately:
https://openqa.suse.de/tests/9090852 (migration and publish qcow)
https://openqa.suse.de/tests/9090853 (regression test of check_upgraded_service)
https://openqa.suse.de/tests/9090854 (migration and publish qcow)
https://openqa.suse.de/tests/9090855 (regression test of check_upgraded_service)