[QA] GUI Test reporting for desktop client

[saw-jan]

From #789 (comment) by @HanaGemela

Regression test

Currently all the tests run nightly. That's correct and that will stay so also in the future. But there is also going to be a smaller subset of all tests called regression test. As it has the highest priority currently all the tests we have and all the test in this issue are going to be part of this regression test. Please tag all these tests as 'regression test' so they can also be run separately on demand.

Reporting

All the tests run nightly generate a report only in case of failure and this is sent to desktop chat. Please send it also to my email.
For the regression test make sure the report gets generated every time after each run and it is send again to the desktop chat and my email.

Something to discuss - after the regression test we need an easily readable short overview what was tested and the results. But after the nightly run we want all the available information for easier debugging and better understanding. So maybe there are going to be two types of report - one for engineering and one for public.

The report for public needs to be readable for all people therefore if possible:

• pdf format
• no Time data table
• no Logs
• can the name of the feature be more user friendly? Currently it's called e.g. tst_addAccount
• can we do something with the structure? Could it be a nice table with e.g. 2 columns Scenario and the result? How it is currently will be a nightmare to read when it gets longer

[saw-jan]

@HanaGemela

Currently all the tests run nightly. That's correct and that will stay so also in the future. But there is also going to be a smaller subset of all tests called regression test. As it has the highest priority currently all the tests we have and all the test in this issue are going to be part of this regression test. Please tag all these tests as 'regression test' so they can also be run separately on demand.

Which automated tests from the existing tests are to be considered regression tests? Could you help us with that? So that we can plan/discuss on how to run those tests separately and on what events.

• pdf format
• no Time data table
• no Logs

can we do something with the structure? Could it be a nice table with e.g. 2 columns Scenario and the result? How it is currently will be a nightmare to read when it gets longer

This might need a custom report parser. we can give it a try

can the name of the feature be more user friendly? Currently it's called e.g. tst_addAccount

Sorry about that 😅 but that's the folder name we have in our test. And we preferred short names for the test case folders. But if we have a custom report parser then maybe we can show more reader-friendly titles instead of folder names

For the regression test make sure the report gets generated every time after each run and it is send again to the desktop chat and my email.

sending to email will need extra email setups in the CI but I think we can easily send to your private chat (rocketchat)

[HanaGemela]

@saw-jan

1. Currently all the tests are the regression tests.
2. If custom parser is a lot of effort, then I'd wait if anyone complains about current state. It's not bad as it is now. I could imagine something more user friendly but it doesn't have a high priority. At this point we should focus on higher test coverage than on this.
   I understand you need a short feature name in your tests. On the other hand it makes it hard for users to understand when they read the report. We can try to find a compromise
3. Yes, my private chat would be also sufficient.

[saw-jan]

@HanaGemela how would you like to receive the chat notification about the Squish report?

• on every push to the master branch?
• on nightly build only?

[HanaGemela]

@HanaGemela how would you like to receive the chat notification about the Squish report?

• on every push to the master branch?
• on nightly build only?

As it is now - failed nightly build. But then we also need a report from regression testing every time no matter if it failed or passed

[saw-jan]

Now we have a html report generated and can be accessible in the ci build.