Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Remove existing reports #657

Closed
atodorov opened this issue Dec 8, 2018 · 20 comments
Closed

RFC: Remove existing reports #657

atodorov opened this issue Dec 8, 2018 · 20 comments

Comments

@atodorov
Copy link
Member

@atodorov atodorov commented Dec 8, 2018

We are planning to remove the existing reports feature because they are not well designed and the code internally is a huge blob that can't be extended very easily. Kiwi TCMS team would like to reimplement existing functionality that folks find useful, add new types of reports (incl. nicer graphics and UI) and make it possible for the reporting sub-system to be more easily extendable.

Phase out is planned to begin after 1st March 2019!

Until then we are looking for feedback from the community:

  1. Which of the existing reports do you use? What kind of decisions do you make based on them?

  2. What kind of reports/decision you have to make that currently do not exist or you feel like the information is not easy to obtain ?

@Prome88
Copy link
Contributor

@Prome88 Prome88 commented Dec 10, 2018

Currently we only rely on:

  • Test plan progress bar
  • Overall report -> Bug list / View all in Issue Tracker
  • Managing team workload -> some kind of real-time metrics to show progress of individual tester.
  • Managing team predictability -> re-implementation of time tracking and adding a metric that would predict the estimated end of test run etc.

@essoftdev
Copy link

@essoftdev essoftdev commented Dec 19, 2018

Great, because I've to say the reports... are pretty weak in General

** Test Run Report **
It's very useful to send the feedback to the developer but I think it's not very clear, for example Red color to mark the non-passed tests... will be very useful to just skip the passed ones

We started using the APP few weeks back, so I can not give you much more feedback, but soon I'll post more.

Thanks!

atodorov added a commit that referenced this issue Dec 22, 2018
atodorov added a commit that referenced this issue Dec 22, 2018
@okainov
Copy link
Contributor

@okainov okainov commented Jan 8, 2019

Current (okay, maybe not so current, but from Kiwi from April 2018) reports are really weak, not informative and almost useless.

However, it's still better than nothing. I'd suggest that you do not remove it unless you have proper replacement (probably designed from scratch with nicer templates and content)

@atodorov
Copy link
Member Author

@atodorov atodorov commented Jan 8, 2019

However, it's still better than nothing. I'd suggest that you do not remove it unless you have proper replacement (probably designed from scratch with nicer templates and content)

Oh we are removing those. They are a mess internally and we can't keep them around for much longer. There's no question about it. The removal will be gradual, not everything at once!

If there are specific reports that you use please tell us about them, but I'm certain that is not all of them! Whatever people don't care about will go away.

We're interested in knowing which reports and for what kind of purpose/information you are using them so we can redesign those and make the better.

@okainov
Copy link
Contributor

@okainov okainov commented Jan 8, 2019

@atodorov I use only my own reports that we created in our fork =)

@atodorov
Copy link
Member Author

@atodorov atodorov commented Jan 8, 2019

@okainov can you share more details what these reports are ? Screenshots, some description of what kind of info you are looking for, etc. If you can't share internal information that is understandable but if you can let's hear it.

@okainov
Copy link
Contributor

@okainov okainov commented Jan 9, 2019

@atodorov

Main template for our reports is table view, where each cell of the table is a test run (link).

For the "release", it's something like this - each "build" (mind that it's not the same as none of "Product version" nor "Build" but more like some substring of "Build") is tested against different OSes (which we get from Environment variable which you have removed in master already)

Of course, there is also list of bugs collected from all the test runs with bug summary, exposure, status (which is updated by cron every couple of hours by asking Jira and the second our internal bug tracker)

image

Then for each "build" you have more detailed page where the upper part is just the row from the table above, but below you have some pivot table with test results for components included to this "build" (as before, each cell is a link to test run if exist). As I already told you several times in previous issues, we heavily use Environment variables to set up these links between test runs and components included.
image

@saurabhdv
Copy link

@saurabhdv saurabhdv commented Jan 30, 2019

It would be great if we can back track the test cases that are failed, as link to test case or test case id

@DanielaPopova
Copy link

@DanielaPopova DanielaPopova commented Feb 6, 2019

Hi guys,
I'd like to make a suggestion (if you're not already considering such feature, of course) to make it possible for a test report to be exported in some kind of file format, html, xml etc.

@atodorov
Copy link
Member Author

@atodorov atodorov commented Feb 6, 2019

make it possible for a test report to be exported in some kind of file format, html, xml etc.

@DanielaPopova can you elaborate more on this? The existing format is HTML so this is already covered. Why do you need XML and how are you going to use it ?

@DanielaPopova
Copy link

@DanielaPopova DanielaPopova commented Feb 7, 2019

Say I need to send a KIWI test report to a client or upload/attach that report in some other system, so what I meant is an easy way to do that - downloading/exporting it in some kind of file format, like html/xml, archive it and use it. Currently I can show that report by sending a link to it, the person I'm sending it to has to have an account, which is not always needed and so on.

@atodorov
Copy link
Member Author

@atodorov atodorov commented Feb 7, 2019

@DanielaPopova, for the record we are talking about this https://demo.kiwitcms.org/runs/776/report/, right ?

Currently I can show that report by sending a link to it, the person I'm sending it to has to have an account, which is not always needed and so on.

we can improve the layout and you can use 'Print to PDF' which will be good for sending the report to a customer, manager or other stakeholders. In fact this page prints relatively well as it is and the new UI with Patternfly prints even better so this use-case should be solved.

Say I need to upload/attach that report in some other system, so what I meant is an easy way to do that

Again, if it is for human consumption Print to PDF will do just fine. If it is for machine consumption we need more details.

@DanielaPopova
Copy link

@DanielaPopova DanielaPopova commented Feb 7, 2019

Yes, it is for human consumption and a "fancier" pdf would be great! Thank you for your response.

@HelgaOR
Copy link

@HelgaOR HelgaOR commented Feb 20, 2019

Hi,
We are just moving to Kiwi and do not have any usage statistics yet, but we do need reports to:

  1. show current progress on test plan (I believe current "Test Execution Report" does that),
  2. show progress over latest test runs (or builds), including filed bugs, so we can tell if builds are getting better or worse from test perspective,
  3. some statistical data like "test cases that fail most often", "average failed test cases per suite", maybe some graphical information like this,
  4. also there is a separate scenario when we need to find out who ran the test case last and when it was done.

We mostly want to get some meaningful statistical info and test case/test suite dynamics so we can see current problems and address them.

@kiwitcms kiwitcms deleted a comment from saifulrip Feb 23, 2019
@atodorov
Copy link
Member Author

@atodorov atodorov commented Feb 26, 2019

@Prome88 a few questions about your very first comment:

Currently we only rely on:
Test plan progress bar

I think you mean Test Run progress bar. See picture and tell me if I am correct or show me what do you mean.

testrun_progres_report

Overall report -> Bug list / View all in Issue Tracker

Which page is this? I'm not sure I follow.

@atodorov
Copy link
Member Author

@atodorov atodorov commented Feb 26, 2019

@HelgaOR :

show current progress on test plan (I believe current "Test Execution Report" does that),

Do you mean the progress bar for Test Run as shown in the screenshot above or something else? Point me to what you mean if different.

@atodorov
Copy link
Member Author

@atodorov atodorov commented Feb 26, 2019

@essoftdev

I think it's not very clear, for example Red color to mark the non-passed tests... will be very useful to just skip the passed

To be clear: you mean that we are using only 2 colors (red for bad results) when there are several negative statuses (FAILED, BLOCKED, ERROR) ? Your objection is to how the visual information is presented, is this correct ?

@HelgaOR
Copy link

@HelgaOR HelgaOR commented Feb 26, 2019

@atodorov :

Do you mean the progress bar for Test Run as shown in the screenshot above or something else? > Point me to what you mean if different.

Yes, I meant progress bar for Test Run and report that is accessed by clicking "Report" button below it. Report is more readable than Test Run main page, in my opinion.
I'm new to your product and may use some terms incorrectly, so please forgive me if this is the case.

@Prome88
Copy link
Contributor

@Prome88 Prome88 commented Feb 26, 2019

a) Yes, you are correct. I've ment Test run progress bar.
Also linking an idea we had about a minor improvement to progress bar - #616

b) /runs/$ID/report/ -> Sections Bug list and View all in Issue Tracker

@atodorov
Copy link
Member Author

@atodorov atodorov commented Mar 3, 2019

New feature specs have been defined at http://kiwitcms.org/blog/kiwi-tcms-team/2019/03/03/legacy-reports-become-telemetry/.

Please use the comments in the blog to follow up. I will open individual issues for features where we need more information and CC those of you who have provided opinions.

The only report capability which isn't yet defined in the post above is the Environment matrix report as described by @okainov. I have the feeling this should be an extension of the TestRun status matrix telemetry but we have to implement it first to see how it can be extended with environment data.

This issue will be closed once the legacy reporting code has been removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants