Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recommended "results dashboard" solutions ? #4640

Closed
david-fliguer-ws opened this issue Jan 13, 2019 · 5 comments
Closed

Recommended "results dashboard" solutions ? #4640

david-fliguer-ws opened this issue Jan 13, 2019 · 5 comments
Labels
type: question general question, might be closed after 2 weeks of inactivity

Comments

@david-fliguer-ws
Copy link

Hi,

Right now we get the results from our automated tests by email, where the email has an HTML attached to it that we generate using pytest-html plugin.

We are thinking about moving to some kind of "dashboard" where we could ideally:

  • Have an history of our different runs
  • Search easily runs
  • Get some statistics
  • See the results of the run in "live time"

Is important that it supports the different pytest results (skipped, xfailed,xpass) and also ideally running in parallel (We plan to do this using pytest-xdist)

Any recommendation about this ? I've heard about the following

  • reportportal
  • pytest-ochrus ? (I'm kind of doubting because I see that the project has 4 commits and last one is from 2018...)

Any help regarding this would be appreciated

Thanks and keep the good work !!!

David

@Zac-HD Zac-HD added the type: question general question, might be closed after 2 weeks of inactivity label Jan 14, 2019
@pfctdayelise
Copy link
Contributor

Are you using a CI server? I feel like it's more common that the CI server would have the function (or plugin) to keep history of changes of test results.

@david-fliguer-ws
Copy link
Author

Hi,

Yes we run our tests in Jenkins, but as far as I know Jenkins has a really basic plugin that stores the results in XML files and displays it on Jenkins itself (I've used a little bit and I didn't like it)

I am looking for a more powerful platform that also allows for searchs, maybe comparison between runs, etc...

@Zac-HD Zac-HD closed this as completed Feb 25, 2019
@david-fliguer-ws
Copy link
Author

Hi, can this be re-opened ? I am still looking for a good solution...

@pfctdayelise
Copy link
Contributor

Hi David,

I think you are looking for something distinct enough that it is outside the scope of core pytest, and I would suggest outside the scope of pytest plugins as well. This product is probably called something like a "test report dashboard" or "test results management" tool.

Have you looked for different Jenkins plugins? Are you using the Test Results Analyzer plugin? https://wiki.jenkins.io/display/JENKINS/Test+Results+Analyzer+Plugin

If you want to use a different tool, the way to do it would likely be: have pytest create junitxml output (this is a defacto standard for test output), have your Jenkins job post this XML to whatever the server for your test report dashboard is.

A few tools I'm aware of:

The pytest team can't solve this problem for you, I'm sorry. But if you find a good solution I am sure we would be interested if you report back what you found.

@adamhasham
Copy link

@david-fliguer-ws interested to hear what you ended up using!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: question general question, might be closed after 2 weeks of inactivity
Projects
None yet
Development

No branches or pull requests

4 participants