New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Integrate Dashboard test results with outside services. #981

Open
jennifer-shehane opened this Issue Nov 29, 2017 · 22 comments

Comments

@jennifer-shehane
Member

jennifer-shehane commented Nov 29, 2017

We'd like to have test results integrate with outside services like:

  • Zapier
  • Slack (#578)
  • GitHub
  • Jira
  • TestRail
  • Gitlab
  • Confluence
  • Sentry
  • ??? have a service you'd like? comment below!

Our Dashboard Service knows when tests fail, it has details about a user's recorded CI run. When certain conditions are met, it can notify other integrated services and do things like:

  • Send basic pass/fail stats
  • Send Stdout (or link)
  • Send Screenshots (or link)
  • Send Video (or link)
  • Analytics like...was this test failing before? the last time it passed?
  • ??? have data you'd like? comment below!

The Dashboard should likely have a UI interface allowing the user to set what integrations they would like and what data they would like sent (on project? on organization?)

This proposal is a WIP currently, so we'd love to hear from the community on what you'd like to see.


Integrations in the pipeline:

  • Zapier
  • GitHub
@cameronc56

This comment has been minimized.

cameronc56 commented Nov 29, 2017

Hipchat would be great

@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Dec 11, 2017

Conversations in Gitter surrounding integrations:

My main goal is to have some sort of notification that a test failed on a git commit
Lots of ways to slice this, a simple solution I found was using a custom reporter but I needed to use multiples to accomplish the goal
Built in slack integration would be best
https://gitter.im/cypress-io/cypress?at=597a2d90c101bc4e3ac74704

yes that solution for slack notification used to work with my protractor tests. I changed because I have not enough information about the failure (only exit code 1). Using the afterLaunch function I have more information about which test is failing...
Thats right with cypress I can send the notification to slack using the link to the dashboard to check the failure report. I love the dashboard.
Have you got any plans to integrate with slack o email failure notifications? Have you got any idea/workarround how can I develop the notifications?
https://gitter.im/cypress-io/cypress?at=58bee213e961e53c7fb0f7b7

Is there currently a way to hook Cypress up to a github pull request? If not, will github integration be a feature at some point?
https://gitter.im/cypress-io/cypress?at=5a1ec30571ad3f873644c9f1

@frankiethekneeman

This comment has been minimized.

frankiethekneeman commented Jan 25, 2018

Would really love it if these could be implemented as a post-run/post-upload event so we can write our own service integration plugins.

@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Jan 25, 2018

@frankiethekneeman We are currently creating an integration with Zapier, I know that isn't exactly what you are asking, but would this help you integrate with the services you want?

@frankiethekneeman

This comment has been minimized.

frankiethekneeman commented Jan 25, 2018

I mean, most of the major ones, probably, but it'd be nice to be able to write my own hooks for services outside that ecosystem, or even totally custom automation, around the results of these tests.

@amirrustam

This comment has been minimized.

Member

amirrustam commented Jan 29, 2018

@jennifer-shehane I would be happy to contribute GitHub and Jira integrations. And due Atlassian Connect, the Jira work would also cover Bitbucket integration. I think this would help with #1173. I'm familiar with the APIs, and would love to help. 🙂

@acthilina

This comment has been minimized.

acthilina commented Mar 13, 2018

Cucumber integration?

@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Mar 21, 2018

@acthilina There is a cucumber plugin that integrates with Cypress you may want to check out. https://docs.cypress.io/plugins/

@acthilina

This comment has been minimized.

acthilina commented Mar 21, 2018

Thank you @jennifer-shehane , If you have more info, please send my way. :-)

@soccer99

This comment has been minimized.

soccer99 commented May 16, 2018

Gitlab please!

@amirrustam

This comment has been minimized.

Member

amirrustam commented May 16, 2018

@soccer99 Thanks for the suggestion. Gitlab currently isn't on our immediate integrations roadmap. But it might be in the future, specially if there is user demand for it. We'd be glad to hear what level of integration you're looking for.

@gscott

This comment has been minimized.

gscott commented Jun 1, 2018

Before doing too many custom integrations I’d recommend implementing outbound webhooks; they can be used to integrate just about anything.

@Nate-Watts

This comment has been minimized.

Nate-Watts commented Jun 15, 2018

We're looking for a possible integration with confluence that prints out bar charts, pass/fail, etc.

@amirrustam

This comment has been minimized.

Member

amirrustam commented Jun 18, 2018

@Nate-Watts feel free to elaborate further on the Confluence integration for the sake of documentation. We might integrate with the Atlassian stack down the line, and it would be nice to have your feedback in our considerations. Thank you.

@prescottprue

This comment has been minimized.

prescottprue commented Jul 11, 2018

Would love to see this.

I agree with @gscott on outbound webhooks. If those exist, almost anything should be able to be wired up, then the more specific integrations can come in the future (potentially built on that same webhook system).

@Shabbir124

This comment has been minimized.

Shabbir124 commented Aug 13, 2018

How we integrate Test-link to cypress? :(

@Knaledge

This comment has been minimized.

Knaledge commented Aug 15, 2018

@jennifer-shehane - is the idea with this proposal that Cypress itself will be able to integrate with all (or at least some) of the listed services? Or is this the Cypress Dashboard service integrating with the listed services?

I'm hoping for the former, though I'd understand the latter.

@jennifer-shehane

This comment has been minimized.

Member

jennifer-shehane commented Aug 15, 2018

@Knaledge The scope of this issue is for the Dashboard Service (any test using --recording), otherwise we don't have the data to integrate or know when to trigger notifications.

There is a separate issue regarding native notifications that could be triggered when running the Cypress test runner by itself here: #2145

@jennifer-shehane jennifer-shehane changed the title from Integrate Dashboard test results with outside services. to Proposal: Integrate Dashboard test results with outside services. Sep 14, 2018

@haleagar

This comment has been minimized.

haleagar commented Sep 28, 2018

This may sound silly but I'd like the dashboard to integrate with the test runner.
If I open the test runner and I'm logged in and it finds a projectId then show the last pass/fail state of the test on each row. And have a button in that row link to the report record on the dashboard.

It would be super convenient for digging in to explore the details of a fail, or to just highlight the rows you're interested in when fixing a bug and you need to re-run the failed test.

This then becomes a more of a request for the test runner, but boy I wish it kept the last run test highlighted even when I close the test window. (I know it has the greed dot but that disappears when I close the test window. It would be nice to indicate a very brief note about it's last pass fail count from local tests as well.

@haleagar

This comment has been minimized.

haleagar commented Sep 28, 2018

Now less silly, Jira would be my number 1 request for the dashboard integration.
Level 1) I give the Cypress dashboard access to my jira project and it scans test names for jira ticket ids. PROJ-#### that exist in my project and make them links to the tickets.

Level 2) First each test in my Suite get's it's own persistent record. (which your going to need for Analytics and api integrations)
There is a URL for that record I can pop into Jira to link to the test results.

Level 3a) There is a API I can hit with ID that I can query for some brief info like last run time and pass fail status, and the URL to link to. I then build a custom integration to show that in Jira

or

Level 3b) The test record gets shown in the Issue Links in the Links to section like GitLab: Mentioned on commit messages do via whatever magic does that.

@haleagar

This comment has been minimized.

haleagar commented Sep 28, 2018

Number 2 integration request is Google Docs

  1. When it is reported to the dashboard set the value of a cell in a google sheet with pass fail and cell with date of test
    That would be super useful for integrating with our manual regression test matrix which is run in a google sheet.

  2. Add a row to a sheet with pass fail, date, test name, link to dashboard
    That would allow for some history and Analytics of tests which is starkly lacking from the dashboard now. With data in a fairly pedestrian format that project managers are familiar with and can generate their own reports.

@nmchaves

This comment has been minimized.

nmchaves commented Nov 4, 2018

Having a notification for test failures would enable a really cool use case for Cypress: running "health checks" to verify that the application is operating normally in production.

Every time you deploy a new version of your application, you could have Cypress automatically visit the site and run a few smoke tests. The vast majority of tests would still run in CI, but this small set of production tests would help give you confidence that the testing environment was consistent with the prod environment. Of course, you can already do this now...but it'll be more useful when Cypress has a first-class way to notify you of test failures. Looking forward to this! ⭐️

@amirrustam amirrustam removed the help wanted label Nov 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment