Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Integrate Dashboard test results with outside services. #981

Closed
jennifer-shehane opened this issue Nov 29, 2017 · 46 comments
Closed

Proposal: Integrate Dashboard test results with outside services. #981

jennifer-shehane opened this issue Nov 29, 2017 · 46 comments

Comments

@jennifer-shehane
Copy link
Member

@jennifer-shehane jennifer-shehane commented Nov 29, 2017

We'd like to have test results integrate with third party services like:

  • Zapier
  • Slack (#578)
  • GitHub (DONE)
  • Jira
  • TestRail
  • Gitlab
  • Confluence
  • Sentry
  • Google Docs
  • PagerDuty
  • ??? have a service you'd like? You can express interest and see progress for this feature on our Roadmap from our Dashboard's product board here: https://portal.productboard.com/cypress-io/1-cypress-dashboard (Look for the card with your desired integration or Submit idea if you don't see it)

Our Dashboard Service knows when tests fail, it has details about a user's recorded CI run. When certain conditions are met, it can notify other integrated services and do things like:

  • Send basic pass/fail stats
  • Send Stdout (or link)
  • Send Screenshots (or link)
  • Send Video (or link)
  • Analytics like...was this test failing before? the last time it passed?
  • ??? have data you'd like? comment below!

The Dashboard should likely have a UI interface allowing the user to set what integrations they would like and what data they would like sent (on project? on organization?)

This proposal is a WIP currently, so we'd love to hear from the community on what you'd like to see.

You can express interest and see progress for this feature on our Roadmap from our Dashboard's product board here: https://portal.productboard.com/cypress-io/1-cypress-dashboard (Look for the card with your desired integration or Submit idea if you don't see it)

@cameronc56
Copy link

@cameronc56 cameronc56 commented Nov 29, 2017

Hipchat would be great

@jennifer-shehane
Copy link
Member Author

@jennifer-shehane jennifer-shehane commented Dec 11, 2017

Conversations in Gitter surrounding integrations:

My main goal is to have some sort of notification that a test failed on a git commit
Lots of ways to slice this, a simple solution I found was using a custom reporter but I needed to use multiples to accomplish the goal
Built in slack integration would be best
https://gitter.im/cypress-io/cypress?at=597a2d90c101bc4e3ac74704

yes that solution for slack notification used to work with my protractor tests. I changed because I have not enough information about the failure (only exit code 1). Using the afterLaunch function I have more information about which test is failing...
Thats right with cypress I can send the notification to slack using the link to the dashboard to check the failure report. I love the dashboard.
Have you got any plans to integrate with slack o email failure notifications? Have you got any idea/workarround how can I develop the notifications?
https://gitter.im/cypress-io/cypress?at=58bee213e961e53c7fb0f7b7

Is there currently a way to hook Cypress up to a github pull request? If not, will github integration be a feature at some point?
https://gitter.im/cypress-io/cypress?at=5a1ec30571ad3f873644c9f1

@frankiethekneeman
Copy link

@frankiethekneeman frankiethekneeman commented Jan 25, 2018

Would really love it if these could be implemented as a post-run/post-upload event so we can write our own service integration plugins.

@jennifer-shehane

This comment has been hidden.

@frankiethekneeman
Copy link

@frankiethekneeman frankiethekneeman commented Jan 25, 2018

I mean, most of the major ones, probably, but it'd be nice to be able to write my own hooks for services outside that ecosystem, or even totally custom automation, around the results of these tests.

@amirrustam
Copy link
Member

@amirrustam amirrustam commented Jan 29, 2018

@jennifer-shehane I would be happy to contribute GitHub and Jira integrations. And due Atlassian Connect, the Jira work would also cover Bitbucket integration. I think this would help with #1173. I'm familiar with the APIs, and would love to help. 🙂

@acthilina2020
Copy link

@acthilina2020 acthilina2020 commented Mar 13, 2018

Cucumber integration?

@jennifer-shehane
Copy link
Member Author

@jennifer-shehane jennifer-shehane commented Mar 21, 2018

@acthilina There is a cucumber plugin that integrates with Cypress you may want to check out. https://docs.cypress.io/plugins/

@acthilina2020
Copy link

@acthilina2020 acthilina2020 commented Mar 21, 2018

Thank you @jennifer-shehane , If you have more info, please send my way. :-)

@soccer99
Copy link

@soccer99 soccer99 commented May 16, 2018

Gitlab please!

@amirrustam
Copy link
Member

@amirrustam amirrustam commented May 16, 2018

@soccer99 Thanks for the suggestion. Gitlab currently isn't on our immediate integrations roadmap. But it might be in the future, specially if there is user demand for it. We'd be glad to hear what level of integration you're looking for.

@gscott
Copy link

@gscott gscott commented Jun 1, 2018

Before doing too many custom integrations I’d recommend implementing outbound webhooks; they can be used to integrate just about anything.

@salty-blue-mango
Copy link

@salty-blue-mango salty-blue-mango commented Jun 15, 2018

We're looking for a possible integration with confluence that prints out bar charts, pass/fail, etc.

@amirrustam
Copy link
Member

@amirrustam amirrustam commented Jun 18, 2018

@Nate-Watts feel free to elaborate further on the Confluence integration for the sake of documentation. We might integrate with the Atlassian stack down the line, and it would be nice to have your feedback in our considerations. Thank you.

@prescottprue
Copy link

@prescottprue prescottprue commented Jul 11, 2018

Would love to see this.

I agree with @gscott on outbound webhooks. If those exist, almost anything should be able to be wired up, then the more specific integrations can come in the future (potentially built on that same webhook system).

@Shabbir124
Copy link

@Shabbir124 Shabbir124 commented Aug 13, 2018

How we integrate Test-link to cypress? :(

@Knaledge
Copy link

@Knaledge Knaledge commented Aug 15, 2018

@jennifer-shehane - is the idea with this proposal that Cypress itself will be able to integrate with all (or at least some) of the listed services? Or is this the Cypress Dashboard service integrating with the listed services?

I'm hoping for the former, though I'd understand the latter.

@jennifer-shehane
Copy link
Member Author

@jennifer-shehane jennifer-shehane commented Aug 15, 2018

@Knaledge The scope of this issue is for the Dashboard Service (any test using --recording), otherwise we don't have the data to integrate or know when to trigger notifications.

There is a separate issue regarding native notifications that could be triggered when running the Cypress test runner by itself here: #2145

@jennifer-shehane
Copy link
Member Author

@jennifer-shehane jennifer-shehane commented Feb 22, 2019

@bdim Could you open a pull request to add this plugin to our documentation? Thanks! Instructions here.

@bdimitrovski
Copy link

@bdimitrovski bdimitrovski commented Feb 22, 2019

@nir0s
Copy link

@nir0s nir0s commented Mar 11, 2019

PagerDuty please :)

@pareshkrc
Copy link

@pareshkrc pareshkrc commented Mar 19, 2019

integration with qTest https://www.qasymphony.com/

@trydionel
Copy link
Member

@trydionel trydionel commented Mar 29, 2019

Hi all! We're continuing to make progress on adding GitHub integration to the Cypress Dashboard and would love to get some hands-on user feedback at this point. Does anyone have a few minutes to talk through the designs with me? You'd get a chance to try out this new integration in your own project, and you'll get to keep access to the feature afterwards!

Please leave a comment here, or email me directly if you're interested. Thank you!

@salty-blue-mango
Copy link

@salty-blue-mango salty-blue-mango commented Mar 29, 2019

@nir0s
Copy link

@nir0s nir0s commented Mar 29, 2019

@prescottprue
Copy link

@prescottprue prescottprue commented Mar 29, 2019

@trydionel Same here!

@trydionel
Copy link
Member

@trydionel trydionel commented Mar 29, 2019

Awesome, thank you so much @Nate-Watts, @nir0s, @prescottprue! Just to confirm, you'll need to have 1) a project recording to the Cypress Dashboard, and 2) the repo for this project hosted on github.com, either publicly or privately. If you meet those criteria, please grab a time slot on my calendar:

https://calendly.com/trydionel/github-integration-research?month=2019-04

Looking forward to talking soon!

@freedomflyer
Copy link

@freedomflyer freedomflyer commented May 14, 2019

@trydionel would also love to hop on this beta, we meet the criteria

@trydionel
Copy link
Member

@trydionel trydionel commented May 15, 2019

@freedomflyer Sure thing! Can you get in touch with me at the email listed on my GitHub profile?

@sravan-s
Copy link

@sravan-s sravan-s commented Jul 4, 2019

+1 to the webhooks, it would be great if we can have webhooks with information such as -

  • Job xxx has finished
  • Summary of the job (success/fail/skip count, time taken etc)
  • Link to get detailed information about the job
@timiyay
Copy link

@timiyay timiyay commented Jul 4, 2019

We use Flowdock for our company chat, so this integration would be useful for us.

@amirrustam
Copy link
Member

@amirrustam amirrustam commented Jul 29, 2019

👋 Hi everyone, we're excited to release GitHub Integration for the Cypress Dashboard today. The Cypress GitHub App tightly integrates with your workflow via commit status checks and pull-request comments. Read all about it below, and let us know what you think (@ me).

Blog post: https://cypress.io/blog/2019/07/29/github-integration-for-the-cypress-dashboard/
Docs: https://on.cypress.io/github-integration.

@mitar
Copy link

@mitar mitar commented Jul 29, 2019

Awesome! :-) Please do GitLab next. :-)

@aequasi
Copy link

@aequasi aequasi commented Aug 1, 2019

Love the integration, but the experience of using it with Slack needs some love.

Right now, there are too many images being used in the commit reply, and it Slack, the message is basically a big block of image links. I like to suggest the use of emoji over the icons that are shown there and the removal of the header image.

image
image

@thomaspicquet
Copy link

@thomaspicquet thomaspicquet commented Aug 5, 2019

Love the integration, but the experience of using it with Slack needs some love.

Right now, there are too many images being used in the commit reply, and it Slack, the message is basically a big block of image links. I like to suggest the use of emoji over the icons that are shown there and the removal of the header image.

It's great to be able to see the Cypress test results on the PR page but as @aequasi mentioned, it makes the integration very noisy in slack and pretty much unusable.

@trydionel
Copy link
Member

@trydionel trydionel commented Aug 5, 2019

Hey @aequasi, @thomaspicquet,

Yea, that's a bit rough 😬 Would you mind to answer some questions about your workflow?

  • Are you using Slack as your primary collaboration hub, or is this comment just noisy compared to others?
  • If we provided direct Slack integration, would you opt into that feature over PR comments?
  • We've also looked into GitHub's "checks" API to place Cypress results into each PR's "Checks" tab. Do you currently use PR "Checks"? If so, would you prefer that Cypress post results there instead of as a PR comment?

Thanks for the feedback!

PS: If you have any other feedback for the Cypress Dashboard, or are just curious about what we're working on, please check out our updated roadmap at https://portal.productboard.com/cypress-io/1-cypress-dashboard/tabs/1-under-consideration.

@thomaspicquet
Copy link

@thomaspicquet thomaspicquet commented Aug 6, 2019

Hey @trydionel,

We use Slack for a lot of things and we have a channel for our CI with:

  • Github pull requests and status checks (circleci / cypress)
  • We use pull reminders (which was just acquired by GitHub) to see the messages on the pull requests and get reminders about code reviews

It's nice to see the Cypress PR message and I'd like to keep that over the Checks tab on github but the message is really big and not really useful when it gets to Slack, unfortunately.
A slack integration sounds good or maybe an option for the Github integration to turn off the message?

@trydionel
Copy link
Member

@trydionel trydionel commented Aug 6, 2019

Cool, I've added your feedback to our product backlog. Thanks for the info!

Also, if you find the PR comments too noisy, you can turn them off independently of the commit status checks. See our documentation.

@arifmasood
Copy link

@arifmasood arifmasood commented Sep 26, 2019

@jennifer-shehane @amirrustam @trydionel +1 for sending cypress run reports to designated emails.

It will be great if we can have summarized results of cypress run (from dashboard) being delivered to specific email addresses. Raising a red alarm in case of any failures or sending a green report in case tests passed.

@jennifer-shehane
Copy link
Member Author

@jennifer-shehane jennifer-shehane commented Oct 28, 2019

You can express interest and see progress for this feature on our Roadmap from our Dashboard's product board here: https://portal.productboard.com/cypress-io/1-cypress-dashboard (Look for the card with your desired integration or Submit idea if you don't see it)

I will be closing this issue and closing it to all comments since this is related to our Dashboard.

All related work for Dashboard features is handled in that ProductBoard and will be addressed by the Dashboard team directly when you comment there. We have already noted the initial comments from this issue there. Please refer all future comments there. Thank you.

@cypress-io cypress-io locked and limited conversation to collaborators Oct 28, 2019
@jennifer-shehane
Copy link
Member Author

@jennifer-shehane jennifer-shehane commented Jun 8, 2020

Slack integration is now available within our Cypress Dashboard. It allows you to see your Cypress test results directly in your teams’ Slack channels. Please read our Slack Integration docs to get started.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet