Allow to rerun only failed tests directly from the Test Runner UI #4886
Comments
|
There's a plugin that can execute your failed test |
|
@jennifer-shehane and @Superkunair you both refer to issues and solutions to handle flaky tests by rerunning them. That's not what I'm looking for. I'm looking for a feature in the Test Runner UI. The use case is:
So this request is really for an improved developer experience while investigating issues, not as a solution to handle flaky tests. |
|
What you're referring to is what we internally call |
cypress-bot
bot
added
the
stage: ready for work
|
Yes, just tagging similar requests - this is a new kind of request which is why the issue was left open. |
jennifer-shehane
added
type: feature
That is for re-running during execution to account for network issues? We want a button in the runner marked "re-run failed tests only) to allow for debugging. Don't confuse the issue. |
THIS! It is a really super simple and basic feature. Every IDE has it. Only Cypress runner fails to provide such a simple thing! |
Neither of those issues relates to this issue in any way. |
|
I can agree with @DavidGWheeler, we're looking forward to see this feature too. Currently there isn't any way to rerun just failed test after fixing it. |
jennifer-shehane
changed the title
cypress-bot
bot
added
stage: proposal 💡
|
I agree this would be an immensely helpful feature. |
|
I've found two options:
|
|
This will be super helpful |
|
This is a must for test-driven development. For me jest in watch mode (via vue-cli) has the ideal behaviour, it goes one step further and only re-runs the tests affected by code changes since the last time they ran (so that if there are 5 failing tests and I modify 1 of them only that one will re-run). However I don't want to confuse the issue and as the OP suggested, an option for only re-running failing tests is a perfect first step. |
|
While I don’t agree with @DavidGWheeler 's statement about it being a feature that is both simple and easy (if it’s so easy, please submit a PR), I think this feature is really essential to the workflows on projects I’m working on. I was really excited about the Cypress 5.0 when I read about the retry ability - but this is really not what I had in mind. I think having the ability to re-run a single failed test, or multiple failed tests in a row, would really help. On a test suite of 130 tests that take 20 minutes to run, seeing 13 failed tests (completely separated) and re-running them is a nightmare once I changed the behavior to fix them. |
|
In some other testing frameworks I've used they have this feature built-in (RSpec, Cucumber) When the test fails it will create a file (eg. This feature is not about mitigating flaky tests, but for TDD or auto-focusing test for fixing purpose. |
|
Hi ! As I filled in #17803, it would nice to have TDD mode : GREEN, RED, REFACTOR which will make Cypress behaves differently depending to the mode. |
|
Is there any update on this issue? I've seen a number of tickets related to this but it's not clear if this functionality is going to be added or where to track the progress. |
|
@jennifer-shehane Sorry to "me too" a feature request, but I think jest has made this an expected feature in a test runner (and much more important for integration test suites that can take multiple minutes to run in the average case scenario). Since it has been open for two years and other tickets are being closed as duplicates, is this currently on the roadmap? |
|
An update on this or a solution - it is soul destroying having to re-watch the entire set of tests to get to the failed ones for each iteration whereby you are trying to fix the problem |
|
Please, bring the HMR test rerun! |
|
For people that center their work on TDD, this feature would be really helpful. |
Agreed, this is extremely painful for a TDD workflow. |
…ibute of an attribute when sorting
To sort the list in place:
```python
ut.sort(key=lambda x: x.count, reverse=True)
```
To return a new list, use the `sorted()` built-in function:
```python
newlist = sorted(ut, key=lambda x: x.body.id_, reverse=True)
```
feat(cypress): Add Cypress commands
For the functions you write a lot you can use commands in `/cypress/support/commands.ts`.
```javascript
Cypress.Commands.add('getById', (selector, ...args) => {
return cy.get(`[data-cy=${selector}]`, ...args)
})
Cypress.Commands.add('getByIdLike', (selector, ...args) => {
return cy.get(`[data-cy*=${selector}]`, ...args)
})
Cypress.Commands.add('findById', {prevSubject: true}, (subject, selector, ...args) => {
return subject.find(`[data-cy=${selector}]`, ...args)
})
```
So you can now do
```javascript
cy.getById('submit')
```
feat(cypress#Select by position in list): Add more ways to select elements
* Select by position in list
Inside our list, we can select elements based on their position in the list,
using `.first()`, `.last()` or `.eq()` selector.
```javascript
cy
.get('li')
.first(); // select "red"
cy
.get('li')
.last(); // select "violet"
cy
.get('li')
.eq(2); // select "yellow"
```
You can also use `.next()` and `.prev()` to navigate through the elements.
* Select elements by filtering
Once you select multiple elements, you can filter within these based on another selector.
```javascript
cy
.get('li')
.filter('.primary') // select all elements with the class .primary
```
To do the exact opposite, you can use `.not()` command.
```javascript
cy
.get('li')
.not('.primary') // select all elements without the class .primary
```
feat(cypress#Finding elements): Finding elements
You can specify your selector by first selecting an element you want to search
within, and then look down the DOM structure to find a specific element you are
looking for.
```javascript
cy
.get('.list')
.find('.violet') // finds an element with class .violet inside .list element
```
Instead of looking down the DOM structure and finding an element within another
element, we can look up. In this example, we first select our list item, and
then try to find an element with a `.list` class.
```javascript
cy
.get('.violet')
.parent('.list') // finds an element with class .list that is above our .violet element
```
feat(cypress#Asserting about elements): Assert on the content of an attribute
```javascript
cy
.get('a')
.invoke('attr', 'href')
.should('eq', 'https://docs.cypress.io')
```
feat(cypress#Use the content of a fixture set in a hook in a test): Use the content of a fixture set in a hook in a test
If you store and access the fixture data using this test context object, make
sure to use `function () { ... }` callbacks both for the hook and the test.
Otherwise the test engine will NOT have this pointing at the test context.
```javascript
describe('User page', () => {
beforeEach(function () {
// "this" points at the test context object
cy.fixture('user').then((user) => {
// "this" is still the test context object
this.user = user
})
})
// the test callback is in "function () { ... }" form
it('has user', function () {
// this.user exists
expect(this.user.firstName).to.equal('Jane')
})
})
```
feat(cypress#issues): Run only failing tests
Cypress doesn't [Allow to rerun failed tests](cypress-io/cypress#4886) but you can use `it.only` on the test you want to run.
feat(vuejs#Make HTTP requests): Make HTTP requests with Vue
Compare [Fetch API](vuejs.md#fetch-api) and [Axios](vuejs.md#axios) when
doing http requests to external services.
Explain how to do them with both methods and arrive to the conclusion
that if you’re working on multiple requests, you’ll find that Fetch requires you to
write more code than Axios, even when taking into consideration the setup needed
for it. Therefore, for simple requests, Fetch API and Axios are quite the same.
However, for more complex requests, Axios is better as it allows you to
configure multiple requests in one place.
If you're making a simple request use the Fetch API, for the other cases use axios because:
* It allows you to configure multiple requests in one place
* Code is shorter.
* It allows you to [place all the API calls under services so that these can be
reused across components wherever they are
needed](https://medium.com/bb-tutorials-and-thoughts/how-to-make-api-calls-in-vue-js-applications-43e017d4dc86).
* It's easy to set a timeout of the request.
* It supports HTTP interceptors by befault
* It does automatic JSON data transformation.
* It's supported by old browsers, although you can bypass the problem with fetch
too.
* It has a progress indicator for large files.
* Supports simultaneous requests by default.
Axios provides an easy-to-use API in a compact package for most of your HTTP
communication needs. However, if you prefer to stick with native APIs, nothing
stops you from implementing Axios features.
For more information read:
* [How To Make API calls in Vue.JS Applications by Bhargav Bachina](https://medium.com/bb-tutorials-and-thoughts/how-to-make-api-calls-in-vue-js-applications-43e017d4dc86)
* [Axios vs. fetch(): Which is best for making HTTP requests? by Faraz
Kelhini](https://blog.logrocket.com/axios-vs-fetch-best-http-requests/)
|
Any update on this feature? |
|
This request should not be limited to "failed" tests. While iteratively writing a test, it is a must to be able to tweak a single test then rerun it, whether it failed or not. |
|
@adamvanaken For that behavior you can add |
|
@danhooper that's fine, but when running several specs (not possible on cypress 10 anymore), it would be very helpful to re-run only the failed ones. This way you shorten your list for each test fixed until you have all fixed while avoiding rerunning tests that already passed |
|
@Carniatto Agreed. My response was only to help with when you are developing tests. I very much want the ability to rerun failed tests. |
|
did you find any solution for this brother, even I am looking for this |
|
@jennifer-shehane any update on this its nearly 3 years still its open |
|
@bahmutov wow this was very useful this opens a way to further investigate on how to integrate this in CI CD thank you :) |
|
This is awesome. |
|
I Hope this solution will get integrated into command too so that we can use it via CI CD |
emilyrohrbough
added
pkg/reporter
|
I have a scenario in which I need to run my tests, then I need to restore the database, and finally run it again but only the failed ones from the previous run. I was thinking in something like a file that is generated after a “cypress run”, so next time we run “cypress run” again, Cypress will first check if such file is already generated. If so, Cypress will run only the test cases listed in this specific file. I took some time investigating some options on the internet, and I don’t see exactly something that fits it. I know this PR is about the Test Runner over the UI, but it would be great to have it like I described for a CI CD approach. |
nagash77
added
the
pkg/desktop-gui
|
This is quite important functionally that Cypress lacks, and it's still only in proposal stage for 5 years. |
|
Our team also suffering without the solution. I would say we need it everywhere: in CI/CD, while running in a headless mode locally as well as through UI. |
and others
Current behavior:
In the test runner, there is no option to only rerun failed tests. After investigating and fixing a failed test, I'd like the watcher to only rerun failed tests.
Desired behavior:
Add a rerun "Rerun failed tests only" checkbox to the tests runner UI.
The text was updated successfully, but these errors were encountered: