Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report progress of individual test cases #6616

Closed
gaearon opened this issue Jul 4, 2018 · 56 comments · Fixed by #10227
Closed

Report progress of individual test cases #6616

gaearon opened this issue Jul 4, 2018 · 56 comments · Fixed by #10227
Milestone

Comments

@gaearon
Copy link
Contributor

gaearon commented Jul 4, 2018

🚀 Feature Proposal

It seems like Jest currently only reports progress when a complete suite passes.
I propose that instead, it should report progress every second for individual test cases.

Otherwise, the progress output is confusing. "Tests: N" being a separate line in the output gives the impression that it's just as granular as "Test Suites: N" and will be incremented immediately when a test case passes.

Motivation

This is confusing for large/slow suites because you don't get any sense of how much time is left and whether tests are stuck for some reason.

demo

@SimenB
Copy link
Member

SimenB commented Jul 4, 2018

Sorta related: #6225. But this has an actual use case attached.

@gaearon
Copy link
Contributor Author

gaearon commented Jul 4, 2018

Btw if you want to reproduce, clone facebook/prepack, run yarn and then yarn test-react --watch.

@SimenB
Copy link
Member

SimenB commented Jul 4, 2018

We don't know the number of tests before executing the test files (and its describes). But populating that information might be possible at that time?

@aaronabramov thoughts?

@jacklinj066

This comment has been minimized.

@gaearon
Copy link
Contributor Author

gaearon commented Jul 5, 2018

I guess my mental model was that each test() or it() call goes into Jest, so Jest should be able to “ping” the watcher about its progress on every exit from an it().

@aaronabramov
Copy link
Contributor

yeah, we should be able to know everything about the number of tests we have once we executed the code in the test file (but before running it)
once we stream the results it should be a one line change to implement that!

@gaearon
Copy link
Contributor Author

gaearon commented Jul 5, 2018

Why is the total number of the tests intersting?

I want to clarify my feature request isn’t about seeing a total number of tests before they run. It’s about seeing the passed number of tests change as each of them completes.

@aaronabramov
Copy link
Contributor

yeah, i agree. we don't need it.
we should just report a single test and increment a counter once it or test is done

@rickhanlonii
Copy link
Member

+1, with this I can finish a port of the ava mini reporter to Jest

Here's the current reporter interface:

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  // these are actually for the test suite, not individual test results
  onTestStart(test: Test) {}
  onTestResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

The ideal for me would be (note some of the types are either renamed of do not exist yet):

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}
 
  onTestSuiteStart(suite: TestSuite) {}

  // new: called at the beginning of a .test() or .it() with the test meta info
  onTestStart(test: Test) {}

  // new: called at the end of a .test() or .it() the test meta info and result info
  onTestResult(suite: TestSuite, test: Test, testResult: TestResult, results: AggregatedResult) {}

  onTestSuiteResult(suite: TestSuite, testSuiteResult: TestSuiteResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

Until a major we could call a "test" the test suite and "assertion" an individual test (which pairs with some of the types we use):

export default class BaseReporter {
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  onTestStart(test: Test) {}
  
  onAssertionStart(assertion: Assertion) {} // note this type doesn't exist

  onAssertionResult(test: Test, assertion: Assertion, assertionResult: AssertionResult, results: AggregatedResult) {}

  onTestSuiteResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

@aaronabramov
Copy link
Contributor

i'm not sure about using the word "assertion" this way.
Right now we have a huge term overload. i think almost everyone at facebook uses "test" to describe the whole file (unlike test suite that we normally use in jest).
then i normally hear the term "test case" to describe individual test or it and assertion to describe an expect or assert call.

@rickhanlonii
Copy link
Member

rickhanlonii commented Jul 5, 2018

Yeah that makes sense, I like "test case" and that doesn't require renaming anything. I only used "assertion" because the type of a test case result is called AssertionResult

Here's what we can go with then:

export default class BaseReporter {
  // Called at the beginning of a run
  onRunStart(results: AggregatedResult, options: ReporterOnStartOptions) {}

  // Called at the beginning of every test file
  onTestStart(test: Test) {}
  
  // Called at the beginning of every .test or .it
  onTestCaseStart(test: Test, testCase: TestCase) {}

  // Called with the result of every .test or .it
  onTestCaseResult(test: Test, testCase: TestCase, testCaseResult: AssertionResult) {}

  // Called with the result of every test file
  onTestResult(test: Test, testResult: TestResult, results: AggregatedResult) {}

  // Called after all tests have completed
  onRunComplete(contexts: Set<Context>, results: AggregatedResult ): ?Promise<void> {}
}

@SimenB
Copy link
Member

SimenB commented Jul 7, 2018

Could we call it Spec instead of TestCase? Matches Jasmine (https://jasmine.github.io/api/3.0/global.html#it).

FWIW the Jest docs calls a single it/test a "test", nowhere I can see is test meant to be the whole file

@gaearon
Copy link
Contributor Author

gaearon commented Jul 7, 2018

nowhere I can see is test meant to be the whole file

👍 I think calling the whole file a "test" is a Facebook-ism, I've never seen this anywhere else. I'd love if Jest stopped using this terminology.

@aaronabramov
Copy link
Contributor

makes sense! if we clean up onTestResult methods and rename them to be onTestSuiteResult we can make it consistent with documentation (and reporters output).
so pretty much what @rickhanlonii wrote earlier:

test_run - all tests/test suites in all runners
test_suite - one file
test - one it/test
assertion - one expect call

@aaronabramov
Copy link
Contributor

hey.. what if we renamed "test suite" to "test file"?
i think this is exactly what it is and it should remove all the confusion.

the conversations i hear all the time (mostly outside fb)

  • we have X tests in our codebase
  • like.. test suites or actual tests?
  • yeah.. suite|tests

@aaronabramov
Copy link
Contributor

@captbaritone you might have some thought on this as well

@thymikee
Copy link
Collaborator

I'm actually fot "test file". I often hear a "test suite" referring to all tests in the project, e.g. "We have 100+ tests in our suite."

@captbaritone
Copy link
Contributor

Yes! I mostly agree with @aaronabramov, but I think leaving it/test as "test" is probably a bit ambiguous. Was there a good reason not to use "test case", as @rickhanlonii proposed above?

  • "Test Run" - all tests/test suites in all runners
  • "Test File" - one file
  • "Test Case" - one it/test
  • "Assertion" - one expect call

The only thing that this leaves is what to call describe(() => {})s. Maybe it's okay for us not to have a name for this, but if we think it's important to have one, the only thing I can think of is "describe block". If we want to use that name, then maybe an it/test should be a "test block" so that we are consistent.

@SimenB I'm not a fan of "spec" since it seems tightly coupled to the Behavior Driven Development methodology encouraged by Jasmine, and I don't think Jest really prescribes/endorses BDD.

@SimenB
Copy link
Member

SimenB commented Jul 10, 2018

I still think "test" is best because, well, the function is named test. I could get behind "test case" as well, the only thing I don't like is "test" being the entire file. "Test file" is a good compromise, and it's explicit (which is 👍).

Not sure what to do about describe (or before{All,Each}, after{All,Each} for that matter). "describe block" makes sense, and consistency with "test block" sounds good.

@aaronabramov
Copy link
Contributor

so:

testRun - all files in all runners
testFile - one file
test - one test/it
describeBlock - one describe
assertion - one expect call

i like that!

we already use DescribeBlock in jest-circus https://github.com/facebook/jest/blob/master/types/Circus.js#L182

@captbaritone
Copy link
Contributor

I'm not crazy about leaving "test" as just "test", even though that's the name of the function. There are lots of people, Facebook people included, who have a existing definitions for what "test" means to them. Additionally, for those still using it(), the relationship may not be as obvious.

If we do opt to use "test", then maybe we can just call a describe() a "describe"? That feels consistent with calling a test() a test.

Just my $0.02.

@gaearon
Copy link
Contributor Author

gaearon commented Jul 10, 2018

My choice would be "test case" and "test file".

@SimenB
Copy link
Member

SimenB commented Aug 19, 2018

I started fiddling with this, and an early snag is that Runner.runTests already takes 6 arguments. Adding more makes it really unwieldy, but changing to passing a single object is a breaking change.

Should we do something about its signature at the same time, or just add more args?

See https://jestjs.io/docs/en/configuration#runner-string

Might make sense to expect runners to be event emitters, then we can just attach listeners instead of passing in callbacks. Could add a property (similar to isSerial) called emitsEvents or something, and fork our logic based on that.

/cc @rogeliog for thoughts on runner api

@hegelstad
Copy link

Jest is really lacking on this part.

Is adding custom JasmineReporters really the only option? The documentation on jasmine (in Jest) is really lacking as jest is rolling it's own jasmine bundle.

I propose documenting where users can find valid jasmine documentation for implementing this themselves.

@SimenB
Copy link
Member

SimenB commented Aug 20, 2018

We're removing the jasmine coupling, so that's no way forward at all

@SimenB
Copy link
Member

SimenB commented Sep 27, 2018

Looked again at this, and just went with an event emitter as that seemed more sane.

However, I'm having an issue with the test running inside a worker with no way of reporting back to jest until it's done. Any ideas? @mjesun thoughts on a worker posting messages to its parent?

My in progress code, if anyone is interested: https://github.com/SimenB/jest/tree/report-more-events

@SimenB
Copy link
Member

SimenB commented Jul 30, 2020

This is finally out in 26.2.0! Note that it only works if you're using jest-circus

@SimenB
Copy link
Member

SimenB commented Jul 30, 2020

@tralston
Copy link

tralston commented Aug 5, 2020

Just tried it, and it didn't work.

npm install jest-circus --save-dev
npm update jest

Confirmed that package.json shows jest dependency as "jest": "^26.2.2"

ran npx jest --testRunner='jest-circus/runner'

A particular file I have has ~30 tests, and it showed the same behavior as before. It's not listing the progress of each individual test case.

Am I doing something wrong?

@retorquere
Copy link

I'm seeing the same problem as @tralston. I'm still seeing all tests output at once when the last one finishes, and nothing before. I've create an MWE at https://github.com/retorquere/jest-circus-test. Would appreciate feedback on what I should be doing differently.

@SimenB
Copy link
Member

SimenB commented Aug 5, 2020

@retorquere there's no summary printed when running a single test, see #9420 (comment). Just duplicating your one test fixes the issue.

in-progress

Not sure it makes sense to hide the summary anymore now since we are able to report stuff as we go

@SimenB SimenB modified the milestones: High priority future, Jest 26.x Aug 5, 2020
@retorquere
Copy link

Then I don't understand what's different now since 26.2.2. In my experience jest has always reported per-file tests this way, even without jest-circus. My use-case is exactly as in my MWE - I generate cases based on sample files, which I now do by pre-generating a test js file per sample, and then run jest, but I'd love to get rid of that.

@tralston
Copy link

tralston commented Aug 6, 2020

I guess I misunderstood the purpose of this issue. I never really had a problem with the totals line (e.g. "6 passed, 6 total") being updated as tests passed or failed. I wanted to see (as I thought the title of the issue was reporting) the name and pass/fail of each individual test for each test file, as it's executed. For example, I have a small project with ~50 tests. 40 of them are in a big file, and that's the one we are working on the most. It currently takes ~30 seconds to execute (due to some slow dependencies that eventually I'll mock out). Most of the time I run jest just watching the file I'm working on. It would be really nice to see each test printed out as it's executed, not just the summary updated. Here's an example of what I'm talking about from (Gradle test runner, but with a Mocha theme):

I realize this may not be the right issue for what I'm asking. If not, are you familiar with one like this? Or should I file a new one? I do understand that running tests asynchronously could present some challenges in displaying live results.

@dipasqualew
Copy link

@tralston Did you get anywhere with indvidual test reporting while they are being executed? I am looking for the same functionality.

@tralston
Copy link

@dipasqualew Unless something changed (@SimenB would know best) this is still a problem for me.

@SimenB
Copy link
Member

SimenB commented Mar 15, 2021

This issue was as far as I know always about "it feels like nothing is happening" which a live counter fixes. Getting each individual test to show up in the summary as they complete is a separate feature IMO. Feel free to open up an issue or send a PR. The data used for updating the counter should be enough to update the verbose reporter as results come in as well (although I'm not entirely sure how that will work with tests running in parallel - might need to use something like ink)

@dipasqualew
Copy link

@SimenB However the live counter only updates when the test case completes and not after each individual test. Do you think that's also out of scope of this issue and part of an extension of the verbose reporter as well?

@SimenB
Copy link
Member

SimenB commented Mar 15, 2021

Practically all bug reports/feature requests are out of scope in a closed issue. 🙂

It should update after every test, if it does not please open up a separate issue with a reproduction

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 10, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.