-
-
Notifications
You must be signed in to change notification settings - Fork 6.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Proposal] Request to add support for CI/CD pipelines for Test reporting #7163
Comments
There is /cc @palmerj3 |
@nigurr thanks for the suggestion - note that we don't really consider jest-community packages as third-party as they're maintained by the jest core team and contributors in the community I don't agree that the jest-junit code should be included in jest core because most users will not use it. Installing jest-junit and including it in the builds you want it output in (either with the config or by command line |
Also, it would be great to update the description to match our Feature Proposal template |
@rickhanlonii @SimenB thanks for your inputs. We can debate on whether Jest-Junit should become the core of Jest or not? MotivationFriction less user test reporting experience in CI/CD pipeline. In general, users doesn't require report file in local dev environment. Wouldn't it be great if CI/CD solution just gets all the results and all user has to do is kick off the pipeline with bare minimum configuration. For example: If we consider Jest CI configuration, We added extra steps and configuration to get the results pumping out. It would be much easier to lift that from user. I really appreciate the feedback, love to be part of this conversation and making better experiences for user. |
How would we know where to put the resulting xml file for CI to pick it up? One thing is telling Jest to output the report (easy enough), another thing is how to get the CI service to pick it up (potentially painful, and not scalable) |
@SimenB jest-junit supports specifying the output file via If it's possible to do this with no configuration change for the user (for example without editing circle.yml to add envars) then I'm v interested in exploring @nigurr do you expect CI systems to make a change to provide these envars by default? |
I wouldn't be against adding jest-junit directly to jest. I just don't buy the argument that it's a requirement. |
Somewhat related: #3536 |
@SimenB @rickhanlonii this can be driven through Environment variable. The environment variable CI/CD systems will set the required environment variables by default, user can still override through their scripts. Since Multiple test runners across language are supporting Junit and pretty much becoming standard report, I would favor JUnit is being the default reporting schema. |
@nigurr do you expect CI systems to make a change to provide these envars by default? |
@rickhanlonii Yes. CI/CD pipelines currently designed to drive configuration through environment variables. Pipelines love to help leveraging richer test reporting without user intervention/trying to figure out how to achieve simple test results publishing. As part of Azure Pipelines, we love to help community/users to achieve the same. |
If CIs start exposing e.g.
Hard-coding all of this into Jest seems unrealistic. Either CIs have to provide the needed configuration in the environment, or I think this is better solved on a case-by-case basis in user-land. Coupling Jest to a bunch of different CIs doesn't seem scalable. I can imagine the issues for "what about my CI of choice?" already. It would be nice if they also supported the same thing for coverage reports, although that might be out of scope for this? |
@SimenB
We are looking out for solution where if JEST_OPTS are provided in environment variable add the option to Jest runner automatically. For example: CI pipelines can essentially add their own specific folders to get the reporting as per their requirements. Regarding coverage, yes it will be in similar lines. Pipelines can now effectively publish both test results and code coverage. It can be extended to any other reporting/enabling/disabling some custom reporter/logger. We (JEST) shouldn't be understanding CI/CD persona, runner should simply read environment variable Please let me know if I can better document it. |
If CIs expose a We currently don't have support for defining However, I'm not sure about bundling a bunch of reporters with Jest - I'm afraid that our slow release cycle might get in the way of good integrations. You could pass Not sure if either of those last two paragraphs are feasible, or good ideas, but would love your thoughts on it. |
To conclude,
|
Yeah, but will they? I've never had a feature request be answered by either Circle or Travis, so I'm not sure how feasible it is. And building support for something very few CIs will use might not be worth it. I might be wrong though! 🙂
👍
I meant the contract between Jest and reporters for it, not the junit format itself. Sorry if I as unclear! This is relevant as we might e.g. move to an event based architecture for reporters instead of call functions on it manually, see #6616
We don't want that (although junit is limited, we don't want to invent the wheel here)
|
Even if we did add JEST_OPTS environment support there is no guarantee of compatibility. We're currently discussing simplifying the configs and this may result in changes to CLI. New features are also affected by this. Azure would have to have some pretty deep knowledge of jest and know the jest version of a project before setting JEST_OPTS and I think that's a risk to user experience. One alternative, from my perspective, would be that github detects you are using jest and comments on a PR with recommended configuration that a user/team could apply. That would increase usage and still keep configuration in user land. |
I'd prefer some env variable saying "test reports in this format should go to this directory, coverage in this format to this directory". |
Yeah something like that makes a lot of sense. A CI standard optional config overrides. |
+1 exactly. Can we agree on this and support environment variable option to accept? |
|
Would suggest |
This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days. |
This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one. |
Issue:
Currently Azure DevOps and Circle CI/CD systems provide rich test experience.
In order to get this experience/track test reporting, users has to explicitly generate test reports (currently supported format JUnit) by 3rd party plugins and publish test results.
Most of the popular open source projects aren't using 3rd party reporting plugins.
To keep local development experience and CI/CD build experience similar, scripts to build/test are written to print test results to console (std_out).
To get richer test experience in CI/CD pipeline, customers has to modify their scripts and often need to add 3rd party plugins.
This is a steep learning and test reporting is one of the important data that should be easily available for open source github projects.
Proposal for solution(s):
Generate test results file based on environment variables:
In this approach, test runners will generate test results file along with regular console output if an environment variable is set.
CI/CD pipelines can set this variable to desired test output location where they can automatically publish test results by consuming test results file generated by test runners (Mocha, Jest, Pytest, Jasmine).
This simple flag will make sure test runners are not going through any change in behavior in terms of execution or context they are running and users don't require go through any modification to their source/scripts.
Happy to listen to community feedback. Feel free to share other ideas if proposed solution is not feasible one.
The text was updated successfully, but these errors were encountered: