[Gradle + TestNg] Listener initialization failures not logged in the build log/console #25857
Comments
|
Sorry that you're having trouble with Gradle! We appreciate the effort that went into filing this issue, but we must ask for more information. As stated in our issue template, a minimal reproducible example is a must for us to be able to track down and fix your problem efficiently. Our available resources are severely limited, and we must be sure we are looking at the exact problem you are facing. If we have a reproducer, we may be able also to suggest workarounds or ways to avoid the problem. The ideal way to provide a reproducer is to leverage our reproducer template. You can also use Gradle Project Replicator to reproduce the structure of your project. This issue will be closed after 7 days unless you provide more information. We need a full reproducer to decide whether is should be a feature or it's just a misconfiguration |
ov7a
added
in:testing-testng
pending:reproducer
github-actions
bot
removed
the
pending:reproducer
|
@ov7a Hi, Vlad. As far as I see, people report the same issue here: https://youtrack.jetbrains.com/issue/IDEA-251105, and the intention is simple: just making it easier to see the exception in the console output. When Gradle reports "FAILURE: Build failed with an exception." it is no clear details of the exception and people need to open the file (for TeamCity, you need to setup report publisher, and for local run, find the proper folder with XML and then check the error itself). All these make 3-4 steps (as people mentioned in https://youtrack.jetbrains.com/issue/IDEA-251105) to just understand what Exception has happened. The same here: "Currently we throw a TestNGException . We can throw something else, but that doesn't change anything because Gradle is basically catching Throwable. So anything we throw is going to be gobbled up." - testng-team/testng#2937 So, I believe, the issue that was posted by @sravanmedarapu makes sense for any developer and we can see different references for it. This minor improvement can just help to get an easy understanding of the issue originally happened by TestNG and Gradle can simply show it instead of gobbling. |
|
@agileseph Thanks for providing the context! It may take some time for us to return with an answer, but we’ll discuss this issue internally. |
7 tasks
|
Including sample logs for reference. Maven Logs: Maven prints the failure reason why the build failed at the end. Example(Click)Gradle logs: Difficult to understand to what went wrong. 1. --info mode(
|
|
This issue needs a decision from the team responsible for that area. They have been informed. Response time may vary. JVM team, can/should we do something around here in order to make these early-stage exceptions to be more visible for the user? It would be nice to have consistent behavior with JUnit, but we don't know if JUnit behaves the same or not. |
ov7a
added
👋 team-triage
jvandort
added
in:testing
a:bug
in:testing-testng
and removed
in:testing-testng
👋 team-triage
|
It seems we have an API to solve this, however after testing it does not appear to work with TestNG. For context, the below code would solve this issue for JUnit Jupiter. We have added this issue to our backlog, specifically to fix this for Test NG. |
|
Out of curiosity, do we have any estimated timelines for this task, @jvandort ? |
|
@sravanmedarapu We're actively looking into this and we're targeting either 8.4 or 8.5 for a fix. |
|
@sravanmedarapu After looking into this deeper, it's currently possible to get these errors by setting the granularity in the test logging to a value 1 or lower. The granularity setting controls what test events are actually logged to the console. The default is to only log events to the console that are associated with individual test executions. Since this is an error associated with the test framework initialization, it is filtered out by default. By setting the Having said that, there was an error associated with formatting exceptions coming from test events not associated with test classes. So this issue will be fixed in 8.4 and at that point, setting the granularity should work for the specific error you were seeing. Longer term, this concept of granularity is a little difficult to reason about and dates back to ancient code from 2012 (Gradle 1.x I think). We should revisit test logging and think about how we can make this work more intuitively and capture these sorts of errors without having to fiddle with the configuration. However, that will require some thinking and might involve breaking changes. So a more fundamental improvement for this will likely come on a longer timeline. |
|
@sravanmedarapu the 8.4 RC1 is scheduled on September 11th and 8.4 GA should happen on September 18th. Note that these dates aren't set in stone and might be changed. |
This was referenced Oct 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
Context: Running TestNG Tests (e.g
./gradlew :clean :test -d)Problem:
When build fail during test listener configuration stage/before launching the test, it's expected to see the failure reason in the build log.
However current build log doesn't contain details about why build failed. Setting log level to
debug(-d)orinfo(-i)doesn't make much difference. (please refer to logs included)Current Behavior (optional)
Build log(./gradlew clean test):
XML report file:
Context
We run large number of builds (in CI) and referring to external files to understand the build not really feasible option.
Environment:
build.gradle.kts:
The text was updated successfully, but these errors were encountered: