Info about skipping of tests aren't writing due to retryAnalyzer at test-methods #27

Brankursine · 2018-05-16T10:22:42Z

Problem

Info about skipping of tests aren't writing due to retryAnalyzer at test-methods
retryAnalyzer at test-methods causes "java.lang.IllegalArgumentException: ItemID should not be null"

Steps to Reproduce

versions at pom.xml

        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.11</version>
        </dependency>
        <dependency>
            <groupId>com.epam.reportportal</groupId>
            <artifactId>agent-java-testng</artifactId>
            <version>4.0.2</version>
        </dependency>
        <dependency>
            <groupId>com.epam.reportportal</groupId>
            <artifactId>logger-java-log4j</artifactId>
            <version>4.0.1</version>
        </dependency>

create class that implement IRetryAnalyzer (even without retry - always false), e.g.

package com.reltio.qa.helpers;

import org.apache.log4j.Logger;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzer implements IRetryAnalyzer {
    private final Logger logger = Logger.getLogger(this.getClass());

    int counter = 0;
    int retryLimit = 2;

    @Override
    public boolean retry(ITestResult result) {
        if (result.isSuccess()) {
            return false;
        }
        if (counter < retryLimit) {
            if (result.getThrowable() != null) {
                logger.warn(String.format("Test %s failed with exception. Will retry... (%s attempt)", result.getName(), counter), result.getThrowable());
            }
            counter++;
            return true;
        }
        return false;
    }
}

create listener which set RetryAnalyzer.class at test-method, e.g.

package com.reltio.qa.helpers.listeners;

import com.reltio.qa.helpers.RetryAnalyzer;
import org.testng.IAnnotationTransformer;
import org.testng.IRetryAnalyzer;
import org.testng.annotations.ITestAnnotation;

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

/**
 * This class is a Listener. It will apply retry mechanism (RetryAnalyzer class) for every test method which doesn't have RetryAnalyzer specified at annotation
 */
public class RetryAnnotationListener implements IAnnotationTransformer {

    @Override
    public void transform(ITestAnnotation testAnnotation, Class testClass,
                          Constructor testConstructor, Method testMethod) {
        IRetryAnalyzer retry = testAnnotation.getRetryAnalyzer();

        if (retry == null) {
            testAnnotation.setRetryAnalyzer(RetryAnalyzer.class);
        }
    }
}

create test class, where non-test-method is failed, e.g.

package com.reltio.qa;

import com.reltio.qa.helpers.RetryAnalyzer;
import org.apache.log4j.Logger;
import org.testng.Assert;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class ReproduceItemIdTest {

    final Logger logger = Logger.getLogger(this.getClass());

    @BeforeClass
    public void bcReproduceItemIdTest() {
        logger.info("bcReproduceItemIdTest");
        Assert.assertTrue(false); // requirement: fail at non @Test method
    }

//    @BeforeMethod
//    public void bmReproduceItemIdTest() {
//        logger.info("bmReproduceItemIdTest");
//        Assert.assertTrue(false); // requirement: fail at non @Test method
//    }

    @Test
    public void test0() {
        logger.info("test0");
        Assert.assertTrue(true);
    }
}

create suite with listener && test-class mentioned above, e.g.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="ALL Tests" parallel="tests" thread-count="1">
    <listeners>
        <listener class-name="com.epam.reportportal.testng.ReportPortalTestNGListener"/>
        <listener class-name="com.reltio.qa.helpers.listeners.RetryAnnotationListener"/>
    </listeners>

    <test name="SandBox">
        <classes>
            <class name="com.reltio.qa.ReproduceItemIdTest"/>
        </classes>
    </test>
</suite>

execute tests via maven with reporting to reportportal, e.g.:

test -Dsurefire.suiteXmlFiles=suites/ReproduceItemId.xml -Drp.enable=true -Drp.endpoint=https://dnsname.com -Drp.uuid=someUUID -Drp.project=someProject -Drp.launch=Debug -Drp.mode=debug

Expected Results

No exception.
Process such scenario with retryAnalyzer as expected - just write info about skipping of tests.

Actual Results

2018-05-16 14:59:30,670 [INFO ][TestNG-tests-1  ][     ReproduceItemIdTest] - bcReproduceItemIdTest
java.lang.IllegalArgumentException: ItemID should not be null
	at rp.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
	at com.epam.reportportal.service.LaunchImpl.finishTestItem(LaunchImpl.java:230)
	at com.epam.reportportal.testng.TestNGService.finishTestMethod(TestNGService.java:157)
	at com.epam.reportportal.testng.BaseTestNGListener.onTestSkipped(BaseTestNGListener.java:103)
	at org.testng.internal.Invoker.runTestListeners(Invoker.java:1723)
	at org.testng.internal.Invoker.runTestListeners(Invoker.java:1714)
	at org.testng.internal.Invoker.registerSkippedTestResult(Invoker.java:1272)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:605)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:744)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:39)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:414)
	at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 7.713 s <<< FAILURE! - in TestSuite
[ERROR] bcReproduceItemIdTest(com.reltio.qa.ReproduceItemIdTest)  Time elapsed: 4.027 s  <<< FAILURE!
java.lang.AssertionError: expected [true] but found [false]
	at com.reltio.qa.ReproduceItemIdTest.bcReproduceItemIdTest(ReproduceItemIdTest.java:16)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   ReproduceItemIdTest.bcReproduceItemIdTest:16 expected [true] but found [false]
[INFO] 
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 1
[INFO] 
[ERROR] There are test failures.

The text was updated successfully, but these errors were encountered:

Brankursine · 2018-05-16T10:28:21Z

ogonekorama · 2018-07-10T18:27:22Z

@DzmitryHumianiuk @avarabyeu @pbortnik Is there any progress on this issue? It is affecting our operations to the extent that there is a dilemma: either retries or ReportPortal. We can't use both retries and Report Portal during test execution at the same time. Please help!

Kanaduchi · 2018-07-17T09:35:55Z

@DzmitryHumianiuk @avarabyeu @pbortnik I also wait for any updates of this issue

avarabyeu · 2018-07-23T10:53:31Z

@Kanaduchi Unfortunately, we cannot reproduce it.
I've pushed your example at our example repository, could you please check it there? The only difference is the latest version of integration. Please, let us know whether it works for you

DzmitryHumianiuk · 2018-08-14T12:23:17Z

@Brankursine i see you closed item.
and deleted example.

it's not actual anymore?

Brankursine · 2018-08-14T12:31:42Z

Sorry for the delay in my response.

Just created simple project (IntelliJ IDEA: pom + suite + test) which reproduce the problem.
ReproduceReTryFail.zip

command line:
test -Dsurefire.suiteXmlFiles=suites/ReproduceItemId.xml -Drp.enable=true -Drp.endpoint=https://******.com -Drp.uuid=******* -Drp.project=platform -Drp.launch=Debug -Drp.tags=debug -Drp.mode=debug

logs:

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building ReproduceReTry 2018.3.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ pj-reproduce ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Pj__IJ\ReproduceReTryFail\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ pj-reproduce ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 2 source files to C:\Pj__IJ\ReproduceReTryFail\target\classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ pj-reproduce ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] Copying 1 resource to suites
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ pj-reproduce ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to C:\Pj__IJ\ReproduceReTryFail\target\test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ pj-reproduce ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2018-08-14 17:29:31,270 [INFO ][TestNG-tests-1 ][ ReproduceItemIdTest] - bcReproduceItemIdTest
java.lang.IllegalArgumentException: ItemID should not be null
at rp.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at com.epam.reportportal.service.LaunchImpl.finishTestItem(LaunchImpl.java:230)
at com.epam.reportportal.testng.TestNGService.finishTestMethod(TestNGService.java:157)
at com.epam.reportportal.testng.BaseTestNGListener.onTestSkipped(BaseTestNGListener.java:103)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1723)
at org.testng.internal.Invoker.runTestListeners(Invoker.java:1714)
at org.testng.internal.Invoker.registerSkippedTestResult(Invoker.java:1272)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:605)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:744)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:39)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:414)
at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 4.377 s <<< FAILURE! - in TestSuite
[ERROR] bcReproduceItemIdTest(com.mysite.ReproduceItemIdTest) Time elapsed: 1.8 s <<< FAILURE!
java.lang.AssertionError: expected [true] but found [false]
at org.testng.Assert.fail(Assert.java:93)
at org.testng.Assert.failNotEquals(Assert.java:512)
at org.testng.Assert.assertTrue(Assert.java:41)
at org.testng.Assert.assertTrue(Assert.java:51)
at com.mysite.ReproduceItemIdTest.bcReproduceItemIdTest(ReproduceItemIdTest.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:523)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:224)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:146)
at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:166)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:105)
at org.testng.TestRunner.privateRun(TestRunner.java:744)
at org.testng.TestRunner.run(TestRunner.java:602)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
at org.testng.SuiteRunner.access$000(SuiteRunner.java:39)
at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:414)
at org.testng.internal.thread.ThreadUtil$1.call(ThreadUtil.java:52)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ReproduceItemIdTest.bcReproduceItemIdTest:15 expected [true] but found [false]
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 1
[INFO]
[ERROR] There are test failures.

Please refer to C:\Pj__IJ\ReproduceReTryFail\target\surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.624 s
[INFO] Finished at: 2018-08-14T17:29:34+05:00
[INFO] Final Memory: 18M/259M
[INFO] ------------------------------------------------------------------------

Process finished with exit code 0

Brankursine · 2018-08-14T12:33:31Z

@DzmitryHumianiuk close it by mistake (
Issue is still actual.
Just did some little notes above about reproduce.

Consumer is:

Current version: Analysis Service: 4.2.0; API Service: 4.2.1; Index Service: 4.2.0; Jira Service: 4.2.0; Authorization Service: 4.2.0; Service UI: 4.2.2;

Kanaduchi · 2018-08-14T12:49:30Z

I can easily reproduce the problem with example provided by @Brankursine

DzmitryHumianiuk · 2018-08-14T15:34:46Z

@avarabyeu and @pbortnik will take a look during this week

Kanaduchi · 2018-10-30T14:01:43Z

will take a look during this week
@DzmitryHumianiuk Any updates on this?

filland · 2019-03-04T10:13:05Z

@Brankursine @Kanaduchi @ogonekorama
Guys, seems like i fixed the problem.
Could you generate a jar file from master branch and try it with your project ?

To use generated jar file I had to add additionalClasspathElements with the path to the .jar file to maven surefire plugin configs (do not forget to remove maven dependency for the testng agent in your pom.xml):

      <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.21.0</version>
            <configuration>
                <additionalClasspathElements>
                    D:\rp_sandbox\agent-java-testng-4.2.2-SNAPSHOT_3_4_2019.jar
                </additionalClasspathElements>
                <testFailureIgnore>true</testFailureIgnore>
                <trimStackTrace>false</trimStackTrace>
                <useFile>false</useFile>
            </configuration>
        </plugin>

DzmitryHumianiuk · 2019-03-05T14:06:03Z

@Brankursine plz check

Brankursine changed the title ~~retryAnalyzer at test-methods causes java.lang.IllegalArgumentException: ItemID should not be null~~ Info about skipping of tests aren't writing due to retryAnalyzer at test-methods May 16, 2018

DzmitryHumianiuk assigned avarabyeu and pbortnik May 16, 2018

DzmitryHumianiuk added the bug label May 16, 2018

Kanaduchi mentioned this issue Jul 17, 2018

agent doesn't work with latest version of TestNG #26

Closed

Brankursine closed this as completed Aug 14, 2018

Brankursine reopened this Aug 14, 2018

DzmitryHumianiuk mentioned this issue Mar 5, 2019

Skipped Tests not showing when retry tests are included #31

Closed

filland added the wait for response label Mar 5, 2019

filland closed this as completed Mar 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Info about skipping of tests aren't writing due to retryAnalyzer at test-methods #27

Info about skipping of tests aren't writing due to retryAnalyzer at test-methods #27

Brankursine commented May 16, 2018 •

edited

Brankursine commented May 16, 2018

ogonekorama commented Jul 10, 2018

Kanaduchi commented Jul 17, 2018

avarabyeu commented Jul 23, 2018

DzmitryHumianiuk commented Aug 14, 2018

Brankursine commented Aug 14, 2018 •

edited

Brankursine commented Aug 14, 2018 •

edited

Kanaduchi commented Aug 14, 2018

DzmitryHumianiuk commented Aug 14, 2018

Kanaduchi commented Oct 30, 2018

filland commented Mar 4, 2019 •

edited

DzmitryHumianiuk commented Mar 5, 2019

Info about skipping of tests aren't writing due to retryAnalyzer at test-methods #27

Info about skipping of tests aren't writing due to retryAnalyzer at test-methods #27

Comments

Brankursine commented May 16, 2018 • edited

Problem

Steps to Reproduce

Expected Results

Actual Results

Brankursine commented May 16, 2018

ogonekorama commented Jul 10, 2018

Kanaduchi commented Jul 17, 2018

avarabyeu commented Jul 23, 2018

DzmitryHumianiuk commented Aug 14, 2018

Brankursine commented Aug 14, 2018 • edited

Brankursine commented Aug 14, 2018 • edited

Kanaduchi commented Aug 14, 2018

DzmitryHumianiuk commented Aug 14, 2018

Kanaduchi commented Oct 30, 2018

filland commented Mar 4, 2019 • edited

DzmitryHumianiuk commented Mar 5, 2019

Brankursine commented May 16, 2018 •

edited

Brankursine commented Aug 14, 2018 •

edited

Brankursine commented Aug 14, 2018 •

edited

filland commented Mar 4, 2019 •

edited