New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TestNG Tests in the dogfooding repo giving error when trying to run with specified mode. #62

Closed
hemanik opened this Issue Jul 17, 2017 · 18 comments

Comments

Projects
None yet
4 participants
@hemanik
Contributor

hemanik commented Jul 17, 2017

Issue Overview

When executing TestNG tests in the dogfooding repo with Smart Testing provider, we get the following error,

The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Command was /bin/sh -c cd /tmp/junit3013285195732663576/smart-testing-dogfood-repo_HistoricalChangesAffectedTestsSelectionExecutionFunctionalTest_should_execute_all_tests_when_smart_testing_is_disabled_irrespective_of_strategy/testng/core && /usr/java/jdk1.8.0_60/jre/bin/java -jar /tmp/junit3013285195732663576/smart-testing-dogfood-repo_HistoricalChangesAffectedTestsSelectionExecutionFunctionalTest_should_execute_all_tests_when_smart_testing_is_disabled_irrespective_of_strategy/testng/core/target/surefire/surefirebooter3375551549339713544.jar /tmp/junit3013285195732663576/smart-testing-dogfood-repo_HistoricalChangesAffectedTestsSelectionExecutionFunctionalTest_should_execute_all_tests_when_smart_testing_is_disabled_irrespective_of_strategy/testng/core/target/surefire 2017-07-17T19-42-19_270-jvmRun1 surefire6725514113717898404tmp surefire_08997551454508629680tmp
Error occurred in starting fork, check output in log

Here's the related issue: https://gist.github.com/juherr/6eb3e93e2db33979b7e90b63ddadc888

Expected Behaviour

Tests should execute as per the strategy and mode selected.

Current Behaviour

The output is as follows,

cat org.jboss.arquillian.testng.TestNGIntegrationTestCase-output.txt 
Configuring TestNG with: TestNG60Configurator
Cannot use a threadCount parameter less than 1; 1 > 0
Usage: <main class> [options] The XML suite files to run
  Options:
    -configfailurepolicy
       Configuration failure policy (skip or continue)
    -d
       Output directory
    -dataproviderthreadcount
       Number of threads to use when running data providers
    -excludegroups
       Comma-separated list of group names to  exclude
    -groups
       Comma-separated list of group names to be run
    -junit
       JUnit mode
       Default: false
    -listener
       List of .class files or list of class names implementing ITestListener or
       ISuiteListener
    -methods
       Comma separated of test methods
       Default: []
    -methodselectors
       List of .class files or list of class names implementing IMethodSelector
    -mixed
       Mixed mode - autodetect the type of current test and run it with
       appropriate runner
       Default: false
    -objectfactory
       List of .class files or list of class names implementing
       ITestRunnerFactory
    -parallel
       Parallel mode (methods, tests or classes)
       Possible Values: [tests, methods, classes, instances, none, true, false]
    -port
       The port
    -reporter
       Extended configuration for custom report listener
    -suitename
       Default name of test suite, if not specified in suite definition file or
       source code
    -suitethreadpoolsize
       Size of the thread pool to use to run suites
       Default: 1
    -testclass
       The list of test classes
    -testjar
       A jar file containing the tests
    -testname
       Default name of test, if not specified in suitedefinition file or source
       code
    -testnames
       The list of test names to run
    -testrunfactory, -testRunFactory
       The factory used to create tests
    -threadcount
       Number of threads to use when running tests in parallel
    -usedefaultlisteners
       Whether to use the default listeners
       Default: true
    -log, -verbose
       Level of verbosity
    -xmlpathinjar
       The full path to the xml file inside the jar file (only valid if -testjar
       was specified)
       Default: testng.xml
Steps To Reproduce
  1. Change the pom.xml as hemanik/smart-testing-dogfood-repo@aa0e855 for the testng tests in dogfooding repo.
  2. Execute mvn clean test -X -pl testng/core
Environment Details
Apache Maven 3.3.9 (NON-CANONICAL_2016-04-07T23:15:34Z_mockbuild; 2016-04-08T04:45:34+05:30)
Maven home: /usr/share/maven
Java version: 1.8.0_60, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_60/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.10.15-100.fc24.x86_64", arch: "amd64", family: "unix"

@bartoszmajsak bartoszmajsak changed the title from [Bug] TestNG Tests in the dogfooding repo giving error when trying to run with ordering mode. to TestNG Tests in the dogfooding repo giving error when trying to run with ordering mode. Jul 18, 2017

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 18, 2017

Member

Thanks for reporting @hemanik. According to the gist you shared it only happens when:

when you have both surefire-junit47 and surefire-testng as dependency.

@MatousJobanek can this be solved in the Surefire Provider itself? We pull in some dependency there, right?

Member

bartoszmajsak commented Jul 18, 2017

Thanks for reporting @hemanik. According to the gist you shared it only happens when:

when you have both surefire-junit47 and surefire-testng as dependency.

@MatousJobanek can this be solved in the Surefire Provider itself? We pull in some dependency there, right?

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 18, 2017

Member

@hemanik In order to unblock your work you can exclude this module from the embedded build for your tests, having configuration equivalent to mvn clean test -X -pl !testng/core (if that's the only module).

@MatousJobanek is there such an option in embedded mvn? Couldn't find it by quickly looking at the API.

Member

bartoszmajsak commented Jul 18, 2017

@hemanik In order to unblock your work you can exclude this module from the embedded build for your tests, having configuration equivalent to mvn clean test -X -pl !testng/core (if that's the only module).

@MatousJobanek is there such an option in embedded mvn? Couldn't find it by quickly looking at the API.

@dipak-pawar

This comment has been minimized.

Show comment
Hide comment
@dipak-pawar

dipak-pawar Jul 18, 2017

Contributor

@bartoszmajsak we do have setProjects options which I believe @hemanik already using here.

Contributor

dipak-pawar commented Jul 18, 2017

@bartoszmajsak we do have setProjects options which I believe @hemanik already using here.

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 18, 2017

Member

It would be awesome to have an ability for exclusion as well. Doesn't seem like we can do it atm. Could you try to contribute this feature to upstream project?

Member

bartoszmajsak commented Jul 18, 2017

It would be awesome to have an ability for exclusion as well. Doesn't seem like we can do it atm. Could you try to contribute this feature to upstream project?

@dipak-pawar

This comment has been minimized.

Show comment
Hide comment
@dipak-pawar

dipak-pawar Jul 18, 2017

Contributor

sure, you mean if we hit this mvn -pl '!core,!app1' clean install, so it should exclude core & app1 using maven embedded. but atm it's not excluding right?

Contributor

dipak-pawar commented Jul 18, 2017

sure, you mean if we hit this mvn -pl '!core,!app1' clean install, so it should exclude core & app1 using maven embedded. but atm it's not excluding right?

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 18, 2017

Member

That's exactly what I mean. Did you try it out?

Member

bartoszmajsak commented Jul 18, 2017

That's exactly what I mean. Did you try it out?

@MatousJobanek

This comment has been minimized.

Show comment
Hide comment
@MatousJobanek

MatousJobanek Jul 18, 2017

Contributor

can this be solved in the Surefire Provider itself? We pull in some dependency there, right?

Yeah, we pull - but it should pull only one provider, so it shouldn't happen that two providers are on classpath.

is there such an option in embedded mvn? Couldn't find it by quickly looking at the API.

everything that is possible to set you can find here: https://github.com/shrinkwrap/resolver/blob/master/maven/api-maven-embedded/src/main/java/org/jboss/shrinkwrap/resolver/api/maven/embedded/pom/equipped/ConfigurationStage.java
technically, you could try to use: https://github.com/shrinkwrap/resolver/blob/master/maven/api-maven-embedded/src/main/java/org/jboss/shrinkwrap/resolver/api/maven/embedded/pom/equipped/ConfigurationStage.java#L90

Contributor

MatousJobanek commented Jul 18, 2017

can this be solved in the Surefire Provider itself? We pull in some dependency there, right?

Yeah, we pull - but it should pull only one provider, so it shouldn't happen that two providers are on classpath.

is there such an option in embedded mvn? Couldn't find it by quickly looking at the API.

everything that is possible to set you can find here: https://github.com/shrinkwrap/resolver/blob/master/maven/api-maven-embedded/src/main/java/org/jboss/shrinkwrap/resolver/api/maven/embedded/pom/equipped/ConfigurationStage.java
technically, you could try to use: https://github.com/shrinkwrap/resolver/blob/master/maven/api-maven-embedded/src/main/java/org/jboss/shrinkwrap/resolver/api/maven/embedded/pom/equipped/ConfigurationStage.java#L90

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 18, 2017

Member

I was more thinking about having list of excluded modules, so higher-level concept than list of exclude files for a reactor :)

Member

bartoszmajsak commented Jul 18, 2017

I was more thinking about having list of excluded modules, so higher-level concept than list of exclude files for a reactor :)

@dipak-pawar

This comment has been minimized.

Show comment
Hide comment
@dipak-pawar

dipak-pawar Jul 18, 2017

Contributor

That's exactly what I mean. Did you try it out?

No I haven't tried

Contributor

dipak-pawar commented Jul 18, 2017

That's exactly what I mean. Did you try it out?

No I haven't tried

@hemanik hemanik changed the title from TestNG Tests in the dogfooding repo giving error when trying to run with ordering mode. to TestNG Tests in the dogfooding repo giving error when trying to run with specified mode. Jul 18, 2017

@dipak-pawar

This comment has been minimized.

Show comment
Hide comment
@dipak-pawar

dipak-pawar Jul 18, 2017

Contributor

Hey I tried with for excluding submodule. We have to explicitly tell to use maven version > 3.2.1 using .useMaven3Version() api because Maven 3.2.1 has added this feature.

@hemanik You can use exclude testng modules using following code

final BuiltProject build = embeddedMaven
                    .useMaven3Version("3.3.9")
                    .setProjects("!testng, !testng/core, !testng/container, !testng/standalone")
                    .setGoals(goals)
                    .setDebug(isMavenDebugOutputEnabled())
                    .setQuiet(disableQuietWhenAnyDebugModeEnabled() && quietMode)
                    .skipTests(false)
                    .setProperties(systemProperties)
                    .ignoreFailure()
                .build();

@bartoszmajsak I have tested this, it's excluding modules if you set maven version > 3.2.1. No need to contribute to upstream as it's already there

Contributor

dipak-pawar commented Jul 18, 2017

Hey I tried with for excluding submodule. We have to explicitly tell to use maven version > 3.2.1 using .useMaven3Version() api because Maven 3.2.1 has added this feature.

@hemanik You can use exclude testng modules using following code

final BuiltProject build = embeddedMaven
                    .useMaven3Version("3.3.9")
                    .setProjects("!testng, !testng/core, !testng/container, !testng/standalone")
                    .setGoals(goals)
                    .setDebug(isMavenDebugOutputEnabled())
                    .setQuiet(disableQuietWhenAnyDebugModeEnabled() && quietMode)
                    .skipTests(false)
                    .setProperties(systemProperties)
                    .ignoreFailure()
                .build();

@bartoszmajsak I have tested this, it's excluding modules if you set maven version > 3.2.1. No need to contribute to upstream as it's already there

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 18, 2017

Member

Thanks for checking it out @dipak-pawar. I was thinking that maybe we could still have a method which has better meaning, for example

 .excludeProjects("testng/core", "testng/container","testng/standalone")

but if it works the way you described above we are good for the time being.

Member

bartoszmajsak commented Jul 18, 2017

Thanks for checking it out @dipak-pawar. I was thinking that maybe we could still have a method which has better meaning, for example

 .excludeProjects("testng/core", "testng/container","testng/standalone")

but if it works the way you described above we are good for the time being.

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 19, 2017

Member

As it turned out using .useMaven3Version("3.3.9") on Travis won't work (and probably also in other setups as we run stuff in parallel). I introduced following improvement to have this feature enabled for travis:
#65

Member

bartoszmajsak commented Jul 19, 2017

As it turned out using .useMaven3Version("3.3.9") on Travis won't work (and probably also in other setups as we run stuff in parallel). I introduced following improvement to have this feature enabled for travis:
#65

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 19, 2017

Member

@hemanik can you provide details about your environment in the issue description?

Member

bartoszmajsak commented Jul 19, 2017

@hemanik can you provide details about your environment in the issue description?

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Jul 19, 2017

Member

It also looks that it only affects ordering mode which till now we didn't have in test-bed. For selecting it's working fine, otherwise all the other tests would be already failing like crazy. Thanks for making one with the ordering mode!

Member

bartoszmajsak commented Jul 19, 2017

It also looks that it only affects ordering mode which till now we didn't have in test-bed. For selecting it's working fine, otherwise all the other tests would be already failing like crazy. Thanks for making one with the ordering mode!

@MatousJobanek

This comment has been minimized.

Show comment
Hide comment
@MatousJobanek

MatousJobanek Jul 25, 2017

Contributor

Hi,
the reason why it is failing only with the ordering mode is that the changes you perform on the repository don't create/modify any TestNG class - only JUnit class. This means that the selective mode doesn't test anything as there are no TestNG tests that would fulfill the runOrder strategies.

As for the failure itself - in short - it is a bug/shortcoming in Surefire plugin itself. I'll create an issue for it in Surefire and send a PR and link it here...

Contributor

MatousJobanek commented Jul 25, 2017

Hi,
the reason why it is failing only with the ordering mode is that the changes you perform on the repository don't create/modify any TestNG class - only JUnit class. This means that the selective mode doesn't test anything as there are no TestNG tests that would fulfill the runOrder strategies.

As for the failure itself - in short - it is a bug/shortcoming in Surefire plugin itself. I'll create an issue for it in Surefire and send a PR and link it here...

@MatousJobanek

This comment has been minimized.

Show comment
Hide comment
@MatousJobanek

MatousJobanek Jul 25, 2017

Contributor

Anyway, even if it was fixed in the next release of surefire, then (if we wanted to support the older versions of surefire such as 2.19.1 and 2.20.0) we would need to have a workaround for it - it should be quite easy, so I'll do it.

Contributor

MatousJobanek commented Jul 25, 2017

Anyway, even if it was fixed in the next release of surefire, then (if we wanted to support the older versions of surefire such as 2.19.1 and 2.20.0) we would need to have a workaround for it - it should be quite easy, so I'll do it.

@MatousJobanek

This comment has been minimized.

Show comment
Hide comment
@MatousJobanek

MatousJobanek Jul 25, 2017

Contributor

Surefire issue link: https://issues.apache.org/jira/browse/SUREFIRE-1398
PR: apache/maven-surefire#160
I'll send the workaround for the issue tomorrow

Contributor

MatousJobanek commented Jul 25, 2017

Surefire issue link: https://issues.apache.org/jira/browse/SUREFIRE-1398
PR: apache/maven-surefire#160
I'll send the workaround for the issue tomorrow

MatousJobanek added a commit to MatousJobanek/smart-testing that referenced this issue Jul 26, 2017

issue arquillian#62 When TestNG provider is used,non-positive threadc…
…ount param is

removed
Part of this commit is also a change from ProviderList to
SurefireProviderFactory to keep related logic in one place
@MatousJobanek

This comment has been minimized.

Show comment
Hide comment
@MatousJobanek
Contributor

MatousJobanek commented Jul 27, 2017

Workaround: MatousJobanek@653914c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment