Skip to content

Loading…

Return exit code 0 when no features are found, or no found features match the tags filter #567

Merged
merged 4 commits into from

4 participants

@brasmusson

This PR is created in response to the following discussion on the mailing list:
[JVM] Exception thrown when no tags are found (https://groups.google.com/forum/#!topic/cukes/wZo0FTQJrVA).

Return exit code 0 when no features are found, or if no found features match the tags filter used. Print the same message, that is used in the currently thrown CucumberException when this occurs, as information before the summary printout.

Jenkins will mark a job as failed even though the exit code was 0, if the JUnit report file contains no testcase elements (http://jenkins.361315.n4.nabble.com/Jenkins-fails-if-there-are-no-testcases-td3899185.html). Therefore add a dummy testcase element to the JUnit report, when no features are found, or no found features match the tags filter used.

brasmusson added some commits
@brasmusson brasmusson Return exit code 0 when no features are found
Return exit code 0 when no features are found, or if no found features
match the tags filter used. Print the same message, that is used in the
currently thrown CucumberException when this occurs, as information
before the summary printout.
365d391
@brasmusson brasmusson Add a dummy test case in JUnit report if no features are found
Jenkins will mark a job as failed even though the exit code was 0, if
the JUnit report file contains no testcase elements. Therefore add a
dummy testcase element to the JUnit report, when no features are found,
or no found features match the tags filter used.
0fcca0a
@dkowis
Cucumber member

I don't think this is the proper solution for this. If there are no tests, or it can't find anything it should be a "failure" because you didn't actually test anything. It probably shouldn't barf an exception, but it shouldn't exit cleanly. That's my $0.02.

Using the "--wip" option will invert this (IIRC), so if no tests are found it'll exit 0

@ffbit ffbit commented on an outdated diff
core/src/main/java/cucumber/runtime/RuntimeGlue.java
@@ -103,7 +103,7 @@ public StepDefinitionMatch stepDefinitionMatch(String uri, Step step, I18n i18n)
@Override
public void writeStepdefsJson(List<String> featurePaths, URL dotCucumber) throws IOException {
if (dotCucumber != null) {
- List<CucumberFeature> features = load(new FileResourceLoader(), featurePaths, NO_FILTERS);
+ List<CucumberFeature> features = load(new FileResourceLoader(), featurePaths, NO_FILTERS, null);
@ffbit Cucumber member
ffbit added a note

What is that null for?
Can you just overload the method?

You are right, overloading is better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@ffbit ffbit commented on an outdated diff
...main/java/cucumber/runtime/model/CucumberFeature.java
@@ -40,11 +40,11 @@
builder.parse(resource, filters);
}
}
- if (cucumberFeatures.isEmpty()) {
+ if (cucumberFeatures.isEmpty() && optionalOut != null) {
@ffbit Cucumber member
ffbit added a note

Maybe a Null Object is appropriate here?

With overloading (according to the previous comment) the need for null check disappears.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@brasmusson

@dkowis On the other hand, Cucumber-Ruby treat it like a success and return exit code 0 in this case. And as far as I know is the "--wip" option not implemented in Cucumber-JVM.

@dkowis
Cucumber member

Ah, I am indeed mistaken. I could've sworn it used to work that way... Very well, this is probably the proper way to do it then. I withdraw my previous objections.

brasmusson and others added some commits
@brasmusson brasmusson Aviod passing null, improve message when missing feature path
Overload CucumberFeature.load to avoid passing null and the need for
null check. Improve the log message when no feature path is given at
the command line.
bc7d976
@ffbit ffbit Created examples/java-no-features to demonstrate Cucumber JVM behaviour
when there are no features found at all.
96a5c15
@ffbit
Cucumber member

@brasmusson Could you, please, merge in your branch code from my branch with the same name?
I added an example project with no features at all and some readme notes.
Thanks.

P.S.
I couldn't Google how to do this without additional PL.

@ffbit
Cucumber member

@brasmusson Here is a little manual how to do what I'm asking for:

git remote add ffbit https://github.com/ffbit/cucumber-jvm.git
git checkout succeed-on-no-features
git pull ffbit succeed-on-no-features
git pull origin succeed-on-no-features

Thanks

@aslakhellesoy
Cucumber member

LGTM!

@brasmusson

@ffbit I reckon that for you to be able to yourself add commit to my pull request, you would need push access to my forked Cucumber-JVM repository. Of course there is always the possibility that you create a PL from your succeed-on-no-features branch to the succeed-on-no-features branch on my forked repository, when that PL is merged, then the commit should be added to this PL. But it seems easier if a fetch the code from your branch and add it to this PL. I'll do that right now ...

@brasmusson brasmusson merged commit 96a5c15 into cucumber:master

1 check passed

Details default The Travis CI build passed
@brasmusson brasmusson deleted the brasmusson:succeed-on-no-features branch
@aslakhellesoy
Cucumber member

@ffbit This maven module is not referenced anywhere, and is not run as part of the build. It also references an outdated version (1.1.4-SNAPSHOT). Can we remove this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Aug 2, 2013
  1. @brasmusson

    Return exit code 0 when no features are found

    brasmusson committed
    Return exit code 0 when no features are found, or if no found features
    match the tags filter used. Print the same message, that is used in the
    currently thrown CucumberException when this occurs, as information
    before the summary printout.
  2. @brasmusson

    Add a dummy test case in JUnit report if no features are found

    brasmusson committed
    Jenkins will mark a job as failed even though the exit code was 0, if
    the JUnit report file contains no testcase elements. Therefore add a
    dummy testcase element to the JUnit report, when no features are found,
    or no found features match the tags filter used.
  3. @brasmusson

    Aviod passing null, improve message when missing feature path

    brasmusson committed
    Overload CucumberFeature.load to avoid passing null and the need for
    null check. Improve the log message when no feature path is given at
    the command line.
Commits on Aug 3, 2013
  1. @ffbit

    Created examples/java-no-features to demonstrate Cucumber JVM behaviour

    ffbit committed
    when there are no features found at all.
View
2 core/src/main/java/cucumber/runtime/RuntimeOptions.java
@@ -121,7 +121,7 @@ private void printUsage() {
}
public List<CucumberFeature> cucumberFeatures(ResourceLoader resourceLoader) {
- return load(resourceLoader, featurePaths, filters);
+ return load(resourceLoader, featurePaths, filters, System.out);
}
public Formatter formatter(ClassLoader classLoader) {
View
13 core/src/main/java/cucumber/runtime/formatter/JUnitFormatter.java
@@ -97,6 +97,9 @@ public void done() {
try {
//set up a transformer
rootElement.setAttribute("failures", String.valueOf(rootElement.getElementsByTagName("failure").getLength()));
+ if (rootElement.getElementsByTagName("testcase").getLength() == 0) {
+ addDummyTestCase(); // to avoid failed Jenkins jobs
+ }
TransformerFactory transfac = TransformerFactory.newInstance();
Transformer trans = transfac.newTransformer();
trans.setOutputProperty(OutputKeys.INDENT, "yes");
@@ -108,6 +111,16 @@ public void done() {
}
}
+ private void addDummyTestCase() {
+ Element dummy = doc.createElement("testcase");
+ dummy.setAttribute("classname", "dummy");
+ dummy.setAttribute("name", "dummy");
+ rootElement.appendChild(dummy);
+ Element skipped = doc.createElement("skipped");
+ skipped.setAttribute("message", "No features found");
+ dummy.appendChild(skipped);
+ }
+
@Override
public void result(Result result) {
testCase.results.add(result);
View
26 core/src/main/java/cucumber/runtime/model/CucumberFeature.java
@@ -1,6 +1,5 @@
package cucumber.runtime.model;
-import cucumber.runtime.CucumberException;
import cucumber.runtime.FeatureBuilder;
import cucumber.runtime.Runtime;
import cucumber.runtime.io.Resource;
@@ -15,6 +14,7 @@
import gherkin.formatter.model.ScenarioOutline;
import gherkin.formatter.model.Step;
+import java.io.PrintStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
@@ -29,29 +29,33 @@
private I18n i18n;
private CucumberScenarioOutline currentScenarioOutline;
+ public static List<CucumberFeature> load(ResourceLoader resourceLoader, List<String> featurePaths, final List<Object> filters, PrintStream out) {
+ final List<CucumberFeature> cucumberFeatures = load(resourceLoader, featurePaths, filters);
+ if (cucumberFeatures.isEmpty()) {
+ if (featurePaths.isEmpty()) {
+ out.println(String.format("Got no path to feature directory or feature file"));
+ } else if (filters.isEmpty()) {
+ out.println(String.format("No features found at %s", featurePaths));
+ } else {
+ out.println(String.format("None of the features at %s matched the filters: %s", featurePaths, filters));
+ }
+ }
+ return cucumberFeatures;
+ }
+
public static List<CucumberFeature> load(ResourceLoader resourceLoader, List<String> featurePaths, final List<Object> filters) {
final List<CucumberFeature> cucumberFeatures = new ArrayList<CucumberFeature>();
final FeatureBuilder builder = new FeatureBuilder(cucumberFeatures);
- boolean resourceFound = false;
for (String featurePath : featurePaths) {
Iterable<Resource> resources = resourceLoader.resources(featurePath, ".feature");
for (Resource resource : resources) {
- resourceFound = true;
builder.parse(resource, filters);
}
}
- if (cucumberFeatures.isEmpty()) {
- if (resourceFound) {
- throw new CucumberException(String.format("None of the features at %s matched the filters: %s", featurePaths, filters));
- } else {
- throw new CucumberException(String.format("No features found at %s", featurePaths));
- }
- }
Collections.sort(cucumberFeatures, new CucumberFeatureUriComparator());
return cucumberFeatures;
}
-
public CucumberFeature(Feature feature, String uri) {
this.feature = feature;
this.uri = uri;
View
26 core/src/test/java/cucumber/runtime/RuntimeTest.java
@@ -3,6 +3,7 @@
import cucumber.api.PendingException;
import cucumber.api.Scenario;
import cucumber.runtime.io.ClasspathResourceLoader;
+import cucumber.runtime.io.Resource;
import cucumber.runtime.io.ResourceLoader;
import cucumber.runtime.model.CucumberFeature;
import gherkin.I18n;
@@ -34,6 +35,7 @@
import static org.mockito.Matchers.any;
import static org.mockito.Matchers.anyCollectionOf;
import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
public class RuntimeTest {
@@ -178,6 +180,16 @@ public void strict_with_errors() {
}
@Test
+ public void should_pass_if_no_features_are_found() {
+ ResourceLoader resourceLoader = createResourceLoaderThatFindsNoFeatures();
+ Runtime runtime = createStrictRuntime(resourceLoader);
+
+ runtime.run();
+
+ assertEquals(0x0, runtime.exitStatus());
+ }
+
+ @Test
public void should_throw_cucumer_exception_if_no_backends_are_found() throws Exception {
try {
ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
@@ -347,6 +359,12 @@ public void runStep(Reporter reporter, Runtime runtime) {
runtime.runStep("<uri>", step, reporter, i18n);
}
+ private ResourceLoader createResourceLoaderThatFindsNoFeatures() {
+ ResourceLoader resourceLoader = mock(ResourceLoader.class);
+ when(resourceLoader.resources(anyString(), eq(".feature"))).thenReturn(Collections.<Resource>emptyList());
+ return resourceLoader;
+ }
+
private Runtime createStrictRuntime() {
return createRuntime("-g", "anything", "--strict");
}
@@ -355,9 +373,17 @@ private Runtime createNonStrictRuntime() {
return createRuntime("-g", "anything");
}
+ private Runtime createStrictRuntime(ResourceLoader resourceLoader) {
+ return createRuntime(resourceLoader, Thread.currentThread().getContextClassLoader(), "-g", "anything", "--strict");
+ }
+
private Runtime createRuntime(String... runtimeArgs) {
ResourceLoader resourceLoader = mock(ResourceLoader.class);
ClassLoader classLoader = mock(ClassLoader.class);
+ return createRuntime(resourceLoader, classLoader, runtimeArgs);
+ }
+
+ private Runtime createRuntime(ResourceLoader resourceLoader, ClassLoader classLoader, String... runtimeArgs) {
RuntimeOptions runtimeOptions = new RuntimeOptions(new Properties(), runtimeArgs);
Backend backend = mock(Backend.class);
Collection<Backend> backends = Arrays.asList(backend);
View
18 core/src/test/java/cucumber/runtime/formatter/JUnitFormatterTest.java
@@ -367,6 +367,24 @@ public void should_handle_one_step_at_the_time_execution() throws Exception {
assertXmlEqual(expected, replaceTimeWithZeroTime(actual));
}
+ @Test
+ public void should_add_dummy_testcase_if_no_features_are_found_to_aviod_failed_jenkins_jobs() throws Exception {
+ final File report = File.createTempFile("cucumber-jvm-junit", ".xml");
+ final JUnitFormatter junitFormatter = createJUnitFormatter(report);
+
+ junitFormatter.done();
+ junitFormatter.close();
+
+ String actual = new Scanner(new FileInputStream(report), "UTF-8").useDelimiter("\\A").next();
+ String expected = "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n" +
+ "<testsuite failures=\"0\">\n" +
+ " <testcase classname=\"dummy\" name=\"dummy\">\n" +
+ " <skipped message=\"No features found\" />\n" +
+ " </testcase>\n" +
+ "</testsuite>\n";
+ assertXmlEqual(expected, replaceTimeWithZeroTime(actual));
+ }
+
private File runFeaturesWithJunitFormatter(final List<String> featurePaths) throws IOException {
return runFeaturesWithJunitFormatter(featurePaths, false);
}
View
65 core/src/test/java/cucumber/runtime/model/CucumberFeatureTest.java
@@ -1,48 +1,63 @@
package cucumber.runtime.model;
-import cucumber.runtime.CucumberException;
import cucumber.runtime.io.Resource;
import cucumber.runtime.io.ResourceLoader;
import org.junit.Test;
import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.io.PrintStream;
import java.util.Collections;
import static java.util.Arrays.asList;
import static java.util.Collections.emptyList;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class CucumberFeatureTest {
@Test
- public void fails_if_no_features_are_found() {
- try {
- ResourceLoader resourceLoader = mock(ResourceLoader.class);
- when(resourceLoader.resources("does/not/exist", ".feature")).thenReturn(Collections.<Resource>emptyList());
- CucumberFeature.load(resourceLoader, asList("does/not/exist"), emptyList());
- fail("Should have failed");
- } catch (CucumberException e) {
- assertEquals("No features found at [does/not/exist]", e.getMessage());
- }
+ public void succeds_if_no_features_are_found() {
+ ResourceLoader resourceLoader = mock(ResourceLoader.class);
+ when(resourceLoader.resources("does/not/exist", ".feature")).thenReturn(Collections.<Resource>emptyList());
+
+ CucumberFeature.load(resourceLoader, asList("does/not/exist"), emptyList(), new PrintStream(new ByteArrayOutputStream()));
+ }
+
+ @Test
+ public void logs_message_if_no_features_are_found() {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ResourceLoader resourceLoader = mock(ResourceLoader.class);
+ when(resourceLoader.resources("does/not/exist", ".feature")).thenReturn(Collections.<Resource>emptyList());
+
+ CucumberFeature.load(resourceLoader, asList("does/not/exist"), emptyList(), new PrintStream(baos));
+
+ assertEquals(String.format("No features found at [does/not/exist]%n"), baos.toString());
+ }
+
+ @Test
+ public void logs_message_if_features_are_found_but_filters_are_too_strict() throws IOException {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ResourceLoader resourceLoader = mock(ResourceLoader.class);
+ Resource resource = mock(Resource.class);
+ when(resource.getPath()).thenReturn("foo.feature");
+ when(resource.getInputStream()).thenReturn(new ByteArrayInputStream("Feature: foo".getBytes("UTF-8")));
+ when(resourceLoader.resources("features", ".feature")).thenReturn(asList(resource));
+
+ CucumberFeature.load(resourceLoader, asList("features"), asList((Object) "@nowhere"), new PrintStream(baos));
+
+ assertEquals(String.format("None of the features at [features] matched the filters: [@nowhere]%n"), baos.toString());
}
@Test
- public void fails_if_features_are_found_but_filters_are_too_strict() throws IOException {
- try {
- ResourceLoader resourceLoader = mock(ResourceLoader.class);
-
- Resource resource = mock(Resource.class);
- when(resource.getPath()).thenReturn("foo.feature");
- when(resource.getInputStream()).thenReturn(new ByteArrayInputStream("Feature: foo".getBytes("UTF-8")));
-
- when(resourceLoader.resources("features", ".feature")).thenReturn(asList(resource));
- CucumberFeature.load(resourceLoader, asList("features"), asList((Object) "@nowhere"));
- fail("Should have failed");
- } catch (CucumberException e) {
- assertEquals("None of the features at [features] matched the filters: [@nowhere]", e.getMessage());
- }
+ public void logs_message_if_no_feature_paths_are_given() {
+ ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ResourceLoader resourceLoader = mock(ResourceLoader.class);
+
+ CucumberFeature.load(resourceLoader, Collections.<String>emptyList(), emptyList(), new PrintStream(baos));
+
+ assertEquals(String.format("Got no path to feature directory or feature file%n"), baos.toString());
}
+
}
View
7 examples/java-no-features/.gitignore
@@ -0,0 +1,7 @@
+/.settings
+/.classpath
+/.project
+.idea
+*.iml
+.DS_Store
+
View
11 examples/java-no-features/README.md
@@ -0,0 +1,11 @@
+# java-no-features
+
+If wonder what would Cucumber JVM do when there are no feature files found at all.
+Just run these cukes by
+
+ mvn clean test
+
+## Read more
+
+ - [Jenkins fails if there are no testcases](http://jenkins-ci.361315.n4.nabble.com/Jenkins-fails-if-there-are-no-testcases-td3899185.html) Jenkins CI thread
+ - [[JVM] Exception thrown when no tags are found](https://groups.google.com/forum/#!topic/cukes/wZo0FTQJrVA) Google group discussion
View
38 examples/java-no-features/pom.xml
@@ -0,0 +1,38 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>info.cukes</groupId>
+ <artifactId>cucumber-jvm</artifactId>
+ <relativePath>../../pom.xml</relativePath>
+ <version>1.1.4-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>java-no-features</artifactId>
+ <packaging>jar</packaging>
+ <name>Examples: No Cucumber features</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>info.cukes</groupId>
+ <artifactId>cucumber-jvm-deps</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>info.cukes</groupId>
+ <artifactId>cucumber-java</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>info.cukes</groupId>
+ <artifactId>cucumber-junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+</project>
View
8 examples/java-no-features/src/test/java/cucumber/examples/java/no/features/RunCukesTest.java
@@ -0,0 +1,8 @@
+package cucumber.examples.java.no.features;
+
+import cucumber.api.junit.Cucumber;
+import org.junit.runner.RunWith;
+
+@RunWith(Cucumber.class)
+public class RunCukesTest {
+}
View
5 junit/src/test/java/cucumber/runtime/junit/CucumberTest.java
@@ -46,9 +46,10 @@ public void finds_features_based_on_explicit_root_package() throws IOException,
assertEquals("Feature: FA", cucumber.getChildren().get(0).getName());
}
- @Test(expected = CucumberException.class)
+ @Test
public void finds_no_features_when_explicit_package_has_nothnig() throws IOException, InitializationError {
- new Cucumber(ExplicitFeaturePathWithNoFeatures.class);
+ Cucumber cucumber = new Cucumber(ExplicitFeaturePathWithNoFeatures.class);
+ assertEquals(0, cucumber.getChildren().size());
}
@RunWith(Cucumber.class)
Something went wrong with that request. Please try again.