Skip to content

lein test returns exit code 0 for multiples of 256 test failures+errors #2718

Description

@frenchy64

Initial debugging steps
Before creating a report, especially around exceptions being thrown when running Leiningen, please check if the error still occurs after:

- [x] Updating to using the latest released version of Leiningen (`lein upgrade`).
- [x] Moving your `~/.lein/profiles.clj` (if present) out of the way. This contains third-party dependencies and plugins that can cause problems inside Leiningen.
- [x] Updating any old versions of plugins in your `project.clj`, especially if the problem is with a plugin not working. Old versions of plugins like nREPL and CIDER (as well as others) can cause problems with newer versions of Leiningen.
- [x] (If you are using Java 9 or newer), updating your dependencies to their most recent versions. Recent JDK's have introduced changes which can break some Clojure libraries.

Describe the bug
lein test returns a 0 exit code for multiples of 256 test failures+errors.

To Reproduce

(defproject lein-test-exit-code "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.10.1"]])
(ns lein-test-exit-code.core-test
  (:require [clojure.test :refer [deftest is]]))

(defmacro gen-failing-deftests [n]
  `(do 
     ~@(for [i (range n)]
         `(deftest ~(symbol (str "failing-" i))
            (is false)))))

(gen-failing-deftests 256)

Actual behavior

$ lein test
...
...
Ran 256 tests containing 256 assertions.
256 failures, 0 errors.

$ echo $?
0

Expected behavior
A non-zero exit code should be returned for test failures.

Environment

  • Leiningen Version:
Leiningen 2.9.5 on Java 1.8.0_275 OpenJDK 64-Bit Server VM
  • Leiningen installation method: manual
  • JDK Version:
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.275-b01, mixed mode)
  • OS: macOS 10.15

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions