Skip to content

lein test returns exit code 0 for multiples of 256 test failures+errors #2718

Closed
@frenchy64

Description

@frenchy64

Initial debugging steps
Before creating a report, especially around exceptions being thrown when running Leiningen, please check if the error still occurs after:

- [x] Updating to using the latest released version of Leiningen (`lein upgrade`).
- [x] Moving your `~/.lein/profiles.clj` (if present) out of the way. This contains third-party dependencies and plugins that can cause problems inside Leiningen.
- [x] Updating any old versions of plugins in your `project.clj`, especially if the problem is with a plugin not working. Old versions of plugins like nREPL and CIDER (as well as others) can cause problems with newer versions of Leiningen.
- [x] (If you are using Java 9 or newer), updating your dependencies to their most recent versions. Recent JDK's have introduced changes which can break some Clojure libraries.

Describe the bug
lein test returns a 0 exit code for multiples of 256 test failures+errors.

To Reproduce

(defproject lein-test-exit-code "0.1.0-SNAPSHOT"
  :dependencies [[org.clojure/clojure "1.10.1"]])
(ns lein-test-exit-code.core-test
  (:require [clojure.test :refer [deftest is]]))

(defmacro gen-failing-deftests [n]
  `(do 
     ~@(for [i (range n)]
         `(deftest ~(symbol (str "failing-" i))
            (is false)))))

(gen-failing-deftests 256)

Actual behavior

$ lein test
...
...
Ran 256 tests containing 256 assertions.
256 failures, 0 errors.

$ echo $?
0

Expected behavior
A non-zero exit code should be returned for test failures.

Environment

  • Leiningen Version:
Leiningen 2.9.5 on Java 1.8.0_275 OpenJDK 64-Bit Server VM
  • Leiningen installation method: manual
  • JDK Version:
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_275-b01)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.275-b01, mixed mode)
  • OS: macOS 10.15

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions