Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Force running eunit tests via -k cmdline switch #254

Closed
wants to merge 3 commits into from

6 participants

@kpy3

rebar fails when there is eunit's test failed in the project or it's dependencies.
This patch forces to run all unit tests despite of any errors.

This might be useful in conjuction with CI server (for example, Jenkins) in case
when someone want run all eunit tests (include dependencies) for later
analyzing.

@kpy3

In original rebar if one of eunit test fails rebar will abort with rebar_abort reason.
With my patch it will continue working until all eunit tests will finish.

Here is the example. First run without --keep-going cmdline switch:

% ./rebar -C rebar-dev.config eunit apps=gh_json_lib,accountdb
==> gh_json_lib (eunit)
======================== EUnit ========================
module 'gh_json_lib'
gh_json_lib: compare_equal_test...ok
gh_json_lib: compare_not_equal_test...ok
gh_json_lib: compare_recursive_and_list_equal_test...ok
module 'gh_json_lib_tests'
gh_json_lib_tests: get_value_test...ok
gh_json_lib_tests: get_keys_test...ok
gh_json_lib_tests: get_values_simple1_test...ok
gh_json_lib_tests: get_values_simple2_test...ok
gh_json_lib_tests: get_values_complex_test...ok
gh_json_lib_tests: merge_with_fun_3_test...failed
::{assertion_failed,[{module,gh_json_lib_tests},
{line,122},
{expression,"Res1to2 =:= D1to2"},
{expected,true},
{value,false}]}

gh_json_lib_tests: merge_with_fun_1_test...ok
gh_json_lib_tests: merge_with_fun_2_test...ok
[done in 0.024 s]

[done in 0.033 s]

Failed: 1. Skipped: 0. Passed: 10.
ERROR: One or more eunit tests failed.
ERROR: eunit failed while processing /Users/yes/Projects/exante/bocore/deps/gh_json_lib: rebar_abort
%

Compare it with:

% ./rebar --keep-going -C rebar-dev.config eunit apps=gh_json_lib,accountdb
==> gh_json_lib (eunit)
======================== EUnit ========================
module 'gh_json_lib'
gh_json_lib: compare_equal_test...ok
gh_json_lib: compare_not_equal_test...ok
gh_json_lib: compare_recursive_and_list_equal_test...ok
module 'gh_json_lib_tests'
gh_json_lib_tests: get_value_test...ok
gh_json_lib_tests: get_keys_test...ok
gh_json_lib_tests: get_values_simple1_test...ok
gh_json_lib_tests: get_values_simple2_test...ok
gh_json_lib_tests: get_values_complex_test...ok
gh_json_lib_tests: merge_with_fun_3_test...failed
::{assertion_failed,[{module,gh_json_lib_tests},
{line,122},
{expression,"Res1to2 =:= D1to2"},
{expected,true},
{value,false}]}

gh_json_lib_tests: merge_with_fun_1_test...ok
gh_json_lib_tests: merge_with_fun_2_test...[0.001 s] ok
[done in 0.025 s]

[done in 0.033 s]

Failed: 1. Skipped: 0. Passed: 10.
==> accountdb (eunit)
======================== EUnit ========================
module 'accountdb'
module 'accountdb_app'
module 'accountdb_database_admin'
module 'accountdb_SUITE'
module 'ad_audit'
module 'ad_cache'
module 'ad_client_info'
module 'ad_common_info'
module 'ad_daily'
module 'ad_event'
module 'ad_event_sender'
module 'ad_storage'
module 'ad_storage_meta'
module 'ad_sup'
module 'ad_transaction'
module 'ad_transaction_sup'
module 'ad_transaction_writer'
module 'ad_user'
module 'ad_utils'
There were no tests to run.
%

As you can see in second run (with --keep-going cmdline switch)
rebar doen't stop after failed test (as in first run)
but continues to run.

@kpy3 kpy3 Merge branch 'master' into eunit_abort_on_error
* master:
  Added Travis CI Hook and Build Status
  Append os family to arch string
  rebar_utils: move internal fun to proper place
  Add missing comment
  Update -c string
  Fix #252 (Reported-by: Maxim-Vladimirsky)
  rebar_ct: rename function for clarity
fe731b8
@hyperthunk

I think this has been implemented already, albeit not by merging this patch. Can you take a look @kpy3 - maybe this pull can get closed?

@kpy3

Just tested latest build from master, rebar stops running eunit target after the first failed test suit (see first example in my previous comment) and runs next target. With my patch it will finish running eunit target (collecting information about passed and failed tests from all eunit tests with dependencies) and when run next target.
Since this changes semantic of the -k cmdline switch may be I need revert back to config flag for eunit?

@hyperthunk

Good question. @tuncer how should this work?

@tuncer

@dizzyd may have been inspired by 'make -k'. @dizzyd how fine-grained (on what level) should -k keep going in case of failure?

@tuncer tuncer was assigned
@kpy3 kpy3 Merge branch 'master' into eunit_abort_on_error
* master: (148 commits)
  eunit: analyze coverage only for cover compiled modules
  xref: allow multiple behaviours and ignore_xref pragmas
  Allow script to be evaluated when app file is loaded
  Use R15B02 EUnit {test,M,F} primitive as suggested by Richard Carlsson
  Correctly use release_handler:create_RELEASES
  Update custom xref query
  Add Ryan Zezeski to THANKS file
  Pass compile flags to protobuffs
  Fix inttest/ct2 ct spec file location (Thanks Peter Andersson)
  inttest/ct2: enable rebar DEBUG log level
  rebar_ct: ?DEBUG log ct_run command
  rebar_ct: do not export variable from case
  Update port_env for Mountain Lion (Noticed-by: Tony Rogvall)
  Comment eunit_test:function_wrapper/2 TODO
  Update custom xref query (eunit_test calls)
  rebar_eunit: adapt to R15B02 changes
  rebar_reltool: remove blank lines
  Add Dmitriy Kargapolov to THANKS file
  Add dynamic .script support for overlay vars file
  Add Markus Nasman to THANKS file
  ...

Conflicts:
	src/rebar_eunit.erl
545e5a9
@yfyf

Bump. What is the status of this?

@dizzyd

After some thought, I'm going to not merge this patch. All the other commands (like compile, clean) will terminate at the first error -- i.e. the next command will run, not the next command on the next dependency. Example...

  • I have app A with two dependencies B and C [A -> B -> C]
  • B has a compilation error
  • ./rebar -k compile will compile C, then try to compile B (which fails) -- it then exits, never attempting to compile A.

So, adding this granularity to -k would be inconsistent with the rest of the commands in the system (and I'm not all that keen on trying to add special handling for every command).

@dizzyd dizzyd closed this
@yfyf

Hm, well I understand what you mean, but IMHO a test-case failure is quite different from a compilation error. As mentioned in the original description, this is a very needed thing for CI/reporting purposes. Is there a reasonable way to introduce such functionality for the specific case of rebar eunit?

(Btw, obviously this problem only arises for people who are unlucky to use projects with multiple applications in them, but there are quite a few projects like that and as I understand that kind of structure is supported by rebar.)

@hyperthunk

Well you've got a couple of options now. One is to fork rebar, but I wouldn't personally recommend that when you know the change you're maintaining has been rejected by the upstream. Another choice would be to wrap the runit invocation in a makefile tht calls runit. Another option is to write a plugin and catch the thrown exception provided when -k is passed. The final option is to rework the eunit functionality in a standalone plugin and use that instead. Again, the cost of keeping in sync with upstream changes/fixes is likely to be prohibitive.

Personally I wouldn't do any of these. If you have multiple dependencies in your project then your ci server can handle this for you. Set up a build pipeline where the dependencies are built first and subsequent project builds aren't triggered unless these builds pass. This is fairly easy to setup in bamboo and probably other ci servers too.

@kpy3

Yes, we will switch to multi-project configuration in near future, so it seems this patch not needed anymore.

@yfyf

@hyperthunk Dependencies are not of concern, this is ignoring the setup with multiple apps using {sub_dirs, [..]} in rebar.config. It can be done with external tools, but then I have to maintain the app list and so on it in two places, instead of just being able to do rebar eunit --test-til-you-drop in the project root. If multiple-apps are not to be used in rebar projects, then I think the semantics of sub_dirs must be somewhat changed, no?

@dizzyd

The crux of what is being proposed here (as I read the patch) is that eunit should NEVER generate a failure when -k (or whatever) is set. Even in a CI setup this seems pretty odd to me, since you wouldn't be able to distinguish a good build from a bad build, right?

@yfyf

I don't think the patch is appropriate. It should still generate a failure at the end, but should not stop executing.

@rvangraan

Hi guys. My project has just run into exactly this problem. We use Jenkins as CI tool and I wanted to use rebar to run all the eunit tests for me. But it stops at the first failed application. So instead of giving me a good overview of the status of my system, I just know that one test failed and not that 10,000 others passed. I don't have a problem with rebar returning a failure at the end, but I need it to run all the tests. In our project we have a habit to leave some tests breaking so that we can come back to them later when we have time. But we cannot work this way if rebar behaves as it does. Is my only option to fork rebar and use the patch?

@hyperthunk

@rvangraan - right, and what you need is exactly what @yfyf proposed. The reason @dizzyd didn't merge this is that is doesn't do what we want (viz run all the tests but then fail at the end if there were any errors).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 20, 2012
  1. @kpy3
Commits on Jun 28, 2012
  1. @kpy3

    Merge branch 'master' into eunit_abort_on_error

    kpy3 authored
    * master:
      Added Travis CI Hook and Build Status
      Append os family to arch string
      rebar_utils: move internal fun to proper place
      Add missing comment
      Update -c string
      Fix #252 (Reported-by: Maxim-Vladimirsky)
      rebar_ct: rename function for clarity
Commits on Sep 14, 2012
  1. @kpy3

    Merge branch 'master' into eunit_abort_on_error

    kpy3 authored
    * master: (148 commits)
      eunit: analyze coverage only for cover compiled modules
      xref: allow multiple behaviours and ignore_xref pragmas
      Allow script to be evaluated when app file is loaded
      Use R15B02 EUnit {test,M,F} primitive as suggested by Richard Carlsson
      Correctly use release_handler:create_RELEASES
      Update custom xref query
      Add Ryan Zezeski to THANKS file
      Pass compile flags to protobuffs
      Fix inttest/ct2 ct spec file location (Thanks Peter Andersson)
      inttest/ct2: enable rebar DEBUG log level
      rebar_ct: ?DEBUG log ct_run command
      rebar_ct: do not export variable from case
      Update port_env for Mountain Lion (Noticed-by: Tony Rogvall)
      Comment eunit_test:function_wrapper/2 TODO
      Update custom xref query (eunit_test calls)
      rebar_eunit: adapt to R15B02 changes
      rebar_reltool: remove blank lines
      Add Dmitriy Kargapolov to THANKS file
      Add dynamic .script support for overlay vars file
      Add Markus Nasman to THANKS file
      ...
    
    Conflicts:
    	src/rebar_eunit.erl
This page is out of date. Refresh to see the latest.
Showing with 7 additions and 3 deletions.
  1. +7 −3 src/rebar_eunit.erl
View
10 src/rebar_eunit.erl
@@ -140,14 +140,18 @@ run_eunit(Config, CodePath, SrcErls) ->
ok
end,
+ KeepGoing = rebar_config:get_global(keep_going, false),
+
%% Stop cover to clean the cover_server state. This is important if we want
%% eunit+cover to not slow down when analyzing many Erlang modules.
ok = cover:stop(),
- case EunitResult of
- ok ->
+ case {EunitResult, KeepGoing} of
+ {ok, _} ->
ok;
- _ ->
+ {_, true} ->
+ ok;
+ {_, false} ->
?ABORT("One or more eunit tests failed.~n", [])
end,
Something went wrong with that request. Please try again.