Never-ending loop wjhen the test case Failed #271

Closed
dsamek opened this Issue Apr 15, 2014 · 44 comments

Projects

None yet

9 participants

@dsamek
dsamek commented Apr 15, 2014

If I use this piece of code:

#define CATCH_CONFIG_MAIN
#include "catch.hpp"

#include "TCACAVAUtilities.h"

TEST_CASE( "TCACAVAUtilities::CreateColorFromRGB", "[CreateColorFromRGB]" ) {
    REQUIRE( TCACAVAUtilities::CreateColorFromRGB(0, 0, 0) == 255 );
}

The result is:

C:\SourceCode\CVS_ROOT\CAVA\develop_clean\win_b64\code\bin>TCACAVACreateBaseCatalog.exe
All tests passed (1 assertion in 1 test case)

But If I use this code:

#define CATCH_CONFIG_MAIN
#include "catch.hpp"

#include "TCACAVAUtilities.h"

TEST_CASE( "TCACAVAUtilities::CreateColorFromRGB", "[CreateColorFromRGB]" ) {
    REQUIRE( TCACAVAUtilities::CreateColorFromRGB(1, 0, 0) == 255 );
    REQUIRE( TCACAVAUtilities::CreateColorFromRGB(0, 0, 0) == 255 );
}

The result is:

-------------------------------------------------------------------------------
TCACAVAUtilities::CreateColorFromRGB
-------------------------------------------------------------------------------
C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(6)
...............................................................................

C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(7): FAILED:
  REQUIRE( TCACAVAUtilities::CreateColorFromRGB(1, 0, 0) == 255 )
with expansion:
  0x10000ff == 255

-------------------------------------------------------------------------------
TCACAVAUtilities::CreateColorFromRGB
-------------------------------------------------------------------------------
C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(6)
...............................................................................

C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(7): FAILED:
  REQUIRE( TCACAVAUtilities::CreateColorFromRGB(1, 0, 0) == 255 )
with expansion:
  0x10000ff == 255

-------------------------------------------------------------------------------
TCACAVAUtilities::CreateColorFromRGB
-------------------------------------------------------------------------------
C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(6)
...............................................................................

C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(7): FAILED:
  REQUIRE( TCACAVAUtilities::CreateColorFromRGB(1, 0, 0) == 255 )
with expansion:
  0x10000ff == 255

-------------------------------------------------------------------------------
TCACAVAUtilities::CreateColorFromRGB
-------------------------------------------------------------------------------
C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(6)
...............................................................................

C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACAVACreateBaseCatalog.cpp(7): FAILED:
  REQUIRE( TCACAVAUtilities::CreateColorFromRGB(1, 0, 0) == 255 )
with expansion:
  0x10000ff == 255

-------------------------------------------------------------------------------
TCACAVAUtilities::CreateColorFromRGB
-------------------------------------------------------------------------------
C:\SourceCode\CVS_ROOT\CAVA\develop_clean\TCACAVACatalog\TCACAVACreateBaseCatalo
g.m\src\TCACforrtl: error (200): program aborting due to control-C event
Image              PC                Routine            Line        Source

libifcoremd.dll    00000000100BAB14  Unknown               Unknown  Unknown
libifcoremd.dll    00000000100B46B9  Unknown               Unknown  Unknown
libifcoremd.dll    00000000100A1017  Unknown               Unknown  Unknown
libifcoremd.dll    0000000010021BF8  Unknown               Unknown  Unknown
libifcoremd.dll    000000001002DDC8  Unknown               Unknown  Unknown
kernel32.dll       00000000774F4803  Unknown               Unknown  Unknown
kernel32.dll       00000000774B652D  Unknown               Unknown  Unknown
ntdll.dll          00000000776EC541  Unknown               Unknown  Unknown

The never-ending loop which should I stop with CTRL+C.

Where is the problem ? Do I something wrong ?

@philsquared
Owner

That's weird.
There was an endlessly looping bug (although, IIRC, that involved nested sections) - which was fixed over a year ago. I've not seen anything similar since. Which version of Catch are you using?

Are you able to substitute TCACAVAUtilities::CreateColorFromRGB with something simple so you can provide a complete self-contained example (i.e. something I can run here)? - it looks like you should be able to.

I tried this:

namespace TCACAVAUtilities {
    inline int CreateColorFromRGB( int r, int g, int b ) {
        return r + ( g << 8 ) + ( b << 16 );
    }
}

but wasn't able to reproduce your issue.

@dsamek
dsamek commented Apr 22, 2014

I changed the code:

#define CATCH_CONFIG_MAIN
#include "catch.hpp"

//#include "TCACAVAUtilities.h"

inline int CreateColorFromRGB( int r, int g, int b ) {
    return r + ( g << 8 ) + ( b << 16 );
}

TEST_CASE( "TCACAVAUtilities::CreateColorFromRGB", "[CreateColorFromRGB]" ) {
    REQUIRE( CreateColorFromRGB(1, 0, 0) == 255 );
    REQUIRE( CreateColorFromRGB(0, 0, 0) == 255 );
}

But the result is same "the never-ending loop".
Maybe it's somehow connected to the my system.
I run it on:

  • Windows 7 Professional x64
  • Visual studio 2008 Version 9.0.30729.4108 QFE
  • The running exe file is created with the Rade developing tool (developing tool for CATIA V5R22)

Is there any limitation for catch usage ?

@philsquared
Owner

Hi @dsamek,

Sorry I missed your response for some reason.
Odd that you get the same behaviour even with my cut down version.
You didn't say what the version of Catch was?

I originally tried it on my Mac - but I've now tried the same test code on a Windows machine too, with both VS2008 and VS2010. Same result in each case.

Which leads me to suspect there is something to do with the "Rade developing tool" - which, I'm afraid, I know nothing about.

If you knock up a minimal test app using pure VS2008 - do you still get the same result?

@gahr
gahr commented May 27, 2014

Two consecutive sections containing failing tests trigger the recursive loop. Minimal test case:

#define CATCH_CONFIG_MAIN
#include <catch.hpp>
TEST_CASE("Never/Ending")
{
    SECTION("First is ok") {
        REQUIRE(1. == 1.);
    }
    SECTION("Second fails") {
        REQUIRE(1. == 0.);
    }
    SECTION("Third fails too") {
        REQUIRE(1. == 0.);
    }
}

Output is:

catch-loop2.cpp:15: FAILED:
REQUIRE( 1. == 0. )
with expansion:
1.0 == 0.0


Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending
Never/Ending

Third fails too

catch-loop2.cpp:4
...............................................................................

catch-loop2.cpp:15: FAILED:
REQUIRE( 1. == 0. )
with expansion:
1.0 == 0.0


etc etc etc

@dsamek
dsamek commented May 27, 2014

Used catch version:

  • CATCH v1.0 build 33 (master branch)
  • Generated: 2014-03-24 18:11:50.426554
@gahr
gahr commented May 27, 2014
  • CATCH v1.0 build 47 (master branch)
  • Generated: 2014-05-20 19:02:15.946806
@gahr
gahr commented May 27, 2014

This might help clarifying what's happening:

#define CATCH_CONFIG_MAIN
#include <catch.hpp>

TEST_CASE("T")
{
    SECTION("A") {
        REQUIRE(1. == 1.);
    }

    SECTION("B") {
        REQUIRE(1. == 0.);
    }

   SECTION("C") {
        REQUIRE(1. == 0.);
    }
}
catch-loop is a Catch v1.0 b47 host application.
Run with -? for options

-------------------------------------------------------------------------------
T
  B
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................

catch-loop.cpp:11: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

-------------------------------------------------------------------------------
T
  C
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................

catch-loop.cpp:15: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

-------------------------------------------------------------------------------
T
  T
  C
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................

catch-loop.cpp:15: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

-------------------------------------------------------------------------------
T
  T
  T
  C
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................
catch-loop.cpp:15: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

-------------------------------------------------------------------------------
T
  T
  T
  T
  C
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................

catch-loop.cpp:15: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

-------------------------------------------------------------------------------
T
  T
  T
  T
  T
  C
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................

catch-loop.cpp:15: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

-------------------------------------------------------------------------------
T
  T
  T
  T
  T
  T
  C
-------------------------------------------------------------------------------
catch-loop.cpp:4
@philsquared
Owner

Thanks @gahr. That's very odd. What compiler/ platform are you using?
I just tried your minimal repro code (thanks for that) in clang (Apple LLVM 5.1) on OS X, VS2008 and VS2010 on Windows.
In all cases everything behaved just fine (no looping).

@gahr
gahr commented May 28, 2014

I get the loop with clang 3.4 on FreeBSD. However, things work smoothly with gcc 4.7.3 and 4.8.3 20140220 (prerelease).

Looks like a compiler bug at this point..

@matalangilbert

For reference, I have this same problem using gcc 4.8.1 on Ubuntu 13.10.

@philsquared
Owner

Thanks guys - I'll have another look shortly

@stalepancakes

FWIW, I encountered the same issue today with the Factorial example from the tutorial using VS2013.
It only happens for me when exception handling is disabled (i.e. the /EHsc flag is not set)

@philsquared
Owner

Sorry - only just got back to this!

@stalepancakes: "It only happens for me when exception handling is disabled"

  • that sounds like the cause then - at least for you. Catch relies on the exception mechanism! You will not be able to use it without exceptions enabled.

@dsamek, @gahr and @matalangilbert - do you have exceptions enabled?

@gahr
gahr commented Jun 30, 2014

I do, and I'm compiling the test program I posted earlier without any options.

These two compilers fail:

FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: x86_64-unknown-freebsd11.0
Thread model: posix

FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
Target: x86_64-unknown-freebsd10.0
Thread model: posix

On the other hand, the following two compilers produce the desired behaviour.

g++47 (FreeBSD Ports Collection) 4.7.3
g++48 (FreeBSD Ports Collection) 4.8.4 20140522 (prerelease)

@dsamek
dsamek commented Jun 30, 2014

I use the Visual Studio 2008 Version 9.0.30729.4108 QFE.
I don't enable/disable the some special setting from the exception handling.
Do you mean something like: http://msdn.microsoft.com/en-us/library/4t3saedz.aspx
Some /EHsc flag ?
Or something else ?

@philsquared
Owner

Thanks for the responses. So sounds like it's not due to lack of exceptions support in your cases.
@dsamek, if you've not set anything specifically you should get exception support by default. But, yes, it's controlled through /EHsc

@matalangilbert

Exception handling is enabled for our build. We are using _GNU_SOURCE and linking with -pthread, beyond that it's a fairly standard build.

@philsquared philsquared added a commit that referenced this issue Jul 9, 2014
@philsquared Section no longer relies on copy-elision for correctness
- should address #293
- *may* address #271
23181ee
@philsquared
Owner

Guys, if you're having the issues mentioned here with Sections (note the originally posted issue did not involve Sections) then please try the latest build (build 51, a time of writing). I've fixed an issue where if copy-constructors where not elided it would screw up Section handling. I don't know if that's what's going on in your cases or not, but it's in the ballpark so thought I'd mention it.

@gahr
gahr commented Jul 9, 2014

That fixes it - thanks a lot!

#define CATCH_CONFIG_MAIN
#include <catch.hpp>
TEST_CASE("T")
{
    SECTION("A") {
        REQUIRE(1. == 1.);
    }
    SECTION("B") {
        REQUIRE(1. == 0.);
    }
    SECTION("C") {
        REQUIRE(1. == 0.);
    }
}

./catch-loop

catch-loop is a Catch v1.0 b51 host application.
Run with -? for options

-------------------------------------------------------------------------------
T
  B
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................

catch-loop.cpp:11: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

-------------------------------------------------------------------------------
T
  C
-------------------------------------------------------------------------------
catch-loop.cpp:4
...............................................................................

catch-loop.cpp:15: FAILED:
  REQUIRE( 1. == 0. )
with expansion:
  1.0 == 0.0

===============================================================================
test cases: 0 passed | 1 failed | total: 1
assertions: 1 passed | 2 failed | total: 3
@dsamek
dsamek commented Jul 9, 2014

I try it but the result is the same => never-ending loop.
BTW: in property page of my testing module is setting Enable C++ exception set to Yes(/EHsc) in visual studio. But I found something on the web about the RADE tool. This tool use the mkmk which:

  • mkmk deactivates by default C++ native exceptions
@joto
joto commented Jul 24, 2014

I have the same problem with the neverending loop (CATCH v1.0 build 52 (master branch)). Unfortunately I can't reproduce it in a small test case, but it has something to do with exceptions. In RunContext::sectionEnded() the std::uncaught_exception() is true. I am not sure what exactly is happening then, but m_testCaseTracker->leaveSection(); is not called as a result of this, so the section is not marked as completed. If I remove the if( std::uncaught_exception() ) { ... } everything seems to run fine.

@dsamek
dsamek commented Jul 25, 2014

I tried it. But in this case all CATIA exceptions are un-handled (each CATIA exception causes the crash) ...

@joto
joto commented Jul 25, 2014

I don't know what you mean with "CATIA exception". But in any case I was not proposing the removal as a fix, just to show that this is the area where the (or a) problem is.

I'll keep it running with this hack for the moment, because a test that (potentially) crashes is still better than an endless loop, which is bad news for automated testing.

@joto
joto commented Jan 16, 2015

The commit doesn't fix the issue for me.

@AlexisWilke

I just ran into this problem with library zipios++. The test works fine under Linux which uses g++ but fails and loops forever under FreeBSD with clang. I would imagine that the difference in compiler is the culprit here, but what do I know?!

If you'd like more precision about the test I have that fails, let me know.

"use Zipios++ to create zip archives with 1 or 3 files each"

The code is in this file:

https://sourceforge.net/p/zipios/git-code/ci/master/tree/tests/catch_zipfile.cpp

@AlexisWilke

After reading the entire thread and testing with Joto suggestion, I found out that I am indeed getting the same problem: std::uncaught_exception(). I'm thinking that this flag is set to true when it should not be, unless I missed something somewhere -- i.e. a throw in a destructor. So I search for such without luck thus far. However, I definitively have an exception which is caught as expected by the REQUIRE_THROWS_AS() macro.

The only particularity of that specific test is that it calls a function that rethrows an exception, although I tried to throw a new exception instead of just using throw it did not help. I think that somehow clang does not want to reset the std::uncaught_exception() flag and I'm not too sure there is anything we can do about it except adding a hack to catch.hpp if compiled by clang.

After further testing, I see that the problem happens when the REQUIRE_THROWS_AS() macro gets called. I print out the current state of the std::uncaught_exception() function and it returns 'true' after the macro, but false before:

std::cerr << "BEFORE: info: uncaught status: " << (std::uncaught_exception() ? "YES" : "NO") << "\n";
REQUIRE_THROWS_AS(...);
std::cerr << "AFTER: info: uncaught status: " << (std::uncaught_exception() ? "YES" : "NO") << "\n";

So... even though the REQUIRE_THROWS_AS() catches the exception as expected, the process of clearing the uncaught exception flag is not working right. This has got to be a problem in the compiler.

Okay, I found a fix in my case, although this is not really a valid general fix. Really clang is doing it wrong. I had two levels that would rethrow. Removing one of the levels fixed the problem. Something like this:

void func()
{
  try { that(); } catch(...) { throw; }
}
void that()
{
  try { there(); } catch(...) { throw; }
}

The fact is I don't really need the try/catch in the that() function and removing it solves the std::uncaught_exception() problem. Yet, I would imagine that this is a bug in clang and not in catch.

@philsquared
Owner

Thanks for the additional info, @AlexisWilke. Interesting that it seems to be a clang thing as I use clang on Mac OS X and have never seen the issue. I can't see why something like this would be platform dependent - but the Mac version is Apple's own fork, so it could be that.
I need to look into this more, again. I'm not against putting in a conditional workaround for clang, if we can find one general enough.
I'm not really getting any time to look at Catch at the moment, as I have a couple of big things going on. I'm hoping to be back on it in a couple of weeks.

@AlexisWilke

It could be that the clang people have a fix which did not make it in the version used by FreeBSD. I'm just using the latest FreeBSD in a virtual host (10.1). If you'd like to test with my test case, you can get zipios++ from the git repository on SourceForge.net and then compile it. You may put catch.hpp in the contrib/... directory so the tests get compiled. You may remove the #ifndef clang although running all the tests in one go generate the problem too. I now have a script that runs the tests one by one and except for the few that are "protected" with the #ifndef, all the other tests pass. Obviously that means I don't cover as much code, but I still think zipios is very well tested under FreeBSD too at the moment.

@joto
joto commented Apr 9, 2015

I have the problem whether I compile with clang or with gcc, so I don't think it is a compiler bug. (Tested on Debian Jessie.)

@AlexisWilke

Joto, just in case I tested my code under Debian (Wheezy and Jessie) and it worked just fine. Are you using the latest version of catch.hpp? phil made some fixes at some point which may not have been updated?

@kirbyfan64

Considering the fact that this can be so hard to reproduce, this might be a C++ stdlib/ABI bug. Can someone who has the bug try this with libc++ or libcxxabi?

@joto
joto commented Apr 13, 2015

@AlexisWilke just tried CATCH v1.1 build 14 and it still hangs with GCC 4.9.2 and Clang 3.5.0.

@AlexisWilke

I tried now with g++ 4.9.1. I suppose the latest libstd++ (or at least the one in 4.9.2) is different enough to were you would get that problem. Either that, or my code does not expose the problem in g++.

How could I test with the latest under Ubuntu? Any easy apt-get install stuff that would help? Actually, I will look for version 15.04. Maybe it has a version that breaks.

@joto
joto commented Apr 14, 2015

Okay, I have boiled down the case I have been having trouble with to a few simple test cases. Here is the code:

#define CATCH_CONFIG_MAIN
#include "catch.hpp"
#include <future>
#include <stdexcept>

TEST_CASE("value") {
    SECTION("works") {
        std::promise<int> result;
        result.set_value(42);
        REQUIRE(result.get_future().get() == 42);
    }
}

TEST_CASE("exception") {
    SECTION("throws") {
        std::promise<int> result;
        try {
            throw std::runtime_error("Example");
        } catch(...) {
            result.set_exception(std::current_exception());
        }
        REQUIRE(result.get_future().get() == 42);
    }
}

TEST_CASE("require") {
    SECTION("hangs") {
        std::promise<int> result;
        try {
            throw std::runtime_error("Example");
        } catch(...) {
            result.set_exception(std::current_exception());
        }
        REQUIRE_THROWS_AS(result.get_future().get(), std::runtime_error);
    }
}

Save as catch_hang.cpp and compile: g++ -std=c++11 catch_hang.cpp -o catch_hang -pthread. Use the compiler of your choice, g++ and clang++ show the same behaviour for me.

Now run these commands in turn:

./catch_hang value        # works as it should
./catch_hang exception # shows that an exception is thrown from the get()
./catch_hang require     # problem case

The last case hangs forever with CATCH v1.0 build 53. With CATCH v1.1 build 14 it also hangs, but when I hit CTRL-C to break it off, it prints 'require' forever and I have to hit CTRL-C again to stop it.

cppreference.com tells me that uncaught_exception() "Detects if the current thread has a live exception object", so maybe it doesn't detect the exception "hidden" in the promise?

@kirbyfan64

I get a slightly different result:

$ ./catch_hang require
# hangs here
^C
require
require
require
require
require
require
require
require
require
require
require
require
require
require
require
require
require
require
require
require
require
...
^Crequire
^Crequire
...
  require
  require
  hangs
-------------------------------------------------------------------------------
loop.cpp:26
...............................................................................

loop.cpp:27: FAILED:
due to a fatal error condition:
  SIGINT - Terminal interrupt signal

===============================================================================
test cases:     1 |     0 passed | 1 failed
assertions: 68313 | 68312 passed | 1 failed

ryan@DevPC-LX:~/langtest/catch$ ^C

The ^C are the times when I pressed control-C to kill the program.

@kirbyfan64

This is a bug in libstdc++. Compiling the program with -stdlib=libc++ works correctly.

Looking into filing a bug report...

@kirbyfan64

Crap, I can't file one because "user account creation has been restricted."

Could someone else look into that?

@philsquared
Owner

For anyone still watching this issue - I've committed some changes (on Develop) that may address it. Please let me know either way if you try it.

@joto
joto commented Sep 27, 2015

@philsquared This fixes it for me! Thanks.

@philsquared
Owner

Thanks for letting me know, @joto.
Anyone else?

@gahr
gahr commented Oct 2, 2015

@philsquared Latest master fixes it, with FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512.

Thanks!

@philsquared
Owner

Doing some clean-up. I know it's been a while, so I don't know if the original posters are still watching, or interested - but it would be nice to be able to close this.
@dsamek, @kirbyfan64, @AlexisWilke, @matalangilbert (sorry if this is a blast from the past)?

@AlexisWilke

From what I recall, it works for me now.

@horenmar horenmar closed this Feb 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment