add timeouts to tests #348
Comments
|
Original comment by
|
Original comment by |
Original comment by |
Original comment by |
Original comment by |
Original comment by |
Original comment by |
Original comment by |
|
Any update on this? This is pretty important functionality for a testing suite... |
|
How would you propose implementing this? We could set a timer at the beginning of a test and then start the work for the test. When that timer expires, what exactly would you expect googletest to DO? If the code being tested isn't cancellable, what does it mean to halt that code with a timeout? What state is the test process in at that point? The test most likely will have objects that are unfinalized and abandoned, even if we could figure out how to stop the CPU from evaluating the testing thread's code. I don't think it's an "essential" feature, so I'd like to just treat this as an ordinary feature request. I think it's nice to have if the client code has hooks to support it, but gtest is still quite useful without it. Googletest does have test hooks in which a client might install its own timeout functionality in a way that the software-under-test will be able to respond to. That requires no changes to googletest, AFAICT. I don't see how googletest a generic one-size-fits-all hook for timeout cancellation, though. |
|
Billy, could you point us to the "test hooks" you mention? I would like to give it a try. Thanks! |
|
Sure. See if this does what you need: On Sat, Sep 26, 2015 at 3:10 AM, Marco Molteni notifications@github.com
|
Billy, a sample test case such as this one (from the doc you supplied above) that demonstrate the use of test hooks to test for timeouts would be useful. |
|
+1 Answering @BillyDonahue - here's how I would implement this feature:
|
|
Here's basically how I've implemented this for sub-second thresholds on a Unix OS : For example the following should generate a fatal failure message: And the following should not: Note that this code has room for improvement. For instance: |
|
@louis-langholtz, you suggest a nice implementation, but answers the needs very partially.
Perhaps an implementation based on the same idea, but implemented in the code that invokes the tests, would be an effective solution. |
|
@ugoren longjmp past destructors is undefined behaviour in C++. I think fork (or moral equivalent on non-Posix) is going to be required for a fully robust solution. |
|
@MartinBonner As much as I found it fun just using the alarm for this, I agree that it's not robust enough for a general solution. Too bad |
|
@louis-langholtz I found your suggestion really useful and I'm using it for a project I'm building. I thought of extending it a bit so you can create a timed block. In the case of Boost's unit test, it provides a parameter for specifying a timeout for a test case. I'm not a C++ expert and don't know enough about google test to attach your code to the TEST macro. I thought of creating timed blocks instead. #include <gtest/gtest.h>
#include <unistd.h>
#include <setjmp.h>
#include <signal.h>
static jmp_buf jmp_env;
static void catch_alarm(int sig)
{
longjmp(jmp_env, 1);
}
#define ASSERT_TIMED_BLOCK_START(name, usecs) { \
const auto val = setjmp(jmp_env); \
if (val == 1) {\
GTEST_FATAL_FAILURE_(#usecs " usecs timer tripped for " #name); \
} else { \
signal(SIGALRM, catch_alarm); \
ualarm((usecs), 0); \
} }
#define ASSERT_TIMED_BLOCK_END() { \
ualarm(0, 0); \
} |
|
I found a better way to do this inspired by this post. #define ASSERT_DURATION_LE(secs, stmt) { \
std::promise<bool> completed; \
auto stmt_future = completed.get_future(); \
std::thread([&](std::promise<bool>& completed) { \
stmt; \
completed.set_value(true); \
}, std::ref(completed)).detach(); \
if(stmt_future.wait_for(std::chrono::seconds(secs)) == std::future_status::timeout) \
GTEST_FATAL_FAILURE_(" timed out (> " #secs \
" seconds). Check code for infinite loops"); \
} |
|
We are not going to add timeouts to tests. This can be accomplished in many ways and there is very little benefit to support it in the framework itself |
|
@gennadiycivil I use GoogleTest to autograde programming assignments, and am pushing for it to be adopted in our department. It would be much better if we didn't have to create our own timeout system that needs to track changes to and be maintained with google test but that timeouts be included in GT. It may be easy for one company or project to have their own timeout facility and share it with each other, but getting anything adopted widely in academia is difficult. Unless it's built in, every prof. will end up having their own TA create this function. Sometimes their solution won't be very good, and that will turn into bad stories and hinder adoption of GT for grading programs. |
|
Bryan
Thank you for this. Could you please explain in details how it *would* work.
…On Wed, May 15, 2019, 12:37 Bryan Lunt ***@***.***> wrote:
@gennadiycivil <https://github.com/gennadiycivil> I use GoogleTest to
autograde programming assignments, and am pushing for it to be adopted in
our department. It would be much better if we didn't have to create our own
timeout system that needs to track and maintain google test but that
timeouts be included in GT.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#348?email_source=notifications&email_token=ADBFEMW5SAAHJKHIDR5EGDTPVQ333A5CNFSM4BQDS432YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPHLDQ#issuecomment-492729742>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADBFEMT2CKYVSSBQVJOKIX3PVQ333ANCNFSM4BQDS43Q>
.
|
|
@gennadiycivil Thanks for the quick reply. I'm not really familiar enough with the inner workings of GT to say yet how it would or could be done. Currently I have a small library of .hpps and things I can include that came from others that allow me to use GT with MPI and a quick-n-dirty timeout thing copied verbatim from here: http://antonlipov.blogspot.com/2015/08/how-to-timeout-tests-in-gtest.html Just trying to make it work ATM. (In fact, that doesn't work for me. Time to learn all the newfangled bells and whistles added to C++ in the last 20 years since I learned it. ...) |
|
Apologies, I was not clear.
Please explain your use-case scenario, how this would work for you as a
user, not how it should be implemented.
Thanks G
…On Wed, May 15, 2019 at 12:51 PM Bryan Lunt ***@***.***> wrote:
@gennadiycivil <https://github.com/gennadiycivil> Thanks for the quick
reply.
Unfortunately I can't yet explain how it would work. I've been using the
linux commanline "timeout" utility and test filters. However, that means I
need to have a lot of post-processing scripts. No .json output is created
when a timeout happens under that system.
I'm not really familiar enough with the inner workings of GT to say yet
how it would or could be done.
Currently I have a small library of .hpps and things I can include that
came from others that allow me to use GT with MPI and a quick-n-dirty
timeout thing copied verbatim from here:
http://antonlipov.blogspot.com/2015/08/how-to-timeout-tests-in-gtest.html
Just trying to make it work ATM.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#348?email_source=notifications&email_token=ADBFEMWZRPVNCDOFMXFSHALPVQ5RHA5CNFSM4BQDS432YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPISKY#issuecomment-492734763>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADBFEMWFHJBPGDC6B3ZPTSTPVQ5RHANCNFSM4BQDS43Q>
.
|
|
Oh, sorry. The simplest thing (from a user perspective) might be to add versions of all the TEST macros that include a timeout parameter. Of course some tests may have memory management inside them, in that case, you might need an additional macro/something that behaves like the "finally" clause of exception handling. Now that I've gotten the macros I linked to working, I do like them, because I can give a failure message when timeouts happen. Also this one lets me set the timeout for individual parts of the test, which is nice. Anyway, I really do need timeouts, because students submit code that has infinite loops or communication deadlocks and it's added quite a lot of complexity to my autograders to deal with that. |
|
Bryan:
Sorry you are still not answering my questions. Lets put aside for a second
macros, scripts anything like that.
Please explain the process how students wold submit their code and how
googletest is used to automatically grade the answers. Keep the explanation
on the workflow level, what happens , what happens after that, etc. I am
trying to understand where timeouts fit in all this
Thanks
G
…On Wed, May 15, 2019 at 1:02 PM Bryan Lunt ***@***.***> wrote:
Oh, sorry.
The simplest thing (from a user perspective) might be to add versions of
all the TEST macros that include a timeout parameter.
Of course some tests may have memory management inside them, in that case,
you might need an additional macro/something that behaves like the
"finally" clause of exception handling.
Now that I've gotten the macros I linked to working, I do like them,
because I can give a failure message when timeouts happen. Also this one
lets me set the timeout for individual parts of the test, which is nice.
Anyway, I really do need timeouts, because students submit code that has
infinite loops or communication deadlocks and it's added quite a lot of
complexity to my autograders to deal with that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#348?email_source=notifications&email_token=ADBFEMV3VCEQOYJVJD2PH4DPVQ62RA5CNFSM4BQDS432YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPJQPA#issuecomment-492738620>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADBFEMUWX4TMR6DABITWYQ3PVQ62RANCNFSM4BQDS43Q>
.
|
|
Ah. Students submit by checking their code into a git repo. Eventually we want to have continuous integration do this, but for the moment I manually submit grading jobs to our compute cluster. (And there are some security reasons to prefer this over CI.) There is a set of scripts that checkout their repo, checkout my repo, copy only the files they were allowed to change into my fresh repo, build, run GT, run GBenchmark, and at the end a python script interprets the GT and GBench .json files to create a grade (also a .json file). It's a parallel programming class, so there's still some difficulty in integrating this directly into a CI system. Anyway, when the student code deadlocks, the first version of the grading system doesn't reach the end and I don't get output. I have improved it to use the "timeout" commandline program. However, that adds several degrees of complication. I can no longer run GT once if it's an assignment with sub-parts, I have to run it for each sub-part of the assignment. It's not so bad, but now the grader has to be ready to process multiple parts, etc. That's a whole lot of exception handling I have to write. Most of it is in BASH. If GT had timeouts, I could at least trust that I would still get GT output in those cases. |
|
I think this also has implications for any project running CI. With timeouts in GT, the continuous integration output would still reach the end and they would know precisely which function deadlocked. Otherwise, they just see that their CI system ran for too long. |
|
@bryan-lunt if it helps, I tried addressing the timeout issue using the macro I posted earlier in the thread #348 (comment). I then combine it with the usual google tests. Something like // run some_function() and compared with some_value
// but end the function if it exceeds 3 seconds
ASSERT_DURATION_LE(3, {
ASSERT_EQ(some_function(), some_value);
});I made some other modifications if you are interested, but the gist of it is to run the same google tests, but have a timeout in place to end the function if it exceeds the threshold. |
|
The issue with trying to add timeouts to the general-purpose testing framework is that it does not really "fit". @bryan-lunt I believe your particular problem would be properly solved by doing proper CI , for example Travis has build-in timeouts for any runs. There are good suggestions in this thread. I would recommend either implementing them in your fork for your specific case and using your fork instead of the googletest master. Thanks |
|
@gennadiycivil Ok, Thanks. (And yes, automatically forking for each test would totally undermine my use-case. That will never play well with MPI. ) As to proper CI, there are a lot of institutional constraints I have to work within. In academia, we can never get what we really want. |
|
Hmm this is odd, I copied the link to my previous post but it gave a link to another. Anyway, this is what I actually meant ... Create a new macro to address timeouts. #define ASSERT_DURATION_LE(secs, stmt) { \
std::promise<bool> completed; \
auto stmt_future = completed.get_future(); \
std::thread([&](std::promise<bool>& completed) { \
stmt; \
completed.set_value(true); \
}, std::ref(completed)).detach(); \
if(stmt_future.wait_for(std::chrono::seconds(secs)) == std::future_status::timeout) \
GTEST_FATAL_FAILURE_(" timed out (> " #secs \
" seconds). Check code for infinite loops"); \
} |
|
Is it not possible to add --gtest_timeout=5000 option? (for setting maximum test case run-time for all test cases to 5000 milliseconds) This will help avoid a test-case running forever and spamming the console. (which makes it hard to find out which test case is failing because all google test output messages are cleared) |
|
How about an assertion that repeatedly retries until it passes? If it doesn't pass within a time limit or number of attempts then it gives up and the test fails. I've found that approach useful when testing some concurrent code. The assertion might look like this: ASSERT_EVENTUALLY_TRUE(condition, numAttempts, timeBetweenAttempts)For example, let's say we've got a classic producer-consumer architecture. Something is produced in one thread and we want to ensure that it is later consumed in another thread. We don't know exactly when it will happen but we know it should happen within a certain time frame. The test might look like this: Queue q;
Producer p{ queue };
Consumer c{ queue };
p.produceSomething();
ASSERT_EVENTUALLY_TRUE(c.hasConsumedSomething(), 10, 100ms);The assertion will immediately evaluate Doing it this way means the test author has complete responsibility for thread/process synchronisation and cleanup. It won't be able to detect a hang or deadlock, nor can it kill long-running code. However, hopefully it's still useful enough for some situations and fairly simple to implement. I'd be happy to work on it if it sounds reasonable. |
|
Hi people! I am looking for something similar mentioned by @daravi Unfortunately, I trusted developers that they write robust automated tests (which do not hang - lock up/waiting for user input) but unfortunately everyone is a human and makes mistaked 😅 Though it makes my interactive session hang forever and I cannot get tests output back even if I kill the session. What I tried to implement was to run the tests individually and give them timeout (scheduling them on a Job) but unfortunately it does not close the device gracefully (sometimes the device is left in the unresponsive state if we just kill the process - needs reconnecting to reset the state which I cannot afford time-wise (we have thousands of tests to run)). However I just wanted to add up to the request - it would be cool to get this option into your framework :) We use it not only for (quick) unit tests but also for integration and system tests with real devices (as described above) and it is extremely useful to us. Your framework is very powerful and I am glad we could use it but it could be even cool-er 😏 Cheers! |
How do we achieve those ways via command line? If I try to pass my own variable for specifying the timeout, it will not run and tell that I should use only the ones which GTest supports. Do you think |
|
Both Bazel and CMake (through CTest) support timeouts. This is how most people run their tests. |
|
That would only apply to running a whole testsuite program not a single test. Using gtest_filter to run each individual tests would be too much boilerplate and overhead. |
Original issue reported on code.google.com by
avia...@gmail.comon 29 Dec 2010 at 7:15The text was updated successfully, but these errors were encountered: