-
Notifications
You must be signed in to change notification settings - Fork 130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to find the input that generates an assert failure in the test tests/file/large_int.bas #128
Comments
Sorry for the slow reply; I've been really busy. Oh, that's a cool project! Did fbc end up in that dataset, with just 8 other programs, just because it meets all the criteria? It seems strange though, because fbc (the compiler) is written in FreeBasic, not C, so its inclusion appears invalid. Only the runtime libraries are written in C. FB is exceedingly closely related to C though; it is basically a subset of C++98 with BASIC syntax (optionally QuickBASIC-compatible) and standard library. The bug you are investigating may be in either FB or C code, I don't know. You're going to have to edit file/large_int.bas, by adding this line before the failing |
@CharlesDDNoble, yes, interesting project. We consider unit testing an important part of our development, so very cool to see fbc used for input and study in your research. The test-suite's CU_ASSERT_* macros generate a generic message that is suitable for most cases where we want to know the expression that failed at a specific file/line number. But as you have discovered, since the context of the assertion is dependent on an input file, the full context is not reported. The older versions of fbc use CUnit for the testing framework. Around December 2017, we re-implemented it as CUnit (fbc 1.05.0 and earlier) fbc's CU_ASSERT_DOUBLE_EQUAL is defined in inc/CUnit/CUnit.bi
I think the most direct way to get more specific results but still in standard format is to define a new assertion macro, for example,
Then use the new macro, for more specific results:
fbcunit (fbc 1.06.0 and later) This might make a nice addition, or some variation of it, to current fbcunit test-suite API. This framework is located in https://github.com/freebasic/fbc/tree/master/tests/fbcunit Solution would be similar:
Where fbcu.CU_ASSERT_ is declared as:
|
@rversteegen @jayrm Thanks a lot! I really appreciate the quick feedback! @rversteegen I am not actually a member of the team that published the paper and created the repository, but I believe FBC was included because it has suffient C source code and some defects in the version history include only changes to C source code (Python and PHP are also projects included for this reason). Many Auto Program Repair (APR) systems are made specifically for C and the purpose of the ManyBugs repository is to provide a system to test APR's. @jayrm Thanks for the info on the the FBC testing framework, this will come in handy when I am looking into the newer versions. I ultimately used the method suggested by @rversteegen of just printing the information before the actual assert. I added the code mentioned before the assert (changing the '<=' to '>' to report failures). This is what I used:
Thanks again! |
I'd forgotten about the fbcunit rewrite, so that was helpful for me too! |
Hello FBC devs,
I'm working on a research project that involves analyzing bugs and the specific input that generates them. I'm evaluating a previous version of FBC that is included in the ManyBugs repository (info for this can be found here, the specific version can be found in the scenario: fbc-bug-5251-5252). The bug in question has to do with the added functionality of octal support using '&'. There are two test files that I am particularly interested: tests/string/val.bas and tests/file/large_int.bas.
My question is this: In the large_int.bas test file, how can I determine which input causes one of the included assertions to fail?
When I run the executable created by compiling large_int.bas, it lists the assertions that fail like so,
I would like to determine which specific input i, like "&2000000", leads to an assert failure. I have included sections of the code that may be pertinent to answering this.
This is the section of code that uses assertions:
This section uses the macro:
This section generates the input file that is being opened and tested:
Thanks for your time!
The text was updated successfully, but these errors were encountered: