Script to auto-check whether benchmark files compile using GCC

[tautschnig]

As previously stated on the mailing list, I have created a script to check whether benchmark files can at all be compiled using GCC, with crucial warnings enabled. That is, with comments:

Turn on warnings and make them fatal

-Wall -Werror

The rules say "GNU C or ANSI C", but a few benchmarks use C99-for-scoping

-std=c99

The following shouldn't affect the semantics

-Wno-unused-label -Wno-unused-function
-Wno-unused-but-set-variable -Wno-unused-variable -Wno-unused-value

These are a bit weird, but probably ok

-Wno-unknown-pragmas -Wno-attributes -Wno-parentheses

We've had discussions about these:

-Wno-maybe-uninitialized -Wno-uninitialized

A case-by-case question to see whether these are actually well-defined

-Wno-pointer-to-int-cast -Wno-int-to-pointer-cast -Wno-pointer-sign

The following should be fixed in the benchmarks triggering these

-Wno-return-local-addr -Wno-return-type
-Wno-main -Wno-sequence-point

The remaining warnings/errors can't be silenced once -Werror is in effect, so I
arrived at a huge blacklist, which is included in the file. Over time, these should be cleaned up.

Best,
Michael

[PhilippWendler]

I think this should be integrated.

What is the idea of the blacklist file that is generated and deleted on the next run?

Also, what are the reasons for the auto-blacklisted warnings?
I think for example at least -Werror=implicit-function-declaration is something we can and should fix.

[tautschnig]

The blacklist file is/was helpful to easily add all the files failing the test to the for-loop that skips over known-bad ones and is included in the script. The reasons for auto-blacklisting were simply such that I did not manually have to ack each and every reported error.

In the near term, all the auto-blacklisting should go away, and indeed the files should be fixed instead. See #3 for some first progress towards this goal.

Best,
Michael

[PhilippWendler]

I have added a few changes to the script in https://github.com/PhilippWendler/sv-benchmarks/tree/checking, maybe you want to include them in this pull request?

[tautschnig]

Thanks a lot, added those improvements!

[mutilin]

The script produces warnings for the file ldv-memsafety/memleaks_test17_2_true-valid-memsafety.i.
For example
header.h:196:30: error: implicit declaration of function ‘typeof’ [-Werror=implicit-function-declaration]
but typeof is in GNU C, see https://gcc.gnu.org/onlinedocs/gcc/Typeof.html

[delcypher]

@mutilin
What version of GCC are you using? The -std=c99 flag might need to be changed to -std=gnu99 if the GNU flavour of C99 is required.

[mutilin]

@delcypher
I use gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, but the options are defined in the check script which is discussed here.
It seems that -std=c99 option should be changed to -std=gnu99 for ldv- benchmarks