Question regarding tests

[dibyendumajumdar]

Hi, I would like to help with the testing of the C to MIR translator. One of the approaches I used for dmrC was to source tests from various places and adapt them. The issue of course is sometimes the tests have their own licences. Do you care about that?

Anyway please let me know if this is of interest.

[vnmakarov]

One of the approaches I used for dmrC was to source tests from various places and adapt them. The issue of course is sometimes the tests have their own licences. Do you care about that?

Thank you for the proposal. Yes, I am interesting in C tests and I believe your experience will be useful to find such tests.

Currently, I am using some tests from GCC testsuite. They are small and currently very useful to find and debug C2MIR translator. Bigger tests are not useful for me right now as some regular C11 features are not implemented yet and there are still a lot of bugs in the translator/MIR-interpreter/MIR-generator.

When I succeed with the small tests, I'll need bigger ones (probably in 1-2 months). So I am open to any tests proposals.

The license issue can be tricky one.

It would be nice to have also tests which check C11 conformance and contains wrong C code too. Unfortunately, such conformant tests (like perennial/plumhall) are not free.

[dibyendumajumdar]

Perhaps I should do this in a separate repo so that there is no pollution re the license.

[dibyendumajumdar]

Hi, I started this at:

https://github.com/dibyendumajumdar/mir-testing

It is on a branch called tests.

I copied across the dmrC tests which are in turn sourced from various places.
Some are small tests and some are bigger.

It should be easy for you to run the tests if you would like to: instructions are in README.

Regards

[dibyendumajumdar]

I added more tests from other sources. A key criteria is that the test must be runnable. So every test has to have a main function. The test success is determined by using a combination of return code from the test and any stdout output.

[dibyendumajumdar]

Here are some stats:

Total tests run 447
Successful tests 230

Not bad

[vnmakarov]

Thank you. I will look at this later when I resume my work on the project.

[dibyendumajumdar]

@vnmakarov I merged the latest updates. The current results are:

Total tests run 447
Successful tests 337

Pretty good improvement!
Thanks

[vnmakarov]

Total tests run 447
Successful tests 337

Pretty good improvement!

Thanks for the tests. I worked on them and they are pretty useful.
Although I found some troubles with them:

• some tests (from TESTED) check errors and are supposed to be failed
• some tests expect wrong output, gcc and clang fail on them too
• some tests expect specific undefined C behaviour
• some tests check GCC extensions (binary constants, case ranges, extended ternary operators etc)

But in general they are useful and I will add them to repository after their analysis.

Meanwhile I've just added about 600 GCC tests I am using myself. They can be called by make c2mir-full-test.

[dibyendumajumdar]

Although I found some troubles with them:

* some tests (from  TESTED) check errors and are supposed to be failed

* some  tests expect wrong output, gcc and clang fail on them too

* some tests expect specific undefined C behaviour

* some tests check GCC extensions (binary constants, case ranges, extended ternary operators etc)

Yes perhaps as I have not checked all of the failures, and also many tests I just picked from other projects. The ones I use for dmrC are a smaller subset; those are included as well of course.

But in general they are useful and I will add them to repository after their analysis.

Meanwhile I've just added about 600 GCC tests I am using myself. They can be called by make c2mir-full-test.

Cool. I hope this won't cause a license issue.
Are all these tests passing?

I am very impressed by your project and am eagerly waiting to start using it.

[dibyendumajumdar]

So looks like passes those 655 gcc tests! Wonderful!

[vnmakarov]

So looks like passes those 655 gcc tests! Wonderful!

They are only tests which are passing but there are a few tests from gcc I use which are not passing. I will add them when they start passing. It is easier this way not to introduce new regressions.

[vnmakarov]

Cool. I hope this won't cause a license issue.

I am not a lawyer and can not give advices to other people but for me it is obvious that it does not violate the GPL license (and its spirit).

[dibyendumajumdar]

Hi, I have deleted the mir-testing repo because I had trouble merging changes from your repo. In any case looks like you are adding the tests to mir so no point having the testing repo.

[vnmakarov]

Hi, I have deleted the mir-testing repo because I had trouble merging changes from your repo. In any case looks like you are adding the tests to mir so no point having the testing repo.

Sorry for creating a conflict. I really appreciate your work and collecting a lot of C tests in mir-test repo. Mine tests from gcc-torture were not enough.

I found that Andrew Chambers and especially LACC tests have much better coverage of C standard. LACC tests revealed a lot of problems with LLVM to MIR translator. Basically, I did not implement composite data operators yet (insert and extract value) which are generated for small structures for some ABIs. Implementing this functionality will take some time. So currently I switched off LLVM2MIR translator testing in Makefile.
As for C2MIR compiler I've managed to fix all LACC tests. The next big step will be achieving a bootstrap for C2MIR.

[dibyendumajumdar]

Hi, I have deleted the mir-testing repo because I had trouble merging changes from your repo. In any case looks like you are adding the tests to mir so no point having the testing repo.

Sorry for creating a conflict. I really appreciate your work and collecting a lot of C tests in mir-test repo. Mine tests from gcc-torture were not enough.

No worries I am glad that it was helpful.

I found that Andrew Chambers and especially LACC tests have much better coverage of C standard. LACC tests revealed a lot of problems with LLVM to MIR translator. Basically, I did not implement composite data operators yet (insert and extract value) which are generated for small structures for some ABIs. Implementing this functionality will take some time. So currently I switched off LLVM2MIR translator testing in Makefile.
As for C2MIR compiler I've managed to fix all LACC tests. The next big step will be achieving a bootstrap for C2MIR.

That's great news.