Some tests fail when building with -UTEST

[madebr]

UpdateCTestConfiguration  from :/home/maarten/programming/cake/src/cmake-build-debug/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/maarten/programming/cake/src/cmake-build-debug/DartConfiguration.tcl
Test project /home/maarten/programming/cake/src/cmake-build-debug
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start   1: cake_cake

1: Test command: /home/maarten/programming/cake/src/cmake-build-debug/cake "-nullable=disable" "-ownership=enable" "-Wstyle" "-fanalyzer" "-Wno-unused-parameter" "-Wno-unused-variable" "-sarif" "token.c" "hash.c" "hashmap.c" "console.c" "tokenizer.c" "osstream.c" "fs.c" "options.c" "expressions.c" "pre_expressions.c" "type.c" "flow_object.c" "parser.c" "visit.c" "flow_visit.c" "error.c" "format_visit.c" "console.h" "tokenizer.h" "parser.h" "error.h" "fs.h" "hash.h" "flow_object.h" "hashmap.h" "osstream.h" "options.h" "token.h" "type.h" "pre_expressions.h" "expressions.h" "visit.h" "format_visit.h"
1: Working Directory: /home/maarten/programming/cake/src
1: Test timeout computed to be: 10000000
1: cake: /home/maarten/programming/cake/src/options.c:148: get_warning_bit_mask: Assertion `wname[0] == '-'' failed.
1/6 Test   #1: cake_cake ........................Subprocess aborted***Exception:   0.11 sec
test 49
    Start  49: test_14100

49: Test command: /home/maarten/programming/cake/src/cmake-build-debug/cake "/home/maarten/programming/cake/src/../tests/unit-tests/test_14100.c"
49: Working Directory: /home/maarten/programming/cake/src/cmake-build-debug
49: Test timeout computed to be: 10000000
49: Cake 0.9.10
49: /home/maarten/programming/cake/tests/unit-tests/test_14100.c
49: /home/maarten/programming/cake/tests/unit-tests/test_14100.c:6:31: error: pragma check failed [E65]
49:  6 |#pragma cake diagnostic check "-Wmaybe-uninitialized"
49: #pragma cake diagnostic check "-Wanalyzer-maybe-uninitialized"
49:     return 0;
49:    |                              ~~~~~~~~~~~~~~~~~~~~~~~                                                                              
49: 
49:  1 files in 0.00 seconds
49:  1 errors 0 warnings 0 notes 
2/6 Test  #49: test_14100 .......................***Failed    0.00 sec
test 59
    Start  59: test_15000

59: Test command: /home/maarten/programming/cake/src/cmake-build-debug/cake "/home/maarten/programming/cake/src/../tests/unit-tests/test_15000.c"
59: Working Directory: /home/maarten/programming/cake/src/cmake-build-debug
59: Test timeout computed to be: 10000000
59: Cake 0.9.10
59: /home/maarten/programming/cake/tests/unit-tests/test_15000.c
59: /home/maarten/programming/cake/tests/unit-tests/test_15000.c:2:1: error: _Static_assert failed "E type"
59:  [E106]
59:  2 |static_assert (_Generic (e, long long: 1, default: 0) == 1, "E type");
59:    |~~~~~~~~~~~~~                                                          
59: 
59:  1 files in 0.00 seconds
59:  1 errors 0 warnings 0 notes 
3/6 Test  #59: test_15000 .......................***Failed    0.00 sec
test 63
    Start  63: test_15400

63: Test command: /home/maarten/programming/cake/src/cmake-build-debug/cake "/home/maarten/programming/cake/src/../tests/unit-tests/test_15400.c"
63: Working Directory: /home/maarten/programming/cake/src/cmake-build-debug
63: Test timeout computed to be: 10000000
63: Cake 0.9.10
63: /home/maarten/programming/cake/tests/unit-tests/test_15400.c
63: int *unsigned short *wchar_t/home/maarten/programming/cake/tests/unit-tests/test_15400.c:2:15: error:  incompatible types at argument  [E95]
63:  2 |wchar_t s[] = L"abcd";
63:    |              ~~~~~~~  
63: 
63:  1 files in 0.00 seconds
63:  1 errors 0 warnings 0 notes 
4/6 Test  #63: test_15400 .......................***Failed    0.00 sec
test 115
    Start 115: test_20000

115: Test command: /home/maarten/programming/cake/src/cmake-build-debug/cake "/home/maarten/programming/cake/src/../tests/unit-tests/test_20000.c"
115: Working Directory: /home/maarten/programming/cake/src/cmake-build-debug
115: Test timeout computed to be: 10000000
115: Cake 0.9.10
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c:15:31: error: pragma check failed [E65]
115:  15 |#pragma cake diagnostic check "-E128" 
115: 
115:     |                              ~~~~~~~   
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c:25:8: warning: pointer is always null [-Wnonnull]
115:  25 |   if (g)    {
115:     |       ~       
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c:10:12: warning: object 'g' can be uninitialized  [-Wanalyzer-maybe-uninitialized]
115:  10 |struct X * g;
115:     |           ~  
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c:26:6: warning: object lifetime ended [-Wlifetime-ended]
115:  26 |     g->i = 1;
115:     |     ~~~~      
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c:26:6: warning: The object 'p' may have been deleted or its lifetime have ended. [-Wlifetime-ended]
115:  26 |     g->i = 1;
115:     |     ~~~~      
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c:20:27: warning: ownership of 'p' not moved before the end of lifetime [-Wmissing-destructor]
115:  20 |   struct X * _Opt _Owner p = calloc(1, sizeof * p);
115:     |                          ~                          
115: /home/maarten/programming/cake/tests/unit-tests/test_20000.c:28:1: note: end of 'p' lifetime
115:  28 |}
115:     |~ 
115: 
115:  1 files in 0.00 seconds
115:  1 errors 5 warnings 0 notes 
5/6 Test #115: test_20000 .......................***Failed    0.00 sec
test 127
    Start 127: test_21100

127: Test command: /home/maarten/programming/cake/src/cmake-build-debug/cake "/home/maarten/programming/cake/src/../tests/unit-tests/test_21100.c"
127: Working Directory: /home/maarten/programming/cake/src/cmake-build-debug
127: Test timeout computed to be: 10000000
127: Cake 0.9.10
127: /home/maarten/programming/cake/tests/unit-tests/test_21100.c
127: /home/maarten/programming/cake/tests/unit-tests/test_21100.c:11:3: error: static_state failed [E108]
127:  11 |  static_state(p, "null moved"); //shows "null" - it should be "null or moved"
127:     |  ~~~~~~~~~~~~                                                                 
127: expected :"null moved"
127: current  :"null"
127: 
127:  1 files in 0.00 seconds
127:  1 errors 0 warnings 0 notes 
6/6 Test #127: test_21100 .......................***Failed    0.00 sec

0% tests passed, 6 tests failed out of 6

Total Test time (real) =   0.13 sec

The following tests FAILED:
	  1 - cake_cake (Subprocess aborted)
	 49 - test_14100 (Failed)
	 59 - test_15000 (Failed)
	 63 - test_15400 (Failed)
	115 - test_20000 (Failed)
	127 - test_21100 (Failed)

(I added CMake project generation to build.c. Let me know if you are interested. ci)

[thradams]

I am a little lost about how to produce these diagnostics.
What I generally do is :

src>       cake ../tests/unit-tests/*.c -test-mode

215 files in 0.27 seconds
 9 errors 16 warnings 0 notes
 10 tests of 215 failed

This run cake in all files in "test mode". 10 tests are failing (under development)

There is also -DTEST that compile cake in test mode runs 63 test functions. Most of it is preprocessor test.

cl -DTEST build.c && build

0 tests failed, 63 tests passed
0 expected to fail - waiting fixes

[thradams]

What is missing is running cake ../tests/unit-tests/*.c -test-mode inside build.c.
There is also other samples that could be used in automatic test.

[thradams]

What -test-mode does is to pass test if we have 0 warnings 0 erros.

Sample:

#pragma safety enable

struct X {
    int i;
};

void f(struct X* _Opt p)
{
    int i = p ? p->i : 0; //no warning
    
    int i3 = p->i;
    #pragma cake diagnostic check "-Wanalyzer-null-dereference"

    int i2 = p ? 0 : p->i; //warning
#pragma cake diagnostic check "-Wanalyzer-null-dereference"
}

pragma is used to dismiss the error/warning and check we have that error/warning.

#pragma cake diagnostic check "error or warning"

[madebr]

I misunderstood the -DTEST option. I now understand that it can be used to build a unittest executable (let's call it cake-unittest).

So running a bare "cake-unittest" succeeds, but "cake" fails to parse certain sources (including SDL).
https://github.com/madebr/cake/actions/runs/9943271740/job/27466699049

[thradams]

Did you add test files on CI?
What I can do is separate in a folder the testes that are currently failing.
I would like to integrate this.

tests\unit-tests                 <-- should work 100%, checked at CI
tests\unit-tests\failing      <-- failing tests (TODO)

The directory

\tests\en-cpp-reference-c
\tests\sqlite

has a lot of source but not prepared to run in test mode.

[madebr]

I do a glob in tests/unit-tests and assumed they should all succeed.
Separating them in ok/failing would be usable.

https://github.com/madebr/cake/actions/runs/9945512803
As a heuristic, if the directory ends on fail, it will expect the test to fail.

ctest expects the return code to be non-zero for a failing test.

[thradams]

I changed the meaning of -DTEST.
Now DTEST is the same exe.
The difference is function tests are included and they are called before any other execution.
The object was to avoid to create two Exes. Now the same exe executes the tests and also compiler cake own source.

[thradams]

CI is not working with cake compiled with x64. But in my machine , with sanitizer etc..it works normally.
then I changed to x86 and it worked.

[madebr]

something you must be aware is that cakeconfig.h is always the cakeconfig.h that is at same path of executable.
I want to change that to search first the same path of each file.

cake folder
  | cake.exe
  | cakeconfig.h
  
mycode
    | cakeconfig.h 
   |   source.c

[thradams]

I thought the return code was fixed. At main()

    if (report.test_mode)
    {
        return report.test_failed ? EXIT_FAILURE : EXIT_SUCCESS;
    }
    return report.error_count > 0 ? EXIT_FAILURE : EXIT_SUCCESS;
}

I also found some weird behavior with pointer to "report". Maybe I need to refactory that.
There this "global report" and "local report" for each file.

Windows x64 is failing but not on my machine. This is also weird.

About the includes, we list on GCC. But maybe we need to add some flag to x64 before?
See -autoconfig / generate_config_file

Maybe this command line needs to be changed x86 x64

char* command = "echo | gcc -v -E - 2>&1";

[thradams]

by the way "en-cpp-reference-c" directory is not prepared for test-mode..
They are just files.. the ones that should compile without errors/warning will pass the test.

Also some samples are linux only.

[madebr]

About the includes, we list on GCC. But maybe we need to add some flag to x64 before? See -autoconfig / generate_config_file

Maybe this command line needs to be changed x86 x64

char* command = "echo | gcc -v -E - 2>&1";

I know some distros have separate include directories for different architectures (i686 vs x86_64).
Mine doesn't, but I think Ubuntu does. So the command should be CC and CFLAGS dependent.

by the way "en-cpp-reference-c" directory is not prepared for test-mode.. They are just files.. the ones that should compile without errors/warning will pass the test.

Also some samples are linux only.

Good to know. I think not-yet-implemented functionality should move to a failing-like folder as well.
In the same spirit, the *nix-only samples should move to a nix-like folder.

[madebr]

For some reason, running cake on the cake sources times out.
This only happens on Windows.

[madebr]

I ported the cakeconfig.h generation to cmake, and I see a little difference between 32-bit vs 64-bit.

32-bit:

 /*
 * This file was generated reading the ouput of
 *
 * echo | /usr/bin/gcc -m32 -v -E - 2>&1
 *
 */

#pragma dir  "/usr/lib/gcc/x86_64-linux-gnu/11/include"
#pragma dir  "/usr/local/include"
#pragma dir  "/usr/include"

64-bit:

/*
 * This file was generated reading the ouput of
 *
 * echo | /usr/bin/gcc -m64 -v -E - 2>&1
 *
 */

#pragma dir  "/usr/lib/gcc/x86_64-linux-gnu/11/include"
#pragma dir  "/usr/local/include"
#pragma dir  "/usr/include/x86_64-linux-gnu"
#pragma dir  "/usr/include"

[thradams]

All tests in /tests/unit-test and /tests/en-cpp-reference-c/ are passing.
The ones are not passing are inside failing. They are not necessarily error but need some work to move them.

Preprocessor errors where not reported to the pass test.. now they are.