Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Test output" does not show the name of the func generating the output #71

Closed
hmijail opened this issue Apr 29, 2016 · 10 comments
Closed

Comments

@hmijail
Copy link
Contributor

hmijail commented Apr 29, 2016

No description provided.

@mvandervoord
Copy link
Member

Can you provide a bit more detail. I'm not sure what circumstances you're looking at here.

@hmijail
Copy link
Contributor Author

hmijail commented Apr 29, 2016

What kind of detail? This is the output I see:

...

TEST OUTPUT

[test_TBuffer.c]

  • "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
  • "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
  • ""

FAILED TEST SUMMARY

[test_TBuffer.c]
Test: test_TBuf_Create_OK
At line (83): "Expected 0x00000004 Was 0x00000040. Function tmalloc Argument size. Function called with unexpected argument value."


OVERALL TEST SUMMARY

TESTED: 3
PASSED: 2
FAILED: 1
IGNORED: 0


BUILD FAILURE SUMMARY

Unit test failures.

Note that the filename test_TBuffer.c is given under TEST OUTPUT, but not the test function names.
Meanwhile, for the FAILED TEST SUMMARY, both test file and test function names are given, which would be the correct behavior I guess.

On 29 Apr 2016, at 15:19, Mark VanderVoord notifications@github.com wrote:

Can you provide a bit more detail. I'm not sure what circumstances you're looking at here.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

@hmijail
Copy link
Contributor Author

hmijail commented Apr 29, 2016

Sorry, looks like the answer I sent through mail got broken? I'll repeat here.

What kind of detail? This is the output I see:

...
-----------
TEST OUTPUT
-----------
[test_TBuffer.c]
  - "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
  - "0/F/TBuffer.c:24/Failed to malloc TBuf: No Memory"
  - ""

-------------------
FAILED TEST SUMMARY
-------------------
[test_TBuffer.c]
  Test: test_TBuf_Create_OK
  At line (83): "Expected 0x00000004 Was 0x00000040. Function tmalloc Argument size. Function called with unexpected argument value."

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  3
PASSED:  2
FAILED:  1
IGNORED: 0

---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.

Note that the filename test_TBuffer.c is given under TEST OUTPUT, but not the test function names.
Meanwhile, for the FAILED TEST SUMMARY, both test file and test function names are given, which would be the correct behavior I guess.

@mvandervoord
Copy link
Member

Ah. Ceedling doesn't make any attempt to correlate data that you are manually dumping to stdout with the data that it is looking for. It simply captures it and displays it so that you can use it for whatever you like. This is intentional. In order to correlate it to a test, Ceedling would have to keep that data around, then apply it to the test when it is completed. In addition to this being a lot of overhead for a feature that is seldom used, it could easily be wrong if the output was dumped from setUp, tearDown, the RUN_TEST macro, etc. In any of those cases, you would see the extra output incorrectly assigned to the next completed test.

The output summary is really just there for the convenience of the test writers. The majority of the time, it is completely empty.

@hmijail
Copy link
Contributor Author

hmijail commented May 7, 2016

OK, I think I understand.

My assumption was that, just as each stderr output is connected to a test function, the stdout would also be connected, and that the problem of not seeing which test function printed something was just a problem in the test result printer.

From what you say, I understand now that in fact the stdout is NOT captured specifically for each test function, and that doing so would be too much overhead. Right? That's a pity, I assumed it would be the same procedure already being done for stderr.

Just to be clear, I can kinda work-around this myself in my tests by adding the printf("%s",__func__); thing as the first line of each test, just to know that the coming text was output by such-and-such test. It's just strange boilerplate. Though I understand that usually (ideally) unit tests don't even use stdout.

So, as a way to reduce the boilerplate in a generic way: would it be possible to have the name of the test be passed to the setUp() function? So I could do something like

void setUp(char * test_func){
    printf("%s",test_func);
}

@hmijail
Copy link
Contributor Author

hmijail commented May 7, 2016

(... or maybe this isn't even about stderr and stdout and then I'm totally off??)

@mvandervoord
Copy link
Member

Actually all the output from stdout is being captured. The part that makes it more challenging is that the name of the test is announced with the results, so it happens at the END of the test. (This is necessary for quite a few reasons, all of which are odd little details. Long ago, we used to print the test name at the beginning and then the pass/fail result at the end of the test... everything else would get inserted in between. this didn't really work very well because people's results parsers would get all messed up by anything they output in the middle).

ANYWAY, it happens at the end now... which means that anything you output would have to get stored away separately, then associated with that test once it was announced. This would be correct most of the time... but if the user chose to write something from setup or teardown or anything like that, it would still get associated with the nearest test, which may not be what they were looking for.

Maybe we add a TEST_INFO_MESSAGE macro to output an info line with whatever you want?

@hmijail
Copy link
Contributor Author

hmijail commented May 7, 2016

How would that TEST_INFO_MESSAGE macro be used?

@mvandervoord
Copy link
Member

mvandervoord commented May 7, 2016

It would output the text you pass it, but prefix it with the usual info... maybe something like filename:line:function:INFO:message? I'm just thinking aloud here.

(The advantage being that you could force this sort of output to appear in the appropriate place in Ceedling's Pretty Print)

@hmijail
Copy link
Contributor Author

hmijail commented May 7, 2016

If it is something that could be used in setUp(), then OK. But things like filename, line, function, can already be done with standard compiler-provided macros, so... I don't know.

In general, IMO anything that maximizes info in case of debugging and minimizes boilerplate is good :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants