Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unity tests no longer show Serial output without verbose option #4276

Closed
torntrousers opened this issue May 18, 2022 · 19 comments
Closed

Unity tests no longer show Serial output without verbose option #4276

torntrousers opened this issue May 18, 2022 · 19 comments

Comments

@torntrousers
Copy link

After an updated from v5 to PlatformIO Core, version 6.0.1 Serial output in tests no longer shows unless the verbose option is used.

For recreate see: https://community.platformio.org/t/unity-test-serial-print-output-gone/27800/3?u=torntrousers

@ivankravets
Copy link
Member

Hi Anthony,

Thanks for the report. Indeed, this is a feature, not a bug. We don't print all junk from the testing frameworks and provide you by default only valuable information. The --verbose mode is the right solution. However, I agree, it prints now a ton of junk from the previous stages (build/upload).

So, what I propose. Allow passing multiple -v options:

  • -v - makes testing verbose
  • -vv - makes building and uploading verbose
  • -vvv - makes building and uploading much more verbose.

Your thoughts?

@torntrousers
Copy link
Author

having all the building etc type messages be moved from -v to -vv / -vvv would definately help.
it wasn't obvious to me how to set verbose mode when running test in vscode, something about project-tasks but it looked a bit complicated.

@torntrousers
Copy link
Author

Honestly though, it seems a bit of a pain, why the change?

@ivankravets
Copy link
Member

ivankravets commented May 19, 2022

You can override it via pio test -v.

@torntrousers
Copy link
Author

Mmm, not yet convinced that this is an improvement over the v5 behaviour. How do I set -v in vscode?

@ivankravets
Copy link
Member

Thanks! The configuration option in INI has been removed. However, the multilevel verbosity is available for pio test command. Also, new "Verbose Test" command has been added platformio/platformio-node-helpers@6931b21
Will be a part of the next release of PlatformIO IDE for VSCode.

Meanwhile, you can declare custom target with pio test -v. It will be available in IDE in the Tasks view.

We also plan to create a separate Test View. Please subscribe to this feature platformio/platformio-vscode-ide#3168

@torntrousers
Copy link
Author

None of those really seem great though when all i want is to see the test output so i can add debugging output while developing new code, like it used to. I suppose the separate Test View might help, but its not available yet.

@j4m3s
Copy link

j4m3s commented Jun 2, 2022

I've run into the same issue. We write log output to std::cout, and it used to be visible in v5.x. It's now lost/ swallowed in v6.

The pio test -v option in v6.0.2 doesn't help with the noise unfortunately, it still shows all of the tests collected and skipped, and doesn't colour the individual test results etc.

FYI the link included in the comment above doesn't seem to point to anything about the verbosity level?:

A new configuration option is added https://docs.platformio.org/en/latest/projectconf/section_env_test.html#test-verbosity-level

You can also override it via pio test -v.

@ivankravets
Copy link
Member

ivankravets commented Jun 3, 2022

Yes, PlatformIO normalizes test results from multiple testing frameworks to the unified output. Typically, people don't use Unit Testing to debug their applications. If you need custom results output, please override https://github.com/platformio/platformio-core/blob/develop/platformio/test/runners/unity.py#L260

See https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/custom/index.html

@torntrousers
Copy link
Author

Typically, people don't use Unit Testing to debug their applications.

For me, that is the way people use this - write some tests and then using debug output to get the tests passing or to work out whats wrong when some change has inadvertently broken a test.

Personally I prefer the v5 behaviour and wonder wouldn't something like a new "quiet" option have been a less disruptive change for those who don't want to see the output.

If you need custom results output, please override https://github.com/platformio/platformio-core/blob/develop/platformio/test/runners/unity.py#L260

See https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/custom/index.html

Can you explain that a bit more - how would I do that override?

@ivankravets
Copy link
Member

ivankravets commented Jun 3, 2022

Just follow this example https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/custom/examples/custom_unity_library.html

platformio.ini

[env:myenv]
test_framework = custom

test_custom_runner.py (put this file to the root of test folder)

import click

from platformio.public import UnityTestRunner

class CustomTestRunner(UnityTestRunner):

    def on_testing_line_output(self, line):
        click.echo(line, nl=False)
        super().on_testing_line_output(line)

@torntrousers
Copy link
Author

That gives:

If you don't see any output for the first 10 secs, please reset board (press reset button)

name 'click' is not defined
------------------------------------------------ arduino-esp32mbedtls:* [ERRORED] Took 16.00 seconds ------------------------------------------------

Do I need an import or something for click?

@ivankravets
Copy link
Member

Sorry, updated the code above. I didn't test it.

@j4m3s
Copy link

j4m3s commented Jun 12, 2022

Thank you for this @ivankravets. I added an if check so that only strings that aren't internally generated by Unity are printed:

import click

from platformio.public import UnityTestRunner


class CustomTestRunner(UnityTestRunner):

    def on_testing_line_output(self, line):
        if super().parse_test_case(line) is None:
            click.echo(line, nl=False)
        super().on_testing_line_output(line)

@TomD53
Copy link

TomD53 commented Jun 19, 2022

Thank you everyone! I agree that this change of behaviour is a little odd. I was also using unit tests to debug small pieces of code for my project, so I think it is a very valid use case. Thanks for providing a solution, works well for me.

@jhaand
Copy link

jhaand commented Jul 15, 2022

This solution looks fine to me.

@lemonkey
Copy link

lemonkey commented Aug 29, 2022

Doesn't seem to work for me with platformio 6.1.4 unless I use the TEST_MESSAGE macro (instead of say Serial.print()).

And this is using the "-v" flag when running pio test. I also have the custom runner as mentioned above..

TEST_MESSAGE macro is defined under .pio/libdeps/<your current project env>/Unity/src/unity.h.

#define TEST_MESSAGE(message) UnityMessage((message), __LINE__)

@ivankravets
Copy link
Member

Could you share your platformio.ini?

@lemonkey
Copy link

lemonkey commented Sep 2, 2022

It turns out I only have to use TEST_MESSAGE in the main test loop. Serial.print() statements are fine from within the unit tests themselves.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants