New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unity: Add macros for test skipping. #7448

Merged
merged 1 commit into from Jul 16, 2018

Conversation

Projects
None yet
7 participants
@davidsaada
Contributor

davidsaada commented Jul 9, 2018

Description

The TEST_SKIP and TEST_SKIP_UNLESS macros (w/ and w/o messages) allow the test to skip the test case execution from the point the macro was called (without failing the test).
This is useful in case the conditions (such as missing H/W, memory limitations etc.) don't allow the test to run properly.

Pull request type

[x] Fix
[ ] Refactor
[ ] New target
[ ] Feature
[ ] Breaking change

@0xc0170 0xc0170 requested review from ARMmbed/mbed-os-tools and removed request for ARMmbed/mbed-os-tools Jul 10, 2018

@davidsaada

This comment has been minimized.

Contributor

davidsaada commented Jul 10, 2018

@0xc0170 0xc0170 requested a review from ARMmbed/mbed-os-test Jul 10, 2018

@cmonr cmonr requested review from ARMmbed/mbed-os-hal, bulislaw, dannybenor, jamesbeyond and adbridge Jul 10, 2018

@adbridge

This comment has been minimized.

Contributor

adbridge commented Jul 10, 2018

OK this fix slightly subverts the way Utest/Unity are supposed to work. You may want to read the Utest readme, in particular:

Test Case Attributes

You can modify test case behavior by returning control_t modifiers:

    CaseNext: never repeats and immediately moves to next test case
    CaseNoRepeat: never repeats.
    CaseRepeatAll: repeats test case with setup and teardown handlers.
    CaseRepeatHandler: repeats test case without set and teardown handlers.
    CaseNoTimeout: immediately moves to next test case.
    CaseAwait: waits indefinitely for callback validation (use with caution).
    CaseTimeout(uint32_t ms): waits for callback validation for ms milliseconds, times out after that (fails with REASON_TIMEOUT).
    CaseRepeatAllOnTimeout(uint32_t ms): waits for callback validation for ms milliseconds, repeats test case with setup and teardown handlers on time out.
    CaseRepeatHandlerOnTimeout(uint32_t ms): waits for callback validation for ms milliseconds, repeats test case without setup and teardown handlers on time out.

From what I can make out your new Unity macro just does an immediate 'return' to skip executing the rest of a test case, however if the test case has been designed to return with one of the above custom control_t types then have you investigated what the knock on effect would be ? ie would it just break Utest?

*-------------------------------------------------------*/
#define UNITY_TEST_SKIP(line, message) { UnitySkipPrint( (message), (UNITY_LINE_TYPE)(line)); return; }
#define UNITY_TEST_SKIP_UNLESS(condition, line, message) if ((condition)) {} else UNITY_TEST_SKIP(line, message)

This comment has been minimized.

@adbridge

adbridge Jul 10, 2018

Contributor

Why isn't this just -
if (!(condition) {UNITY_TEST_SKIP((line), (message))} ?
Note also other brackets around parameters

This comment has been minimized.

@davidsaada

davidsaada Jul 11, 2018

Contributor

Right on both. Will change. Regarding the condition, it was just a copy/paste from the UNITY_TEST_ASSERT macro, but that's not an excuse.

This comment has been minimized.

@davidsaada

davidsaada Jul 11, 2018

Contributor

Pushed a fix now.

@davidsaada

This comment has been minimized.

Contributor

davidsaada commented Jul 11, 2018

From what I can make out your new Unity macro just does an immediate 'return' to skip executing the rest of a test case, however if the test case has been designed to return with one of the above custom control_t types then have you investigated what the knock on effect would be ? ie would it just break Utest?

The vast majority of the tests return nothing. Only tests that return the control_t type are in storage_abstraction, which is deprecated, and in the Utest tests themselves of course. Using the macro in these tests would produce an error/warning for not returning a return value. In all other tests (practically all tests), using this macro would simply end the function.
I can add macros with a return value, but again as I see it, it's not used anywhere in the code.
This macro is really intended as a convenient one liner for gracefully ending a test case, which is something which I believe is needed. So instead of having:

if (!normal_state) {
  printf(__LINE__)
  printf(message);
  return;
}

Just call TEST_SKIP_UNLESS(normal_state, message)
But anyway, will add a comment for clarity sake.

@OPpuolitaival

Looks good

@davidsaada davidsaada force-pushed the davidsaada:david_unity_test_skip branch from 8a0aadb to e728da1 Jul 11, 2018

Unity: Add macros for test skipping.
The TEST_SKIP and TEST_SKIP_UNLESS macros (w/ and w/o messages) allow the test
to skip the test case execution from the point the macro was called (without
failing the test).

@davidsaada davidsaada force-pushed the davidsaada:david_unity_test_skip branch from e728da1 to 2b7d836 Jul 11, 2018

@0xc0170

This comment has been minimized.

Member

0xc0170 commented Jul 13, 2018

/morph build

@0xc0170 0xc0170 requested review from ARMmbed/mbed-os-test and removed request for ARMmbed/mbed-os-hal Jul 13, 2018

@mbed-ci

This comment has been minimized.

mbed-ci commented Jul 13, 2018

Build : SUCCESS

Build number : 2601
Build artifacts/logs : http://mbed-os.s3-website-eu-west-1.amazonaws.com/?prefix=builds/7448/

Triggering tests

/morph test
/morph uvisor-test
/morph export-build
/morph mbed2-build

@mbed-ci

This comment has been minimized.

@mbed-ci

This comment has been minimized.

@fkjagodzinski

This might come in handy. 👍

@cmonr cmonr merged commit be215a3 into ARMmbed:master Jul 16, 2018

14 checks passed

AWS-CI uVisor Build & Test Success
Details
ci-morph-build build completed
Details
ci-morph-exporter build completed
Details
ci-morph-mbed2-build build completed
Details
ci-morph-test test completed
Details
continuous-integration/jenkins/pr-head This commit looks good
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
travis-ci/astyle Passed, 791 files
Details
travis-ci/docs Local docs testing has passed
Details
travis-ci/events Passed, runtime is 8860 cycles
Details
travis-ci/gitattributestest Local gitattributestest testing has passed
Details
travis-ci/licence_check Local licence_check testing has passed
Details
travis-ci/littlefs Passed, code size is 9960B
Details
travis-ci/tools-py2.7 Local tools-py2.7 testing has passed
Details

@0xc0170 0xc0170 removed the needs: review label Jul 16, 2018

@davidsaada davidsaada deleted the davidsaada:david_unity_test_skip branch Aug 1, 2018

pan- pushed a commit to pan-/mbed that referenced this pull request Aug 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment