Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with test explorer showing parameterized tests with ceedling/unity TEST_RANGE and TEST_MATRIX #135

Open
lapo4719 opened this issue Apr 12, 2024 · 0 comments

Comments

@lapo4719
Copy link

I installed ceedling 0.31.0 and upgraded the unity version to 2.6.0 to support the TEST_CASE, TEST_RANGE, and TEST_MATRIX parametric testing features. ThrowTheSwitch/Ceedling#241 (comment)

Not sure if parametric testing is very popular or not. I'd like to use all these w/ the ceedling test explorer extension.

Issues:

  1. When using the test explorer having tests using the TEST_RANGE macro causes the explorer to freeze and not allow the panel to run tests or update.
    image
    The test test_parametric_arguments_inclusive_range() is not showing in panel. The panel is frozen in this picture, and the spinner is constantly reloading.

  2. If I comment out TEST_RANGE tests and enable TEST_MATRIX() tests, the explorer will show the tests and allow the tests to run, but the Test Status will be gray and never update to pass or fail. Also it will only show as a single test, not a list of the matrix tests.
    image

In ceedling, these tests do all pass via the command line when all enabled.

Expected:

  • At least be able to run all tests with the test explorer even with test using the TEST_RANGE() macro
  • At least be able to see pass/fail status of the tests w/ TEST_RANGE and TEST_MATRIX tests. All or nothing pass/fail.
  • showing all the sub-parametric tests would be a plus and beneficial. But secondary.

To Reproduce:

My project.yml

# Notes:
# Sample project C code is not presently written to produce a release artifact.
# As such, release build options are disabled.
# This sample, therefore, only demonstrates running a collection of unit tests.

:project:
  :use_exceptions: FALSE
  :use_test_preprocessor: TRUE
  :use_preprocessor_directives: TRUE
  :use_auxiliary_dependencies: TRUE
  :build_root: build
#  :release_build: TRUE
  :test_file_prefix: test_
  :which_ceedling: vendor/ceedling
  :ceedling_version: 0.31.1
  :default_tasks:
    - test:all

:unity:
  :includes:
    - parametric_macros.h
  :use_param_tests: true

:test_runner:
  :use_param_tests: true

:test_build:
 :use_assembly: TRUE

:release_build:
 :output: MyApp.out
 :use_assembly: FALSE

:environment:

:extension:
  :executable: .out

:paths:
  :test:
    - +:test/unit_tests/**
    - -:test/support
  :source:
    - src/**
  :support:
    - test/support
  :libraries: []

:defines:
  # in order to add common defines:
  #  1) remove the trailing [] from the :common: section
  #  2) add entries to the :common: section (e.g. :test: has TEST defined)
  :common: &common_defines []
  :test:
    - *common_defines
    - CEEDLING_UNIT_TEST
    - UNITY_SUPPORT_TEST_CASES
  :test_preprocess:
    - *common_defines
    - CEEDLING_UNIT_TEST
    - UNITY_SUPPORT_TEST_CASES

:cmock:
  :mock_prefix: mock_
  :when_no_prototypes: :warn
  :enforce_strict_ordering: TRUE
  :plugins:
    - :ignore
    - :callback
  :treat_as:
    uint8:    HEX8
    uint16:   HEX16
    uint32:   UINT32
    int8:     INT8
    bool:     UINT8

# Add -gcov to the plugins list to make sure of the gcov plugin is found
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
  :utilities:
    - gcovr
  :reports:
    - HtmlDetailed
  :gcovr:
    :html_medium_threshold: 75
    :html_high_threshold: 90

#:tools:
# Ceedling defaults to using gcc for compiling, linking, etc.
# As [:tools] is blank, gcc will be used (so long as it's in your system path)
# See documentation to configure a given toolchain for use

# LIBRARIES
# These libraries are automatically injected into the build process. Those specified as
# common will be used in all types of builds. Otherwise, libraries can be injected in just
# tests or releases. These options are MERGED with the options in supplemental yaml files.
:libraries:
  :placement: :end
  :flag: "-l${1}"
  :path_flag: "-L ${1}"
  :system: []    # for example, you might list 'm' to grab the math library
  :test: []
  :release: []

# :module_generator:
#   :project_root: ./
#   :source_root: src/
#   :test_root: test/unit_tests/
#   :boilerplate_files:
#     :src: '<template_folder>\src_boilerplate.c'
#     :inc: '<template_folder>\inc_boilerplate.h'
#     :tst: '<template_folder>\tst_boilerplate.c'

# :xml_tests_report:
#   :artifact_filename: xml_report.xml
:plugins:
  :load_paths:
    - vendor/ceedling/plugins
  :enabled:
    - stdout_pretty_tests_report
    - raw_output_report
    - xml_tests_report
    # - gcov
    # - module_generator

My test file:

// /****************************************
// 	File Name: Tests
// *************************************** */
#include <stdlib.h>
#include "unity.h"
#include "parametric_macros.h"
#include "stdbool.h"


void setUp(void) {

}

void tearDown(void) {

}


void test_hello_world(void)
{
    bool myvar = true;
    TEST_ASSERT_EQUAL(myvar, true);
}

TEST_CASE(1)
TEST_CASE(3)
void test_parametric_arguments(uint8_t my_param)
{
    printf("testing parametric arguments: %d", my_param);
    TEST_ASSERT_TRUE_MESSAGE(my_param, "I print if assert is false");
}

TEST_RANGE([0, 2, 1])
void test_parametric_arguments_inclusive_range(uint8_t my_param)
{
    printf("testing parametric arguments: %d", my_param);

	// make a little debug message:
	char *debug_msg = (char*)malloc(128 * sizeof(char));
	sprintf(debug_msg, "my_param: %d", my_param);

	// assert:
    TEST_ASSERT_TRUE_MESSAGE(my_param < 5, debug_msg);
}

TEST_RANGE(<0,1,1>)
void test_parametric_arguments_exclusive_range(uint8_t my_param)
{
	// make a little debug message:
	char *debug_msg = (char*)malloc(128 * sizeof(char));
	sprintf(debug_msg, "my_param: %d", my_param);

    TEST_ASSERT_TRUE_MESSAGE(my_param < 1, debug_msg);
}

TEST_MATRIX([3, 4, 7])
void test_parametric_arguments_with_test_matrix(uint8_t my_param)
{
	// make a little debug message:
	char *debug_msg = (char*)malloc(128 * sizeof(char));
	sprintf(debug_msg, "my_param: %d", my_param);

	printf("I AM HERE AND RUNNING");

    TEST_ASSERT_TRUE_MESSAGE(my_param < 8, debug_msg);
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant