meta-shift

Shift-left testing for the Yocto project

About

Shift-left testing is an approach to software testing in which testing is performed earlier and often in the software development lifecycle.

The benefits of the shift-left testing approach are:

• Improved software quality since defects are detected in earlier stages
• Cost effective as early detected defects are cheaper to fix
• Increased efficiency in the software development process
• Reduced time to market since the QA stage does not take much time

The meta-shift layer is a set of recipes and classes for the Bitbake build system, which allows developers to test or examine their software modules in the host build environment. By enabling meta-shift, developers are able to easily use various tasks for their recipes via the bitbake command, such as:

• Lines of code
• Cache hit ratio
• Static analysis
• Comments
• Cyclomatic complexity
• Duplication
• Unit testing
• Code coverage
• Mutation testing

Features

The main purpose of meta-shift is to provide the shift-left testing tools to the Yocto build environment satisfying the following needs.

• Easy to configure
• Host-based testing
• Supports major build systems (cmake, qmake, autotools)
• Supports SDK
• Supports various Yocto releases and BSPs
• Jenkins integration

Quick start

Please visit a build-sample repository to find more information about how to configure and use features provided by the meta-shift layer.

Usage

This is a meta layer for the Yocto project. Please find more information about the meta layer if you are not familiar with.

Dependencies

Mandatory layers

• meta-oe (meta-openembedded)
• meta-pyton (meta-openembedded)

Optional layers

• meta-qt5: To support QT5 based recipes
• meta-qt6: To support QT6 based recipes
• meta-clang: To use clang-tidy and the mutation testing

Installation

Clone this repository and add the layer to your bblayer.conf

$ git clone -b dunfell https://github.com/shift-left-test/meta-shift.git
$ source oe-build-init-env
$ bitbake-layers add-layer ../meta-shift

Supported tasks

List of tasks via the bitbake command

• do_checkcache
• do_checkcacheall
• do_checkcode
• do_checkcodeall
• do_checkrecipe
• do_checkrecipeall
• do_checktest
• do_checktestall
• do_coverage
• do_coverageall
• do_report
• do_reportall
• do_test
• do_testall

Supported bitbake tools

List of bitbake tool commands

• devtool cache
• devtool show
• bitbake-layers inspect
• bitbake-layers status
• bitbake-layers test-layers
• bitbake-layers test-recipes
• recipetool check
• recipetool inspect

Configuration

These options can be used by adding to conf/local.conf.

• SHIFT_CHECKCODE_EXCLUDES: Paths to exclude from the static analysis
• SHIFT_CHECKCODE_TOOLS: Indicates which static analysis tools to use (cppcheck, cpplint, and clang-tidy)
• SHIFT_CHECKRECIPE_SUPPRESS_RULES: Exclude rules from bitbake script analysis (A list of rules can be found at https://github.com/priv-kweihmann/oelint-adv)
• SHIFT_CHECKTEST_ENABLED: Enables mutation testing (disabled by default)
• SHIFT_CHECKTEST_EXCLUDES: Excludes paths from mutation testing
• SHIFT_CHECKTEST_EXTENSIONS: Extensions of source files to be mutated
• SHIFT_CHECKTEST_GENERATOR: Set the mutation generator (random, uniform, or weighted)
• SHIFT_CHECKTEST_LIMIT: Set the maximum limit of mutants
• SHIFT_CHECKTEST_MAX_TIMEOUT: Set the maximum timeout duration of each test
• SHIFT_CHECKTEST_SCOPE: Indicate which source code to mutate (all or commit)
• SHIFT_CHECKTEST_SEED: Random seed for the mutation generator
• SHIFT_CHECKTEST_VERBOSE: Silence the test ouput while running the do_checktest task
• SHIFT_COVERAGE_EXCLUDES: Exclude paths from code coverage analysis
• SHIFT_COVERAGE_BRANCH: Generate branch coverage data
• SHIFT_REPORT_DIR: A path to store report files
• SHIFT_TEST_FILTER: Run tests matching regular expression
• SHIFT_TEST_SHUFFLE: Randomize the order of tests
• SHIFT_TEST_SUPPRESS_FAILURES: Do not return non-zero exit code when tests fail

Jenkins integration

It is recommended to set up The meta-shift plugin for Jenkins for your Jenkins instance.

Development

To prepare the meta-shift development environment via Docker:

$ git clone https://github.com/shift-left-test/dockerfiles.git
$ cd dockerfiles
$ docker build -f yocto-dev/20.04/Dockerfile -t yocto-dev-20.04 .
$ docker run --rm -it yocto-dev-20.04

To run all tests:

$ pytest

Contributing

This project is open to any patches. The patches can be submitted as Github pull request in https://github.com/shift-left-test/meta-shift or to the project mailing list.

License

This project source code is available under MIT license. See LICENSE. If individual files are licensed under different terms, the terms and conditions can be found in the individual file or file header.