Embedded Software Testing Principles
Software testing is one of the most critical phases in the software development life cycle. This phase is completed through verification and system evaluation of the softwares functionality to ensure specified requirements are met. The primary objective of this testing is identification and correction of defects and errors to improve the reliability of the software. In order to begin talking about software testing principles in embedded systems, we must first know what embedded systems are.
Embedded systems are computer systems where their hardware and software components are specialized to perform a specific function. These systems normally operate in real time, and have specific limited constraints on size, power consumption, and processing power. These embedded systems are normally found in devices that we use everyday such as smartphones and home appliances; however, as the end user, we are unaware of these embedded systems in our products, or the testing principles that go into making them function.
Software testing principles in embedded systems consist of different testing stages, which include software unit testing, integration testing, system unit testing, system integration testing, and system validation testing (Figure 1). These embedded testing principles can be applied to any software development cycle methodology but for this article we will focus on the V Development Cycle (Figure 2). Each testing level corresponds to a different stage of the development cycle where a failure in the testing corresponds to a needed revision in that development stage.
Figure 1:
Figure 2:
We will begin by describing the first level, which is software unit testing. The software unit testing level corresponds to designing the modules in the V Development Cycle. This testing level consists of isolating each individual module and feature of the application to perform tests on it. Software unit tests can be automated, which allows the tests to rerun everytime there is a change in the code. The goal of this testing is to show the stability and soundness of that specific segment of code.
The benefits of unit testing for embedded software are: early bug detection, as this testing is early in the development process, which means catching them here does not allow them to become more complex; and it keeps code clean and makes it easier to refactor, as it provides clear information of how a specific component is supposed to be used and where the test failed.
Figure 3:
Integration testing is the second level of testing and consists of making sure that there is correct interaction and integration between the different components in the embedded systems software. After each component has been unit tested it goes through integration testing to ensure proper interaction between components.
A few different methodologies for integration testing are Big-Bang, Top-down, Bottom-up, and Mixed/Sandwich integration testing. In Big-Bang integration testing, all the modules and components are integrated together and tested as one. With this integration testing method, bugs can be more difficult to pinpoint. In Top-down integration testing, the components at a higher level are tested and then lower components are tested. In Bottom-up integration testing, a similar technique as the Top-down testing is used, except that it goes from testing lower level components to testing higher level components. In Mixed/Sandwich integration testing, it combines both the Bottom-up and Top-down integration techniques to make testing more efficient by testing the higher level modules with the lower level modules and testing the lower level modules with the higher level modules.
The benefits to integration testing for embedded software are: resolution of interface issues, as with many interconnected components we are bound to have unintended interactions between components; end to end validation, as we know component to component will flow through the software without error; and collaboration among teams, as teams will work on different components and integration testing forces interaction, and therefore ensuring continuity of the continuing software between teams.
Figure 4:
Figure 5:
Figure 6:
System unit testing, the third level of testing, consists of making sure that groups of software modules work properly. The software modules are grouped together then isolated from other groups, and finally each group is tested as one unit. This is similar to software unit testing talked about earlier, but the difference is that the individual software modules are grouped together to make unit systems and then each is tested as a whole.
Figure 7:
The fourth level of testing is system integration testing. System integration testing consists of testing how the software and hardware interact together, which is crucial to integration testing of embedded systems. When testing how the software and hardware interact with each other, the tests are intended to make sure that there is proper communication and control back and forth between the two. This includes making sure that the sensors and other devices are properly working and communicating how they are expected to with the software. System integration testing also includes testing to verify that it can handle errors properly, and that the system works as expected under real time constraints. System integration testing has a few methods of execution which are similar to that of integration testing, but system integration testing has an additional method. Hardware-in-the-loop testing consists of connecting the software to a hardware-in-the-loop simulator or the actual hardware through interfaces. This allows the simulation of multiple inputs and interactions, so that the hardware-in-the-loop can verify that the expected output is given.
The last level of testing is system validation testing. System validation testing consists of ensuring that the entire embedded system is working properly. This level helps reveal any issues before the product is released to the consumer and is intended to ensure a functional and reliable product. Any bugs that are missed by this last level could be catastrophic as they would be available for the consumer to be forced to deal with. System validation testing includes end-to-end testing, real-world conditions, and usability and user experience. End-to-end testing tests all the components as a whole and verifies that everything is working properly much like the integration testing. Real-world conditions allow testing in a similar environment to what the embedded system would be released into. Usability/user experience and user experience make sure that the product is user friendly and runs the way it should in order to best serve the end user. An embedded device can functionally work fine but be very difficult for the user to operate or work with which makes it a device that still has flaws.
The benefits of system validation testing for embedded software outweigh the benefits of all the other testing levels because it is the final defense against the end user getting a product that is unusable or the next big thing. Testing in real-world conditions is the closest we can get to user base testing without spoiling the experience of any users. System validation testing for embedded devices makes the difference between a great idea and a great device with exceptional software.
Understanding the principles of software testing in embedded systems is crucial for ensuring the reliability and functionality of the products we use in our everyday lives. Each level that has been gone over in this article tries to catch bugs and issues as soon as possible. This leveled approach serves two purposes in the quest to root out issues in the embedded code. We look for the bugs starting from a small view and continue to zoom out which helps to protect small issues from turning into big issues and causing more problems. By targeting an issue as soon as possible from a very up close few to one looking at the embedded code as a whole we also save money in development as the time needed to be spent on costly issues is reduced. Each level of testing has a specific and important purpose in identifying bugs and flaws in the implementation and design of the embedded system.
- https://www.embedded.com/the-basics-of-embedded-software-testing-part-1/
- https://www.hurix.com/embedded-testing-types-and-challenges/#:~:text=Embedded%20Testing%20is%20a%20process,software)%20fulfills%20the%20business%20requirements.
- https://www.guru99.com/embedded-software-testing.html
- https://lahirukasun666.medium.com/a-guide-to-testing-embedded-systems-2d3d9c4d391e
- https://www.spiceworks.com/tech/devops/articles/what-is-integration-testing/
- https://www.simplilearn.com/what-is-integration-testing-examples-challenges-approaches-article
- https://www.impactqa.com/blog/embedded-testing-vs-software-testing-key-differences/