Safety development concept (for IEC61508 certification)

[romkell]

Introduction

The quality and finally safety (IEC61508 SIL3 certification) and
security development of Zephyr OS requires

• to follow a or several processes
• to have a tool environment installed to complete tasks and support the processes.

This RFC defines a rough concept how the safety committee intends to setup, both,
tools and processes to do safety development.

Serveral areas, in example requirements management, are not restricted
to safety and shall be setup at the best of our current knowledge to suite
for the full extend of the ecosystem.

Nevertheless, in order to be able to finish with the safety certifcation task,
it must be accepted that the focus of the safety committee and WG and therefore
this RFC is safety.

Please feel invited

• to suggest improvement
• to disclose gaps
• to ask questions when unclear
• to state your opinion

Problem description

An overall concept how to do safety is not (fully) defined, described and known to the safety WG
and TSC.
It is necessary to achieve a common understanding.

Proposed change

Create an overall concept by defining processes and a tool environment that allows
to get 'main' in a 'safety certifiable ready' state of which an auditable can
be branched with basically no gaps to reach certification.
Doing safety on auditable and forward merge all code changes, all tests, requirements etc.
has shown to not work.

We start with an rough concept and iteratively detail and implement the different areas.

Detailed RFC

Proposed change (Detailed)

Overview

Requirements Management

For safety (IEC61508) the following specifications need to be written:

Requirements specifications

• User Story (US) - not requested by 61508 but makes requirements more understandable
• Software Requirements Specification (SRS)
• Software Architecture & Interface Specification (SAIS)
• Software Module Design Specification (SMDS)

Requirements management(SRS, SAIS, STS) is done using

• Doorstop (https://doorstop.readthedocs.io/en/latest/,https://github.com/doorstop-dev/doorstop)
• Doorhole (https://github.com/sevendays/doorhole) may serve as a GUI frontend for convenience

Storage location is zephyrproject-rtos/reqmgmt repo (https://github.com/zephyrproject-rtos/reqmgmt).

Test specifications

• Software Test Specification (STS)
• Software Integration Test Specification (SITS)
• Software Module Test Specification (SMTS)

Remark: possibly replace term 'module' as in 61508 by 'component' as in 50128 or 'unit' as in unit test for SMDS and SMTS, since easily confused with git (sub-)module.

Definition: Module as in 61508 is in most cases referred to as a implementation / header file pair in a C/C++ context (my_module.c/.h).

Test specifications are written as doxygen comments to the unit-, integration- and software tests into the zephyrproject-rtos/zephyr repo (https://github.com/zephyrproject-rtos/zephyr).
The linkage to a Doorstop requirements (SRS, SAIS, SMDS) is done by doxygen tag referencing the requirement by its ID.

Requirements process

Adding requirements or test specifications uses the same workflow as any code change does:

• Create a fork repo
• Create a branch where all changes are done
• Create a Github PR
• Review the PR
• Merge the PR

The standard github features (PR, diff, review) are used

• since Doorstop uses single yaml files per requirement (less merging than a single file approach as e.g. with StrictDoc, but would still work)
• the yaml file content is human readable
• users are acustomed to the github process (no major changes compared to source code)

Requirements wording

Requirememts shall be

• https://en.wikipedia.org/wiki/Requirement#Characteristics_of_good_requirements

and follow the 'Easy Approach to Requirements Syntax' (EARS) principle

• https://alistairmavin.com/ears/
• https://ieeexplore.ieee.org/document/5328509

Traceability

Traceability is created by a dediated 'tracedoc' tool.
The tool takes the

• requirements specification from Doorstop (or any other requirements tool) in a defined csv based format as input
  (other formats would be feasible but for simplicity shall not be considered in the beginning)
• scans the zephyrproject-rtos/zephyr code base for dedicated doxygen comments with reference tags to the requirements
• checks for dead links and missing traceability (report at the end of or highlighted in traceability documentation)
• creates a doxygen / sphinx base requirements and traceability documentation

Remark: Intel has a prove-of-concept implemented.

By defining an open and simple input interface (csv) into tracedoc, requirements tools are not limited to Doorstop
and the requirements do not necessarily have to come from the same source (repo, system etc.)

Future: tracedoc could be extended to process multiple csv input files from different requirements tools and scan over multiple repos. We start with the simple 1:1 case.

Linkages

The following link roles will be used:

• refines (in the requirements management tool (Doorstop):
  • software requirement <-- software architecture & interface requirement
  • software architecture & interface requirement <-- software module design requirement
• implements (Doxygen tag - exact syntax tbd):
  • software module design requirement <-- source code
• verifies (Doxygen tag - exact syntax tbd):
  • software requirement <-- software test requirement
  • software architecture & interface requirement <-- software integration test requirement
  • software module design requirement <-- software module test requirement

Safety Scope

Why a safety scope? The Zephyr OS is by far too big, that we can achieve a safety certification of the full code base.
Hence we need to restrict it to a smaller scope and provide measures to distinguish non-safety from safety (for us and for a user).

The safety scope as defined will be marked using two mechanisms:

1. The source files in the safety scope will carry an SPDX tag, similar to the SPDX license tag, indicating the safety relevance/scope

• In that way safety relevant code is clear identified and immediately visible to developers and maintainers
• Special (stricter) guidelines may apply to source code within the safety scope than to the rest of the code (e.g.: use of dynamic memory)
• The SPDX tool can be used to verify that a built binary only comprises of sources out of the safety scope and no other
  (similar to that a binary only comprises of sources under a certain license model (e.g. no GPL)

2. The maintainer.yml will define folders and file belonging to the safety scope

• In that manner PRs affecting the safety scope can be auto-assigned to dedicated safety maintainers
• Possibly to an include syntax also an exclude syntax need to be defined and implemented (/folder/A but not /folder/A/fileB.c)

Using both, SPDX and maintainer.yml, will be used to validate the safety scope against each other for plausibility / congruence (tool: scope validator, added to CI)

Coding Guidelines

The coding guidelines will be enforced as far as possible using a static code analyzer.
Guidelines which cannot be enforce by tooling, will be verified by code review prior to merging the github PR.

Static Code Analyzer (SCA)

• CI
  • The project uses a commercial grade SCA (Parasoft cpptest is foreseen) in its CI, creating a violation report accessible via github PR for each development branch and main.
  • The violation report will be differential for a development branch, between the development branch and the origin the development branch was branched from, only showing and enforcing the newly introduced violations.
  • A full violation report should be available / visible but not be enforced to be cleaned up - in hope some feel motivated to fix low hanging violation fruits
    (That could be skipped in CI and be left to the local SCA scans, where differential scans cannot be easily provided)
• Local
  • Commercial SCAs typically have a rather high price tag for the community and the project providing floating license will be come expensive and requires an elaborated infrastructure (license server, access management, preventing from use outside of the project scope etc.)
  • Different companies use different commercial SCAs, they will be able to integrate their SCA.
  • The open source SCA "cppcheck" will be available and integrated into the build system to be used on a local machine. Even if the open source SCA does not have the same quality as a commercial grade solutions, it will be suitable to detect and this way fix 80 - 90% of the violations in a fast development cycle. The remaining 10 - 20% will have to be done using the CI integrate SCA with slower development cycles.
  • The local scans will only provide a full scan of the source files, showing all existing violations (whom ever introduced them).

Testing

Testing will be done according to IEC61508 on the three levels:

• (full) software: verifying the software from a user perspective (black box testing)
  • The software is approached from a user API perspective
• integration: verifying the interaction between dedicated modules (grey box testing)
  • A number of modules are used in a test application. The modules under test will be approached via theirs API, external dependencies will be provided by faking (FFF) or using a device simulation.
• module: verifying single functions of a module including code coverage (white box testing)
  • module API:
    • Tests the module only via accessing the module API
  • module unit:
    • Takes the module-under-test as include into the test implementation file to allow access also to the local (static) functions
    • Limitation: testing caller functions on all possible return values, where the callee is in the same module will not be possible
      in all cases (see module stubbing).
  • module stubbing:
    • functions in the module that call static functions of the module might not be fully testable without stubbing or faking the called function
      (example: the called function interacting with some hardware which does not fail during tests and therefore always returning 0 = OK) but the callers reaction needs to be tested on error codes. That caller's code would not be reachable.
    • The function calls to module own functions need to be stubbed by suffixing the called function by "_stub" to allow to use fake function framework (FFF) to fake that called function (This will require a tool manipulating the source code file, suffixing the relevant function calls)

Testing folder structure

The /tests folder (as already existing) will resemble (except for the /tests folder itself) the repos folder structure and place the tests under dedicated sub-folders.

• /_s_test, for "software-test", will be used as sub-folder for the (full) software tests
• /_i_test, for "integration-test", will be used as sub-folder for the integration tests
  • at the top most level, which includes all participating modules
• /_m_a_test, for "module-api-test", will be used as sub-folder for the module api tests
  • It will be under the module names folder
  • Which is, what we most have today
• /_m_u_test, for "module-unit-test", will be used as sub-folder for the module unit tests
  • It will be under the module names folder
• /_m_s_test, for "module-stubbing-test", will be used as sub-folder for the module unit tests
  • It will be under the module names folder

An example to depict the idea
(the shown existing tests might have been placed in the wrong category/location, creating the shown folder structure will
require a transition phase from the existing structure to the shown)

• /tests
  • /_i_tests
    /sched_subsys_xyz
  • /_s_tests
    • /queue
    • /semaphore
    • /sched
      • /deadline
      • /preempt
    • /
  • /kernel
    • /_i_test
      • /sched_mutex
      • /sched_queue
    • /queue
      • /_m_a_test
      • /_m_u_test
    • /semaphore
      • /_m_a_test
      • /_m_u_test
    • /sched
      • /_m_a_test (former schedule_api test?)
      • /_m_u_test

Dependencies

Introducing a system for quality to achieve a safety certification will introduce changes in various areas.
It has to be an iterative step by step process in which the direction very likely will have to be corrected
now and then.

Concerns and Unresolved Questions

Is part of this RFC's discussion to adjust the concept.

Alternatives

Is part of this RFC's discussion to adjust the concept.

[gregshue]

Nice! Thank you!

1. What level of security must be in place to make these safety claims? This will help identify which security functionality must be included in the safety scope.

2. Some users will also need to extend or develop/verify safety-certifiable code in other repositories (e.g., libc, HAL, mcuboot, downstream forks of Zephyr). Please be explicit about how this process extends across multiple repositories, especially for each of the specified workspace topologies (T1, T2, T3).

3. End users will need to trace Zephyr requirements to their higher-level system requirements. Please describe how that can be done while still using Zephyr's safety-certifiable SHA.

[gregshue]

🤔 If "module" refers to an individual .c/.h pair, then we need to expect multiple SRS/SAIS recursions between the requirements on the executable and the requirements on an individual file. The number of recursions may change as the architecture evolves.

[nicpappler]

That's a lot to discuss in a single RFC - how about we split this into several sub RFCs?

[nicpappler]

@gregshue - regarding the security and safety claims - as already discussed, there are no security claims attached to the safety claims. Both safety and security need a certain level of systematic capability. We aim for the systematic capability of SIL 3 (IEC 61508:2010), that also is enough for security, but does not make specific claims regarding security features, mechanisms, etc.

[romkell]

That's a lot to discuss in a single RFC - how about we split this into several sub RFCs?

I would leave the rough concept in here as a central point and to have an overview which we loose when splitting - and the overview was one goal.
But we can create topic RFC such as "Safety development concept - RQM" etc. linked to here and discuss details there.

[gregshue]

@gregshue - regarding the security and safety claims - as already discussed, there are no security claims attached to the safety claims. Both safety and security need a certain level of systematic capability. We aim for the systematic capability of SIL 3 (IEC 61508:2010), that also is enough for security, but does not make specific claims regarding security features, mechanisms, etc.

I repeated the question here to capture the topic within this issue. The question originated from the following statement in Embedded Software Development for Safety-Critical Systems, Second Edition (Chris Hobbs, CRC Press, 2020), p 10,

Safety is interlinked with security - the behavior of an insecure device is effectively unpredictable once a hacker has found a vulnerability.

This may be another subject appropriate for a topic RFC.

[RobertoBagnara]

Regarding the choice of Doorstop, I think it is not a good one,
for the following reasons.

Doorstop is arguably unmaintained – its creator only sends Python
support patches from time to time. For any advanced patches from other
people, there are long unfinished threads hanging around with no
progress being made.

About "Doorstop uses single yaml files per requirement", this is not
so good granularity, if one considers that everything in the industry
is kept in documents (requirement specifications, test specifications,
verification control document, etc.). StrictDoc intentionally packages
requirements in .sdoc document files because people create and read
the whole documents and requirement sections. Maintaining separate
files without first-class document support is cumbersome. Take the
"level:" field for example
https://github.com/doorstop-dev/doorstop/blob/develop/reqs/REQ002.yml#LL4
– it is so inconvenient to manage it on the single req's YAML file
level.

Merge conflict argument works a little better with Doorstop, but I could
argue that if there is a conflict, this already means that the engineers
working on requirements went out of sync. To get a merge conflict in an
.sdoc file, you really have to work on the same or closely neighboring
requirements or someone has to start moving sections around, which
already indicates that people work without any synchronization.

Additionally, StrictDoc has made and is making a lot of progress recently:
see the GitHub project and compare it with the one for DoorStop.
In particular, StrictDoc has at least an 80%-working bridge to ReqIF export
and import. I personally used it to successfully import Polarion-generated
requirements in ReqIF format. Doorstop doesn't have ReqIF support.

[RobertoBagnara]

The following sentence misses one point: "Even if the open source SCA does not have the same quality as a commercial grade solutions, it will be suitable to detect and this way fix 80 - 90% of the violations in a fast development cycle."
It is not just that cppcheck has lots of false negatives, it also has lots of false positives. The danger is that developers that have not been properly trained on the coding guidelines will not be able to recognize the false positives and will believe they have to to something about them. In my experience, this always causes a decrease in code quality.

[gregshue]

We need a requirements management system keeping with the composability and extensibility of Zephyr.

1. It needs to support a single "document" (e.g., StRS, SyRS, SyAID, SRS, SAID, etc.) that has requirements spread across multiple repositories in the workspace.
2. It needs to support "variation management" - tracking moderately different requirement sets for closely related products.

I have attempted to use Doorstop and found it at least unable to link to requirements in other repositories. I didn't look into variation management or aggregating requirements from multiple repositories.

@RobertoBagnara How well does StrictDoc support those needs?

[sebhub]

Zephyr is not the first project for which a pre-qualification according to safety related standards is available. We did something similar for the RTEMS real-time operating system. The pre-qualification for RTEMS was performed using software development standards from the European Cooperation for Space Standardization: ECSS-E-ST-40C and ECSS-Q-ST-80C Rev.1. We took also other standards such as IEC 61508,
ISO 26262, DO-178, DO-330, DO-333, and Galileo Software Standard (GSWS) into account. The work started in 2020 and the first results are available on a ESA website: https://rtems-qual.io.esa.int/. The technical note QT-109 contains some background information. The latest documentation of the RTEMS pre-qualification process is available here: https://docs.rtems.org/branches/master/eng/req/index.html. A key observation is that the standards require a high degree of traceability from requirements to source code and tests and vice versa.

Interestingly, the proposal here has some overlaps with what we did. I can highly recommend the EARS syntax for requirements.

We also use Doxygen to make the link from the specification to the source code and vice versa. In a first pass, Doxygen is invoked to produce a tagfile. This tagfile is used to link requirements to source code. Then a file is generated which contains Doxygen comments for source code elements such as functions. The generated comments augment the existing Doxygen documentation and provide links to the specification.

For the requirements (or specification in general) we also started with Doorstop. However, while working with it, it turned out that some customization was required: https://docs.rtems.org/branches/master/eng/req/tooling.html. The RTEMS specification contains not just requirements, so we added a hierarchical type system for specification items: https://docs.rtems.org/branches/master/eng/req/items.html. These types are also specified by specification items: https://github.com/RTEMS/rtems-central/tree/master/spec/spec. It can be easily customized to meet project-specific requirements.

Real-time operating systems such as Zephyr or RTEMS are just a collection of state machines and data structures. For RTEMS the majority of the function specification is done by the specification of state machines. We use a specialized specification item for this: https://docs.rtems.org/branches/master/eng/req/howto.html#action-requirements. It contains the specification along with the corresponding validation test code. It needs a bit of learning to be able to write these files (example: https://github.com/RTEMS/rtems-central/blob/master/spec/c/req/posix-memalign.yml), however, it turned out to be very useful for the functional specification of RTEMS. We reach 100% branch and code coverage for the pre-qualified feature set of RTEMS through validation tests. Since we enumerate all state transitions, this yields more or less MC/DC coverage automatically.

Performance requirements are stated in terms of runtime measurement requests. The measurements are done using the RTEMS Test Framework. This test framework would also work with Zephyr.

We use Python modules for the tooling (BSD-2-Clause license): https://github.com/RTEMS/rtems-central/tree/master/rtemsspec.

Just let me know if you are interested in this work. Maybe it can be generalized so that it can be used also for Zephyr.

[romkell]

@sebhub

Zephyr is not the first project for which a pre-qualification according to safety related standards is available. We did something similar for the RTEMS real-time operating system.

Many thanks for reaching out to us.
We are aware of RTEMS in general, basically due to your evaluation of Doorstop and your decision to implement your own RQM tool for various reasons.
At least I have not looked into your detailed implementation and requirements of RTEMS, but appreciate your input.

The Zephyr OS code is existing and rewriting everything to yield MC/DC coverage will I assume effort-wise not work at least in the first place. It also is not demanded by IEC61508 SIL3, which we aim for.
Let me have a look at all the links you provided.