This repository contains a set of network function test cases and the framework to build more. It also generates reports (claim.json) on the result of a test run. The tests and framework are intended to test the interaction of Cloud-Native Network Functions (CNFs) with OpenShift Container Platform.
The suite is provided here in part so that CNF Developers can use the suite to test their CNFs readiness for certification. Please see "CNF Developers" below for more information.
At a minimum, the following dependencies must be installed prior to running make deps-install
.
Dependency | Minimum Version |
---|---|
GoLang | 1.14 |
golangci-lint | 1.32.2 |
jq | 1.6 |
OpenShift Client | 4.4 |
All other dependencies required to run tests can be installed using the following command:
make deps-install
Note: You must also make sure that $GOBIN
(default $GOPATH/bin
) is on your $PATH
.
Note: Efforts to containerize this offering are considered a work in progress.
There are two categories for CNF tests; 'General' and 'CNF-specific'.
The 'General' tests are designed to test any commodity CNF running on OpenShift, and include specifications such as 'Default' network connectivity.
'CNF-specific' tests are designed to test some unique aspects of the CNF under test are behaving correctly. This could
include specifications such as issuing a GET
request to a web server, or passing traffic through an IPSEC tunnel.
The general-purpose category covers most tests. It consists of multiple suites that can be run in any combination as is appropriate for the CNF(s) under test:
Suite | Test Spec Description | Minimum OpenShift Version |
---|---|---|
diagnostic | The diagnostic test suite is used to gather node information from an OpenShift cluster. The diagnostic test suite should be run whenever generating a claim.json file. | 4.4.3 |
generic | The generic test suite is used to test Default network connectivity between containers. It also checks that the base container image is based on RHEL . |
4.4.3 |
multus | The multus test suite is used to test SR-IOV network connectivity between containers. | 4.4.3 |
operator | The operator test suite is designed basic Kubernetes Operator functionality. | 4.4.3 |
container | The container test suite is designed to test container functionality and configuration | 4.4.3 |
Further information about the current offering for each test spec is included below.
The diagnostic
test spec issues commands to poll environment information which can be appended to the claim file.
This information is necessary to ensure a properly spec'd environment is used, and allows the claim to be reproduced.
As of today, the diagnostic
test suite just polls Node
information for all Node
s in the cluster. Future
iterations may consider running lshw
or similar types of diagnostic tests.
The generic
test spec tests:
Default
network connectivity between containers.- That CNF container images are RHEL based.
To test Default
network connectivity, a test partner pod
is installed on the network. The test partner pod is instructed to issue ICMPv4 requests to each container listed in
the test configuration, and vice versa. The test asserts that
the test partner pod receives the correct number of replies, and vice versa.
In the future, other networking protocols aside from ICMPv4 should be tested.
Similar to the generic
test spec, the multus
test spec is utilized for CNFs that utilize multiple network
interfaces. As of today, the multus
test suite just tests that
a test partner pod can successfully ping the secondary
interface of the CNF containers. Since SR-IOV is often utilized, and the secondary interface of a CNF cannot be
accessed in user space, the test is unidirectional.
Currently, the operator
test spec is limited to a single test case called OPERATOR_STATUS
. OPERATOR_STATUS
just
checks that the CSV
corresponding to the CNF Operator is properly installed.
In the future, tests surrounding Operational Lifecycle Management
will be added.
The container
test spec has the following test cases:
Test Name | Description |
---|---|
HOST_NETWORK_CHECK |
Ensures that the CNF pods do not utilize host networking. Note: This test can be disabled for infrastructure CNFs that should utilize host networking. |
HOST_PORT_CHECK |
Ensures that the CNF pods do not utilize host ports. |
HOST_PATH_CHECK |
Ensures that the CNF pods do not utilize the host filesystem. |
HOST_IPC_CHECK |
Ensures that the CNF pods do not utilize host IPC namespace to access or control host processes. |
HOST_PID_CHECK |
Ensures that the CNF pods do not utilize host PID namespace to access or control host processes. |
CAPABILITY_CHECK |
Ensures that the CNF SCC is not configured to allow NET_ADMIN or SYS_ADMIN . |
ROOT_CHECK |
Ensure that the CNF pods are not run as root . |
PRIVILEGE_ESCALATION |
Ensure that the CNF SCC is not configured to allow privileged escalation. |
In the future, we are considering additional tests to ensure aspects such as un-alteration of the container image.
Currently, all available tests are part of the "CNF Certification Test Suite" test suite, which serves as the entrypoint
to run all test specs. CNF Certification 1.0
is not containerized, and involves pulling, building, then running the
tests.
By default, test-network-function
emits results to test-network-function/cnf-certification-tests_junit.xml
.
The included default configuration is for running generic
and multus
suites on the trivial example at
cnf-certification-test-partner. To configure for your
own environment, please see the Test Configuration section, below.
In order to pull the code, issue the following command:
mkdir ~/workspace
cd ~/workspace
git clone git@github.com:redhat-nfvpe/test-network-function.git
cd test-network-function
In order to build the test executable, first make sure you have satisfied the dependencies.
make build-cnf-tests
Gotcha: The make build*
commands run unit tests where appropriate. They do NOT test the CNF.
Once the executable is built, a CNF can be tested by specifying which suites to run using the run-cnf-suites.sh
helper
script.
Any combintation of the suites listed above can be run, e.g.
./run-cnf-suites.sh diagnostic
./run-cnf-suites.sh diagnostic generic
./run-cnf-suites.sh diagnostic generic multus
./run-cnf-suites.sh diagnostic operator
./run-cnf-suites.sh diagnostic generic multus container operator
Gotcha: The generic test suite requires that the CNF has both ping
and ip
binaries installed. Please add them
manually if the CNF under test does not include these. Automated installation of missing dependencies is targetted
for a future version.
There are a few pieces of configuration required to allow the test framework to access and test the CNF:
Config File | Purpose |
---|---|
generic_test_configuration.yml | Describes the CNF or CNFs that are to be tested, the container that will run the tests, and the test orchestrator. |
cnf_test_configuration.yml | Defines which containers and operators are to be tested, and in which roles. |
testconfigure.yml | Defines roles, and which tests are appropriate for which roles. It should not be necessary to modify this. |
Combining these configuration files is a near-term goal.
The config file generic_test_configuration.yml
contains three sections:
-
containersUnderTest:
describes the CNFs that will be tested. Each container is defined by the combination of itsnamespace
,podName
, andcontainerName
, which are also used to connect to the container when required.- Each entry for
containersUnderTest
must also define thedefaultNetworkDevice
of that container. There is also an optionalmultusIpAddresses
that can be omitted if the multus tests are not run.
- Each entry for
-
partnerContainers:
describes the containers that support the testing. MultiplepartnerContainers
allows for more complex testing scenarios. At the time of writing, only one is used, which will also be the test orchestrator. -
testOrchestrator:
references a partner containers that is used for the generic test suite. The test partner is used to send various types of traffic to each container under test. For example the orchestrator is used to ping a container under test, and to be the ping target of a container under test.
The included example defines a single container to be tested, and a single partner to do the testing.
Testing operator and containers in specific roles is currently configured separately from the generic tests. This is
configured using cnf_test_configuration.yml
.
cnf_test_configuration.yml
defines the roles under which operators and containers are to be tested.
The included example config is set up with some examples of this:
It will run the "OPERATOR_STATUS"
tests (as defined in testconfigure.yml
) against an etcd operator, and the
"PRIVILEGED_POD"
and "PRIVILEGED_ROLE"
tests against an nginx container.
A more extensive example is provided in pkg/config/example.yaml
The test suite generates a "claim" file, which describes the system(s) under test, the tests that were run, and the outcome of all of the tests. This claim file is the proof of the test run that is evaluated by Red Hat when "certified" status is being considered. For more information about the contents of the claim file please see the schema. You can read more about the purpose of the claim file and CNF Certification in the Guide.
e.g. Adding a cnf platform test results to your existing claim file.
You can use the claim cli tool to append other related test suite results to your existing claim.json file. The output of the tool will be an updated claim file.
go run cmd/tools/cmd/main.go claim-add --claimfile=claim.json --reportdir=/home/$USER/reports
Args:
--claimfile is an existing claim.json file
--repordir :path to test results that you want to include.
The tests result files from the given report dir will be appended under the result section of the claim file using file name as the key/value pair. The tool will ignore the test result, if the key name is already present under result section of the claim file.
"results": {
"cnf-certification-tests_junit": {
"testsuite": {
"-errors": "0",
"-failures": "2",
"-name": "CNF Certification Test Suite",
"-tests": "14",
When run the CNF test suite will output a report to the terminal that is primarily useful for Developers to evaluate and address problems. This output is similar to many testing tools.
Here's an example of a Test pass. It shows the Test running a command to extract the contents of /etc/redhat-release
and using a regular expression to match allowed strings. It also prints out the string that matched.:
------------------------------
generic when test(test) is checked for Red Hat version
Should report a proper Red Hat version
/Users/$USER/cnf-cert/test-network-function/test-network-function/generic/suite.go:149
2020/12/15 15:27:49 Sent: "if [ -e /etc/redhat-release ]; then cat /etc/redhat-release; else echo \"Unknown Base Image\"; fi\n"
2020/12/15 15:27:49 Match for RE: "(?m)Red Hat Enterprise Linux Server release (\\d+\\.\\d+) \\(\\w+\\)" found: ["Red Hat Enterprise Linux Server release 7.9 (Maipo)" "7.9"] Buffer: "Red Hat Enterprise Linux Server release 7.9 (Maipo)\n"
•
The following is the output from a Test failure. In this case, the test is checking that a CSV (ClusterServiceVersion) is installed correctly, but does not find it (the operator was not present on the cluster under test):
------------------------------
operator Runs test on operators when under test is: my-etcd/etcdoperator.v0.9.4
tests for: CSV_INSTALLED
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122
2020/12/15 15:28:19 Sent: "oc get csv etcdoperator.v0.9.4 -n my-etcd -o json | jq -r '.status.phase'\n"
• Failure [10.002 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:58
Runs test on operators
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:71
when under test is: my-etcd/etcdoperator.v0.9.4
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:121
tests for: CSV_INSTALLED [It]
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122
Expected
<int>: 0
to equal
<int>: 1
Developers of CNFs, particularly those targeting CNF Certification with Red Hat on OpenShift, can use this suite to test the interaction of their CNF with OpenShift. If you are interested in CNF Certification please contact Red Hat.
Refer to the rest of the documentation in this file to see how to install and run the tests as well as how to interpret the results.
You will need an OpenShift 4.4 installation running your CNF, and at least one other machine available to host the test suite. The cnf-certification-test-partner repository has a very simple example of this you can model your setup on.