diff --git a/doc/ref_arch/kubernetes/README.md b/doc/ref_arch/kubernetes/README.md deleted file mode 100644 index a42595034c..0000000000 --- a/doc/ref_arch/kubernetes/README.md +++ /dev/null @@ -1,54 +0,0 @@ -[<< Back](../) - -# Kubernetes based Reference Architecture - -This is Kubernetes based Reference Architecture (RA-2) - -## Release Information - -**Bundle: _7_** - -**Version: _0_** - -**Release Date: _4th Jan 2022_** - -## Bundle/Version History - -| Bundle.Version | Date | Note -| --- | --- | --- | -| 1.0-alpha | 10th January 2020 | Snezka Release | -| 3.0 | 15th May 2020 | Baldy Release | -| 4.0 | 25th Sep 2020 | Baraque Release | -| 5.0 | 29th Jan 2021 | Elbrus Release | -| 6.0 | 1st Jul 2021 | Kali Release | -| 7.0 | 4th Jan 2022 | Lakelse Release | - -## Overall Status - -| Chapter | Status | -| --- | --- | -| Chapter 01 | Complete | -| Chapter 02 | Lots of SME feedback | -| Chapter 03 | Lots of SME feedback | -| Chapter 04 | Lots of SME feedback | -| Chapter 05 | Lots of SME feedback | -| Chapter 06 | Still developing content | -| Chapter 07 | Lots of SME feedback | -| Appendix B - Guidance For workload isolation (Multitenancy) with Kubernetes for application vendors | Still developing content | - -## Table of Contents - -* [Chapter 01 - Overview](chapters/chapter01.md) -* [Chapter 02 - Architecture Requirements](chapters/chapter02.md) -* [Chapter 03 - L2: High Level Architecture](chapters/chapter03.md) -* [Chapter 04 - L3: Component Level Architecture](chapters/chapter04.md) -* [Chapter 05 - Security Guidance](chapters/chapter05.md) -* [Chapter 06 - Special Interest Group level requirements](chapters/chapter06.md) -* [Chapter 07 - Gaps, Innovation, and Development](chapters/chapter07.md) -* [Appendix A - Guidance For workload isolation (multitenancy) with Kubernetes for application vendors](chapters/appendix-a.md) - -## Required versions of most important components - -| Component | Required version(s) | -| -----------|---------------------| -| Kubernetes | 1.22 | diff --git a/doc/ref_arch/kubernetes/README.rst b/doc/ref_arch/kubernetes/README.rst new file mode 100644 index 0000000000..03c9f9dcc5 --- /dev/null +++ b/doc/ref_arch/kubernetes/README.rst @@ -0,0 +1,68 @@ +Kubernetes based Reference Architecture +======================================= + +This is Kubernetes based Reference Architecture (RA-2) + +Release Information +------------------- + +**Bundle: 7** + +**Version: 0** + +**Release Date: 4th Jan 2022** + +Bundle/Version History +---------------------- + +============== ================= =============== +Bundle.Version Date Note +============== ================= =============== +1.0-alpha 10th January 2020 Snezka Release +3.0 15th May 2020 Baldy Release +4.0 25th Sep 2020 Baraque Release +5.0 29th Jan 2021 Elbrus Release +6.0 1st Jul 2021 Kali Release +7.0 4th Jan 2022 Lakelse Release +============== ================= =============== + +Overall Status +-------------- + +=================================================================================================== ======================== +Chapter Status +=================================================================================================== ======================== +Chapter 01 Complete +Chapter 02 Lots of SME feedback +Chapter 03 Lots of SME feedback +Chapter 04 Lots of SME feedback +Chapter 05 Lots of SME feedback +Chapter 06 Still developing content +Chapter 07 Lots of SME feedback +Appendix B - Guidance For workload isolation (Multitenancy) with Kubernetes for application vendors Still developing content +=================================================================================================== ======================== + +Table of Contents +----------------- + +.. toctree:: + :numbered: + :maxdepth: 1 + + chapters/chapter01 + chapters/chapter02 + chapters/chapter03 + chapters/chapter04 + chapters/chapter05 + chapters/chapter06 + chapters/chapter07 + chapters/appendix-a + +Required versions of most important components +---------------------------------------------- + +========== =================== +Component Required version(s) +========== =================== +Kubernetes 1.22 +========== =================== diff --git a/doc/ref_arch/kubernetes/chapters/appendix-a.md b/doc/ref_arch/kubernetes/chapters/appendix-a.rst similarity index 66% rename from doc/ref_arch/kubernetes/chapters/appendix-a.md rename to doc/ref_arch/kubernetes/chapters/appendix-a.rst index 0bb7c23e53..45bea6d245 100644 --- a/doc/ref_arch/kubernetes/chapters/appendix-a.md +++ b/doc/ref_arch/kubernetes/chapters/appendix-a.rst @@ -1,35 +1,29 @@ -[<< Back](../../kubernetes) +Appendix A - Guidance For workload isolation (Multitenancy) with Kubernetes for application vendors +=================================================================================================== -# Appendix A - Guidance For workload isolation (Multitenancy) with Kubernetes for application vendors +Overview +-------- -

scope

- -## Table of Contents - -- [Appendix A - Guidance For workload isolation (Multitenancy) with Kubernetes for application vendors](#appendix-a---guidance-for-workload-isolation-multitenancy-with-kubernetes-for-application-vendors) - - [A.1 Overview](#a1-overview) - - [A.2 Solution Areas](#a2-solution-areas) - - [A.3 Multitenancy Models](#a3-multitenancy-models) - - [A.3.1 Soft Multitenancy with Kubernetes Namespaces per tenant](#a31-soft-multitenancy-with-kubernetes-namespaces-per-tenant) - - [A.3.2 Hard Multitenancy with dedicated Kubernetes clusters per tenant](#a32-hard-multitenancy-with-dedicated-kubernetes-clusters-per-tenant) - -## A.1 Overview - -Problem statement: A single Kubernetes Cluster does not provide hard multitenancy* by design. Within a Cluster, Kubernetes Namespace is a mechanism to provide Soft isolation multitenancy. +Problem statement: A single Kubernetes Cluster does not provide hard multitenancy\* by design. Within a Cluster, Kubernetes Namespace is a mechanism to provide Soft isolation multitenancy. A Kubernetes Namespace does provide isolation by means of role based access control (RBAC), Resource Isolation and Network Policy, however they are still within the same trust domain and a potential breach of Cluster Admin Role could lead to the Blast Radius across the entire Cluster and all its Kubernetes Namespaces. So there is a need to define various use cases or ways to build Multitenancy Deployment Models and define the Best Practices to secure each Model. Kubernetes Namespace is a logical representation of namespace(boundary for resources) within the Kubernetes Cluster. -This is different from the [linux namespaces](https://en.wikipedia.org/wiki/Linux_namespaces) which are defined at the operating system kernel level. +This is different from the `linux namespaces `__ which are defined at the operating system kernel level. + +.. image:: ../figures/Model2-cluster-isolation.png + :alt: "Cluster Isolation" + +.. image:: ../figures/Model1-ns.png + :alt: "Network Service" -

scope

-

scope

Use cases: 1. Two CNFs which are in the same trust domain (e.g.: they are from the same vendor) are running in a container infrastructure 2. Two CNFs which are in different trust domains (e.g.: they are from different vendors) are running in a container infrastructure -## A.2 Solution Areas +Solution Areas +-------------- The scope is to identify the solution area which is needed to secure the CNF workloads. Securing the platform might happen as part of it but not directly the focus or objective here. @@ -40,34 +34,38 @@ The scope is to identify the solution area which is needed to secure the CNF wor 5. RBAC rules and secrets Management for each tenant 6. Separate Isolated view of Logging, Monitoring, Alerting and Tracing per tenant -## A.3 Multitenancy Models +Multitenancy Models +------------------- Solution Models : 1. **Soft Multitenancy**: Separate Kubernetes Namespace per tenant within a Single Kubernetes Cluster. The same Kubernetes Cluster and its control plane are being shared between multiple tenants. 2. **Hard Multitenancy**: Separate Kubernetes Cluster per tenant. -The Kubernetes Clusters can be created using Baremetal Nodes or Virtual Machines, either on Private or Public Cloud. -The workloads do not share the same resources and Clusters are isolated. + The Kubernetes Clusters can be created using Baremetal Nodes or Virtual Machines, either on Private or Public Cloud. + The workloads do not share the same resources and Clusters are isolated. -### A.3.1 Soft Multitenancy with Kubernetes Namespaces per tenant +Soft Multitenancy with Kubernetes Namespaces per tenant +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Soft multitenancy with Namespaces per tenant can be implemented, resulting in a multi-tenant cluster - where multiple trusted workloads share a cluster and its control plane. This is mainly recommended to reduce management overhead when the tenants belong to the same trust domain, and have the same Cluster configuration requirements (incl. release, add-ons, etc.). The tenants will share the cluster control plane and API, including all add-ons, extensions, CNIs, CSIs, and any Custom Resources and Controllers. -To manage access control, the Kubernetes RBAC must be configured with rules to allow specific tenants to access only the objects within their own Namespace, using a `Role` Object to group the resources within a namespace, and a `RoleBinding` Object to assign it to a user or a group of users within a Namespace. +To manage access control, the Kubernetes RBAC must be configured with rules to allow specific tenants to access only the objects within their own Namespace, using a ``Role`` Object to group the resources within a namespace, and a ``RoleBinding`` Object to assign it to a user or a group of users within a Namespace. -In order to prevent (or allow) network traffic between Pods belonging to different Namespaces, `NetworkPolicy` must be created as well. +In order to prevent (or allow) network traffic between Pods belonging to different Namespaces, ``NetworkPolicy`` must be created as well. Resource quotas enable the cluster administrator to allocate CPU and Memory to each Namespace, limiting the amount of resources the objects belonging to that Namespace can consume. This may be configured in order to ensure that all tenants use no more than the resources they are assigned. By default, the Kubernetes scheduler will run pods belonging to any namespace on any cluster node. If it is required that pods from different tenants are run on different hosts, then affinity rules should be created by using the desired Node Labels on the Pod definition. Alternatively, Node taints can be used to reserve certain nodes for a predefined tenant. -### A.3.2 Hard Multitenancy with dedicated Kubernetes clusters per tenant +Hard Multitenancy with dedicated Kubernetes clusters per tenant +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When tenants do not belong to the same trust domain, or the requirements on the cluster setup and configuration are irreconciliable, Hard Multitenancy must be implemented by creating multiple Kubernetes clusters for each tenant or group of compatible tenants. All the default design decision for Kubernetes clusters apply in this case, and no special segregation or capacity management is required to be setup within the clusters. From an operational perspective, the increased computational and operational overhead and the Cluster LCM (incl. Cluster provisioning automation) are the most impactful aspects. + diff --git a/doc/ref_arch/kubernetes/chapters/chapter01.md b/doc/ref_arch/kubernetes/chapters/chapter01.md index 4b40463338..752030c7ee 100644 --- a/doc/ref_arch/kubernetes/chapters/chapter01.md +++ b/doc/ref_arch/kubernetes/chapters/chapter01.md @@ -1,23 +1,7 @@ -[<< Back](../../kubernetes) -# 1. Overview +# Overview -

scope

- -## Table of Contents - -- [1. Overview](#1-overview) - - [1.1 Introduction](#11-introduction) - - [1.2 Terminology](#12-terminology) - - [1.3 Principles](#13-principles) - - [1.3.1 Cloud Native Principles](#131-cloud-native-principles) - - [1.3.2 Exceptions](#132-exceptions) - - [1.3.2.1 Technology Exceptions](#1321-technology-exceptions) - - [1.3.2.2 Requirements Exceptions](#1322-requirements-exceptions) - - [1.4 Scope](#14-scope) - - [1.5 Approach](#15-approach) - -## 1.1 Introduction +## Introduction The intention of this Reference Architecture is to develop a usable Kubernetes based platform for the Telecom operator community. The RA will be based on the standard Kubernetes platform where ever possible. This Reference Architecture for Kubernetes will describe the high level system components and their interactions, taking the [goals and requirements](../../../common/chapter00.md) and mapping them to real-world Kubernetes (and related) components. This document needs to be sufficiently detailed and robust such that it can be used to guide the production deployment of Kubernetes within an operator, whilst being flexible enough to evolve with and remain aligned with the wider Kubernetes ecosystem outside of Telco. @@ -27,15 +11,15 @@ To assist with the goal of creating a reference architecture that will support T The Kubernetes Reference Architecture will be used to determine a Kubernetes Reference Implementation. The Kubernetes Reference Implementation would then also be used to test and validate the supportability and compatibility with Kubernetes-based Network Function workloads, and lifecycle management of Kubernetes clusters, of interest to the Anuket community. It is expected that the Kubernetes Reference Architecture, Reference Implementation, and Reference Conformance will be developed building on the work already in place for OpenStack in the Anuket project. The intention is to expand as much of the existing test frameworks to be used for the verification and conformance testing of Kubernetes-based workloads, and Kubernetes cluster lifecycle management. -### 1.2 Terminology +### Terminology For terminology used in this document refer to the [glossary](../../../common/glossary.md). -## 1.3 Principles +## Principles This Reference Architecture conforms with the principles defined [here](../../../common/chapter00.md#2.0). -### 1.3.1 Cloud Native Principles +### Cloud Native Principles The definition for Cloud Native is somewhat controversial. For the purposes of this document, the CNCF TOC's (Technical Oversight Committee) definition of Cloud Native will be used: >CNCF Cloud Native Definition v1.0 @@ -62,13 +46,13 @@ Individual contributors who are also active in the CNCF TUG (Telecom User Group) - **robust automation** - **high-impact changes frequently and predictably** -### 1.3.2 Exceptions +### Exceptions -Anuket specification define certain policies and [principles](../../../common/chapter00.md#2.0) and strives to coalesce the industry towards conformant Cloud Infrastructure technologies and configurations. With the currently available technology options, incompatibilities, performance and operator constraints (including costs), these policies and principles may not always be achievable and, thus, require an exception process. These policies describe how to handle [non-conforming technologies](../../../common/policies.md#cntt-policies-for-managing-non-conforming-technologies). In general, non-conformance with policies is handled through a set of exceptions (please also see [Exception Types](../../../gov/chapters/chapter09.md#942-exception-types)). +Anuket specification define certain policies and [principles](../../../common/chapter00.md#2.0) and strives to coalesce the industry towards conformant Cloud Infrastructure technologies and configurations. With the currently available technology options, incompatibilities, performance and operator constraints (including costs), these policies and principles may not always be achievable and, thus, require an exception process. These policies describe how to handle [non-conforming technologies](../../../common/policies.md#cntt-policies-for-managing-non-conforming-technologies). In general, non-conformance with policies is handled through a set of exceptions (please also see [Exception Types](../../../gov/chapters/chapter09.md#exception-types)). The following sub-sections list the exceptions to the principles of Anuket specifications and shall be updated whenever technology choices, versions and requirements change. The Exceptions have an associated period of validity and this period shall include time for transitioning. -#### 1.3.2.1 Technology Exceptions +#### Technology Exceptions The list of Technology Exceptions will be updated or removed when alternative technologies, aligned with the principles of Anuket specifications, develop and mature. @@ -76,7 +60,7 @@ The list of Technology Exceptions will be updated or removed when alternative te |-----|------|-------------|-------------|-----------|-------------| | ra2.exc.tec.001 | SR-IOV | This exception allows workloads to use SR-IOV over PCI-PassThrough technology. | TBD | Emulation of virtual devices for each virtual machine creates an I/O bottleneck resulting in poor performance and limits the number of virtual machines a physical server can support. SR-IOV implements virtual devices in hardware, and by avoiding the use of a switch, near maximal performance can be achieved. For containerisation the downsides of creating dependencies on hardware is reduced as Kubernetes nodes are either physical, or if virtual have no need to "live migrate" as a VNF VM might.| | -#### 1.3.2.2 Requirements Exceptions +#### Requirements Exceptions The Requirements Exceptions lists the Reference Model (RM) requirements and/or Reference Architecture (RA) requirements that will be either waived or be only partially implemented in this version of the RA. The exception list will be updated to allow for a period of transitioning as and when requirements change. @@ -84,7 +68,7 @@ The Requirements Exceptions lists the Reference Model (RM) requirements and/or R |-----|------|-------------|-------------|-----------|-------------| | ra1.exc.req.001 | Requirement | xxx | xxxxxxxxxxxxx. | | | | -## 1.4 Scope +## Scope The scope of this particular Reference Architecture can be described as follows (the capabilities themselves will be listed and described in subsequent chapters), also shown in Figure 1-1: @@ -97,10 +81,11 @@ The following items are considered **out of scope**: - **Kubernetes-based Application / VNF Management**: similar to VNFM, this is an application layer capability that is out of scope of Anuket. This includes Kubernetes-based Application Package Management, such as Helm, as this is a client application and set of libraries that would be part of a modern/cloud native VNFM, not part of the infrastructure itself. -

Kubernetes Reference Architecture scope

-

Figure 1-1: Kubernetes Reference Architecture scope

+![**Figure 1-1:**: Kubernetes Reference Architecture scope](../figures/ch01_scope_k8s.png) + +**Figure 1-1:**: Kubernetes Reference Architecture scope -## 1.5 Approach +## Approach The approach taken in this Reference Architecture is to start as simply as possible (i.e. with a basic Kubernetes architecture), and then add detail and additional features/extensions as is required to meet the requirements of the Reference Model and the functional and non-functional requirements of common cloud native network functions. diff --git a/doc/ref_arch/kubernetes/chapters/chapter01.rst b/doc/ref_arch/kubernetes/chapters/chapter01.rst new file mode 100644 index 0000000000..11ffd3b8e0 --- /dev/null +++ b/doc/ref_arch/kubernetes/chapters/chapter01.rst @@ -0,0 +1,115 @@ +Overview +======== + +Introduction +------------ + +The intention of this Reference Architecture is to develop a usable Kubernetes based platform for the Telecom operator community. The RA will be based on the standard Kubernetes platform where ever possible. This Reference Architecture for Kubernetes will describe the high level system components and their interactions, taking the `goals and requirements <../../../common/chapter00.md>`__ and mapping them to real-world Kubernetes (and related) components. This document needs to be sufficiently detailed and robust such that it can be used to guide the production deployment of Kubernetes within an operator, whilst being flexible enough to evolve with and remain aligned with the wider Kubernetes ecosystem outside of Telco. + +To set this in context, it makes sense to start with the high level definition and understanding of Kubernetes. `Kubernetes `__ is a "portable, extensible, open-source platform for managing containerised workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available" [`kubernetes.io `__]. Kubernetes is developed as an open source project in the `kubernetes/kubernetes `__ repository of GitHub. + +To assist with the goal of creating a reference architecture that will support Telco workloads, but at the same time leverage the work that already has been completed in the Kubernetes community, RA2 will take an "RA2 `Razor `__" approach to build the foundation. This can be explained along the lines of "if something is useful for non-Telco workloads, we will not include it only for Telco workloads". For example, start the Reference Architecture from a vanilla Kubernetes (say, v1.16) feature set, then provide clear evidence that a functional requirement cannot be met by that system (say, multi-NIC support), only then the RA would add the least invasive, Kubernetes-community aligned extension (say, Multus) to fill the gap. If there are still gaps that cannot be filled by standard Kubernetes community technologies or extensions then the RA will concisely document the requirement and approach the relevant project maintainers with a request to add this functionality into the feature set. + +The Kubernetes Reference Architecture will be used to determine a Kubernetes Reference Implementation. The Kubernetes Reference Implementation would then also be used to test and validate the supportability and compatibility with Kubernetes-based Network Function workloads, and lifecycle management of Kubernetes clusters, of interest to the Anuket community. It is expected that the Kubernetes Reference Architecture, Reference Implementation, and Reference Conformance will be developed building on the work already in place for OpenStack in the Anuket project. The intention is to expand as much of the existing test frameworks to be used for the verification and conformance testing of Kubernetes-based workloads, and Kubernetes cluster lifecycle management. + +Terminology +~~~~~~~~~~~ + +For terminology used in this document refer to the `glossary <../../../common/glossary.md>`__. + +Principles +---------- + +This Reference Architecture conforms with the principles defined `here <../../../common/chapter00.md#2.0>`__. + +Cloud Native Principles +~~~~~~~~~~~~~~~~~~~~~~~ + +The definition for Cloud Native is somewhat controversial. For the purposes of this document, the CNCF TOC's (Technical Oversight Committee) definition of Cloud Native will be used: + + CNCF Cloud Native Definition v1.0 + Approved by TOC: 2018-06-11 + +.. + + “Cloud native technologies empower organizations to build and run **scalable** applications in modern, **dynamic environments** such as public, private, and hybrid clouds. Containers, **service meshes**, **microservices**, **immutable infrastructure**, and **declarative APIs** exemplify this approach. + + These techniques enable **loosely coupled** systems that are **resilient**, **manageable**, and **observable**. Combined with **robust automation**, they allow engineers to make **high-impact changes frequently and predictably** with minimal toil. + +.. + + The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. We democratize state-of-the-art patterns to make these innovations accessible for everyone.” + +Individual contributors who are also active in the CNCF TUG (Telecom User Group), formed in June 2019, are also working on a set of Cloud Native Principles that are more suited to the requirements of the Telecom community and with more detail than the existing CNCF definition: `Expanded Cloud Native Principles `__. There are many similarities, but the key principles from both, which are applicable to this document, are: + +- **scalable** +- **dynamic environments** +- **service meshes** +- **microservices** +- **immutable infrastructure** +- **declarative APIs** +- **loosely coupled** +- **resilient** +- **manageable** +- **observable** +- **robust automation** +- **high-impact changes frequently and predictably** + +Exceptions +~~~~~~~~~~ + +Anuket specification define certain policies and `principles <../../../common/chapter00.md#2.0>`__ and strives to coalesce the industry towards conformant Cloud Infrastructure technologies and configurations. With the currently available technology options, incompatibilities, performance and operator constraints (including costs), these policies and principles may not always be achievable and, thus, require an exception process. These policies describe how to handle `non-conforming technologies <../../../common/policies.md#cntt-policies-for-managing-non-conforming-technologies>`__. In general, non-conformance with policies is handled through a set of exceptions (please also see `Exception Types <../../../gov/chapters/chapter09.md#exception-types>`__). + +The following sub-sections list the exceptions to the principles of Anuket specifications and shall be updated whenever technology choices, versions and requirements change. The Exceptions have an associated period of validity and this period shall include time for transitioning. + +Technology Exceptions +^^^^^^^^^^^^^^^^^^^^^ + +The list of Technology Exceptions will be updated or removed when alternative technologies, aligned with the principles of Anuket specifications, develop and mature. + +=============== ====== ============================================================================== =========== ================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================ =========== +Ref Name Description Valid Until Rationale Implication +=============== ====== ============================================================================== =========== ================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================ =========== +ra2.exc.tec.001 SR-IOV This exception allows workloads to use SR-IOV over PCI-PassThrough technology. TBD Emulation of virtual devices for each virtual machine creates an I/O bottleneck resulting in poor performance and limits the number of virtual machines a physical server can support. SR-IOV implements virtual devices in hardware, and by avoiding the use of a switch, near maximal performance can be achieved. For containerisation the downsides of creating dependencies on hardware is reduced as Kubernetes nodes are either physical, or if virtual have no need to "live migrate" as a VNF VM might. +=============== ====== ============================================================================== =========== ================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================ =========== + +Requirements Exceptions +^^^^^^^^^^^^^^^^^^^^^^^ + +The Requirements Exceptions lists the Reference Model (RM) requirements and/or Reference Architecture (RA) requirements that will be either waived or be only partially implemented in this version of the RA. The exception list will be updated to allow for a period of transitioning as and when requirements change. + +=============== =========== =========== ============== ========= =========== +Ref Name Description Valid Until Rationale Implication +=============== =========== =========== ============== ========= =========== +ra1.exc.req.001 Requirement xxx xxxxxxxxxxxxx. +=============== =========== =========== ============== ========= =========== + +Scope +----- + +The scope of this particular Reference Architecture can be described as follows (the capabilities themselves will be listed and described in subsequent chapters), also shown in Figure 1-1: + +- Kubernetes capabilities required to conform to the Reference Model requirements +- Support for CNFs that consist wholly of containers +- Support for CNFs that consist partly of containers and partly of VMs, both of which will be orchestrated by Kubernetes +- **Kubernetes Cluster lifecycle management**: including Cluster creation/upgrade/scaling/deletion, and node customisation due to workload requirements. **Note**: *detailed requirements and component specification of cluster LCM are out of scope for this release.* + +The following items are considered **out of scope**: + +- **Kubernetes-based Application / VNF Management**: similar to VNFM, this is an application layer capability that is out of scope of Anuket. This includes Kubernetes-based Application Package Management, such as Helm, as this is a client application and set of libraries that would be part of a modern/cloud native VNFM, not part of the infrastructure itself. + +.. image:: ../figures/ch01_scope_k8s.png + :alt: "Figure 1-1:: Kubernetes Reference Architecture scope" + + +**Figure 1-1:**: Kubernetes Reference Architecture scope + +Approach +-------- + +The approach taken in this Reference Architecture is to start as simply as possible (i.e. with a basic Kubernetes architecture), and then add detail and additional features/extensions as is required to meet the requirements of the Reference Model and the functional and non-functional requirements of common cloud native network functions. + +For example, while the management of VMs through Kubernetes is included, the intention is to start with the "native" control of containers and add support for VMs at a later date. The final decision will be determined and documented in the Roadmap section. + +This document will start with a description of interfaces and capabilities (the "what") before at a later date providing guidance on "how" those elements are deployed. The details of how the elements will be used together will be documented in full detail in the Reference Implementation. + diff --git a/doc/ref_arch/kubernetes/chapters/chapter02.md b/doc/ref_arch/kubernetes/chapters/chapter02.md index 3b2fb94f90..26b991b74f 100644 --- a/doc/ref_arch/kubernetes/chapters/chapter02.md +++ b/doc/ref_arch/kubernetes/chapters/chapter02.md @@ -1,33 +1,16 @@ -[<< Back](../../kubernetes) -# 2. Architecture Requirements +# Architecture Requirements -

scope

- -## Table of Contents - -- [2. Architecture Requirements](#2-architecture-requirements) - - [2.1 Introduction](#21-introduction) - - [2.1.1 Definitions](#211-definitions) - - [2.2 Reference Model Requirements](#22-reference-model-requirements) - - [2.2.1 Cloud Infrastructure Software Profile Capabilities](#221-cloud-infrastructure-software-profile-capabilities) - - [2.2.2 Virtual Network Interface Specifications](#222-virtual-network-interface-specifications) - - [2.2.3 Cloud Infrastructure Software Profile Requirements](#223-cloud-infrastructure-software-profile-requirements) - - [2.2.4 Cloud Infrastructure Hardware Profile Requirements](#224-cloud-infrastructure-hardware-profile-requirements) - - [2.2.5 Cloud Infrastructure Management Requirements](#225-cloud-infrastructure-management-requirements) - - [2.2.6 Cloud Infrastructure Security Requirements](#226-cloud-infrastructure-security-requirements) - - [2.3 Kubernetes Architecture Requirements](#23-kubernetes-architecture-requirements) - -## 2.1 Introduction +## Introduction This chapter will use the requirements defined in the overall Reference Model and only make additional entries in section [2.3](#2.3) if there are additional requirements needed for this Reference Architecture. -## 2.1.1 Definitions +## Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://www.ietf.org/rfc/rfc2119.txt). -## 2.2 Reference Model Requirements +## Reference Model Requirements The tables below contain the requirements from the Reference Model to cover the Basic and High-Performance profiles. The table also includes a reference to the specification from [Chapter 04 - Component Level Architecture](./chapter04.md) and from [Chapter 05 - Security Guidance](/chapter05.md) to ensure traceability. If the related Specification does not exist, the reference will read "N/A" (and in bold "**N/A**" for mandatory requirements). @@ -38,269 +21,272 @@ To ensure alignment with the infrastructure profile catalogue, the following req - Those relating to Cloud Infrastructure Management - Those relating to Cloud Infrastructure Security -### 2.2.1 Cloud Infrastructure Software Profile Capabilities +### Cloud Infrastructure Software Profile Capabilities | Reference Model Section | Reference | Description | Requirement for Basic Profile | Requirement for High-Performance Profile| Specification Reference | |---|---|---|---|---|---| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.001 | Max number of vCPU that can be assigned to a single Pod by the Cloud Infrastructure | At least 16 | At least 16 | [ra2.ch.011](chapter04.md#42-kubernetes-node)| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.002 | Max memory in MB that can be assigned to a single Pod by the Cloud Infrastructure | at least 32 GB | at least 32 GB | [ra2.ch.012](chapter04.md#42-kubernetes-node)| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.003 | Max storage in GB that can be assigned to a single Pod by the Cloud Infrastructure | at least 320 GB | at least 320 GB | [ra2.ch.010](chapter04.md#42-kubernetes-node)| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.004 | Max number of connection points that can be assigned to a single Pod by the Cloud Infrastructure | 6 | 6 | [ra2.ntw.003](chapter04.md#45-networking-solutions) | -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.005 | Max storage in GB that can be attached / mounted to Pod by the Cloud Infrastructure | Up to 16TB(1) | Up to 16TB() | N/A | -| [4.2.2](../../../ref_model/chapters/chapter04.md#422-profiles-specifications--capability-mapping) | e.cap.006 | CPU pinning support | Not required | Must support | [ra2.k8s.009](chapter04.md#43-kubernetes)| -| [4.2.2](../../../ref_model/chapters/chapter04.md#422-profiles-specifications--capability-mapping) | e.cap.007 | NUMA support | Not required | Must support | [ra2.k8s.006](chapter04.md#43-kubernetes)| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.008 | IPSec Acceleration using the virtio-ipsec interface | Not required | Optional | N/A | -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.009 | Crypto Acceleration using the virtio-crypto interface | Not required | Optional | N/A | -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.010 | Transcoding Acceleration | Not required | Not required | N/A | -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.011 | Programmable Acceleration | Not required | Not required | N/A | -| [4.1.2](../../../ref_model/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.012 | Enhanced Cache Management: L=Lean; E=Equal; X=eXpanded | E | E | N/A | -| [4.2.2](../../../ref_model/chapters/chapter04.md#422-profiles-specifications--capability-mapping) | e.cap.013 | SR-IOV over PCI-PT | Not required | Must support | [ra2.ch.002](chapter04.md#42-kubernetes-node)
[ra2.ch.003](chapter04.md#42-kubernetes-node)
[ra2.k8s.007](chapter04.md#43-kubernetes)
[ra2.ntw.004](chapter04.md#45-networking-solutions)
[ra2.ntw.008](chapter04.md#45-networking-solutions)| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.014 | Hardware coprocessor support (GPU/NPU) | Not required | Not required | N/A| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.015 | SmartNICs | Not required | Optional | N/A | -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | e.cap.016 | FPGA/other Acceleration H/W | Not required | Optional | [ra2.k8s.007](chapter04.md#43-kubernetes)
[ra2.ntw.012](chapter04.md#45-networking-solutions)| -| [4.1.2](../../../ref_model/chapters/chapter04.md#412-exposed-infrastructure-capabilities) | *e.cap.017* | *Ability to monitor L2-L7 data from workload* | *n/a(2)* | *n/a(2)* | N/A | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.cap.014 | Specifies the proportion of CPU cores consumed by the Cloud Infrastructure system on the worker nodes. If SMT is used, it indicates the number of consumed SMT threads. | 2 | 2 | [ra2.k8s.008](chapter04.md#43-kubernetes) | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.cap.015 | Indicates the memory consumed by Cloud Infrastructure on the worker nodes | 16 GB | 16GB | | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.cap.016 | Number of virtual cores per physical core; also known as CPU overbooking ratio that is required | 1:1 | 1:1 | [ra2.ch.004](chapter04.md#42-kubernetes-node)
[ra2.ch.005](chapter04.md#42-kubernetes-node)| -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.cap.017 | QoS enablement of the connection point (vNIC or interface)| Not required | Must support | **N/A** | -| [4.1.4](../../../ref_model/chapters/chapter04.chapter04.md#414-internal-infrastructure-capabilities) | i.cap.018 | Support for huge pages | Not required | Must support | [ra2.ch.001](chapter04.md#42-kubernetes-node)| -| [4.1.4](../../../ref_model/chapters/chapter04.chapter04.md#414-internal-infrastructure-capabilities) | i.pm.001 | Monitor worker node CPU usage, per nanosecond | Must support | Must support | **N/A** | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.pm.002 | Monitor pod CPU usage, per nanosecond | Must support | Must support | **N/A** | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.pm.003 | Monitor worker node CPU utilisation (%) | Must support | Must support | **N/A** | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.pm.004 | Monitor pod CPU utilisation | Must support | Must support | **N/A** | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.pm.005 | Measure external storage IOPs | Must support | Must support | **N/A** | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.pm.006 | Measure external storage throughput | Must support | Must support | **N/A** | -| [4.1.4](../../../ref_model/chapters/chapter04.md#414-internal-infrastructure-capabilities) | i.pm.007 | Measure external storage capacity | Must support | Must support | **N/A** | - -

Table 2-1: Reference Model Requirements: Cloud Infrastructure Software Profile Capabilities

- -**(1)** Defined in the `.bronze` configuration in section [4.2.6 Storage Extensions](../../../ref_model/chapters/chapter04.md#426-storage-extensions)
+| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.001 | Max number of vCPU that can be assigned to a single Pod by the Cloud Infrastructure | At least 16 | At least 16 | [ra2.ch.011](chapter04.md#kubernetes-node)| +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.002 | Max memory in MB that can be assigned to a single Pod by the Cloud Infrastructure | at least 32 GB | at least 32 GB | [ra2.ch.012](chapter04.md#kubernetes-node)| +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.003 | Max storage in GB that can be assigned to a single Pod by the Cloud Infrastructure | at least 320 GB | at least 320 GB | [ra2.ch.010](chapter04.md#kubernetes-node)| +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.004 | Max number of connection points that can be assigned to a single Pod by the Cloud Infrastructure | 6 | 6 | [ra2.ntw.003](chapter04.md#networking-solutions) | +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.005 | Max storage in GB that can be attached / mounted to Pod by the Cloud Infrastructure | Up to 16TB (1) | Up to 16TB (1) | N/A | +| [4.2.2](../../../ref_model/chapters/chapter04.md#profiles-specifications--capability-mapping) | e.cap.006 | CPU pinning support | Not required | Must support | [ra2.k8s.009](chapter04.md#kubernetes)| +| [4.2.2](../../../ref_model/chapters/chapter04.md#profiles-specifications--capability-mapping) | e.cap.007 | NUMA support | Not required | Must support | [ra2.k8s.006](chapter04.md#kubernetes)| +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.008 | IPSec Acceleration using the virtio-ipsec interface | Not required | Optional | N/A | +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.009 | Crypto Acceleration using the virtio-crypto interface | Not required | Optional | N/A | +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.010 | Transcoding Acceleration | Not required | Not required | N/A | +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.011 | Programmable Acceleration | Not required | Not required | N/A | +| [4.1.2](../../../ref_model/chapter04.md#exposed-infrastructure-capabilities) | e.cap.012 | Enhanced Cache Management: L=Lean; E=Equal; X=eXpanded | E | E | N/A | +| [4.2.2](../../../ref_model/chapters/chapter04.md#profiles-specifications--capability-mapping) | e.cap.013 | SR-IOV over PCI-PT | Not required | Must support | [ra2.ch.002](chapter04.md#kubernetes-node), [ra2.ch.003](chapter04.md#kubernetes-node), [ra2.k8s.007](chapter04.md#kubernetes), [ra2.ntw.004](chapter04.md#networking-solutions), [ra2.ntw.008](chapter04.md#networking-solutions)| +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.014 | Hardware coprocessor support (GPU/NPU) | Not required | Not required | N/A| +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.015 | SmartNICs | Not required | Optional | N/A | +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | e.cap.016 | FPGA/other Acceleration H/W | Not required | Optional | [ra2.k8s.007](chapter04.md#kubernetes), [ra2.ntw.012](chapter04.md#networking-solutions)| +| [4.1.2](../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities) | *e.cap.017* | *Ability to monitor L2-L7 data from workload* | *n/a (2)* | *n/a (2) * | N/A | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.cap.014 | Specifies the proportion of CPU cores consumed by the Cloud Infrastructure system on the worker nodes. If SMT is used, it indicates the number of consumed SMT threads. | 2 | 2 | [ra2.k8s.008](chapter04.md#kubernetes) | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.cap.015 | Indicates the memory consumed by Cloud Infrastructure on the worker nodes | 16 GB | 16GB | | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.cap.016 | Number of virtual cores per physical core; also known as CPU overbooking ratio that is required | 1:1 | 1:1 | [ra2.ch.004](chapter04.md#kubernetes-node), [ra2.ch.005](chapter04.md#kubernetes-node)| +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.cap.017 | QoS enablement of the connection point (vNIC or interface)| Not required | Must support | **N/A** | +| [4.1.4](../../../ref_model/chapters/chapter04.chapter04.md#internal-infrastructure-capabilities) | i.cap.018 | Support for huge pages | Not required | Must support | [ra2.ch.001](chapter04.md#kubernetes-node)| +| [4.1.4](../../../ref_model/chapters/chapter04.chapter04.md#internal-infrastructure-capabilities) | i.pm.001 | Monitor worker node CPU usage, per nanosecond | Must support | Must support | **N/A** | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.pm.002 | Monitor pod CPU usage, per nanosecond | Must support | Must support | **N/A** | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.pm.003 | Monitor worker node CPU utilisation (%) | Must support | Must support | **N/A** | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.pm.004 | Monitor pod CPU utilisation | Must support | Must support | **N/A** | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.pm.005 | Measure external storage IOPs | Must support | Must support | **N/A** | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.pm.006 | Measure external storage throughput | Must support | Must support | **N/A** | +| [4.1.4](../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities) | i.pm.007 | Measure external storage capacity | Must support | Must support | **N/A** | + +**Table 2-1:** Reference Model Requirements: Cloud Infrastructure Software Profile Capabilities + +**(1)** Defined in the `.bronze` configuration in section [4.2.6 Storage Extensions](../../../ref_model/chapters/chapter04.md#storage-extensions) + **(2)** In Kubernetes based infrastructures packet monitoring is out of the scope for the infrastructure. -### 2.2.2 Virtual Network Interface Specifications +### Virtual Network Interface Specifications The required number of connection points to a Pod is described in `e.cap.004` above. This section describes the required bandwidth of those connection points. | Reference Model Section | Reference | Description | Requirement for Basic Profile | Requirement for High-Performance Profile| Specification Reference | |---|---|---|---|---|---| -| [4.2.5](../../../ref_model/chapters/chapter04.md#425-virtual-network-interface-specifications) | n1, n2, n3, n4, n5, n6 | 1, 2, 3, 4, 5, 6 Gbps | Must support | Must support | **N/A** | -| [4.2.5](../../../ref_model/chapters/chapter04.md#425-virtual-network-interface-specifications) | nn10, n20, n30, n40, n50, n60 | 10, 20, 30, 40, 50, 60 Gbps | Must support | Must support | **N/A** | -| [4.2.5](../../../ref_model/chapters/chapter04.md#425-virtual-network-interface-specifications) | n25, n50, n75, n100, n125, n150 | 25, 50, 75, 100, 125, 150 Gbps | Must support | Must support | **N/A** | -| [4.2.5](../../../ref_model/chapters/chapter04.md#425-virtual-network-interface-specifications) | nn50, n100, n150, n200, n250, n300 | 50, 100, 150, 200, 250, 300 Gbps | Must support | Must support | **N/A** | -| [4.2.5](../../../ref_model/chapters/chapter04.md#425-virtual-network-interface-specifications) | n100, n200, n300, n400, n500, n600 | 100, 200, 300, 400, 500, 600 Gbps | Must support | Must support | **N/A** | +| [4.2.5](../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications) | n1, n2, n3, n4, n5, n6 | 1, 2, 3, 4, 5, 6 Gbps | Must support | Must support | **N/A** | +| [4.2.5](../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications) | nn10, n20, n30, n40, n50, n60 | 10, 20, 30, 40, 50, 60 Gbps | Must support | Must support | **N/A** | +| [4.2.5](../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications) | n25, n50, n75, n100, n125, n150 | 25, 50, 75, 100, 125, 150 Gbps | Must support | Must support | **N/A** | +| [4.2.5](../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications) | nn50, n100, n150, n200, n250, n300 | 50, 100, 150, 200, 250, 300 Gbps | Must support | Must support | **N/A** | +| [4.2.5](../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications) | n100, n200, n300, n400, n500, n600 | 100, 200, 300, 400, 500, 600 Gbps | Must support | Must support | **N/A** | -

Table 2-2: Reference Model Requirements: Network Interface Specifications

+**Table 2-2:** Reference Model Requirements: Network Interface Specifications -### 2.2.3 Cloud Infrastructure Software Profile Requirements +### Cloud Infrastructure Software Profile Requirements | Reference Model Section | Reference | Description | Requirement for Basic Profile | Requirement for High-Performance Profile| Specification Reference | |---|---|---|---|---|---| -| [5.1.1](../../../ref_model/chapters/chapter05.md#511-virtual-compute) | infra.com.cfg.001 | CPU allocation ratio | 1:1 | 1:1 | [ra2.ch.005](chapter04.md#42-kubernetes-node)
[ra2.ch.006](chapter04.md#42-kubernetes-node)| -| [5.1.1](../../../ref_model/chapters/chapter05.md#511-virtual-compute) | infra.com.cfg.002 | NUMA awareness | Not required | Must support |[ra2.k8s.006](chapter04.md#43-kubernetes)| -| [5.1.1](../../../ref_model/chapters/chapter05.md#511-virtual-compute) | infra.com.cfg.003 | CPU pinning capability | Not required | Must support |[ra2.k8s.009](chapter04.md#43-kubernetes)| -| [5.1.1](../../../ref_model/chapters/chapter05.md#511-virtual-compute) | infra.com.cfg.004 | Huge pages | Not required | Must support | [ra2.ch.001](chapter04.md#42-kubernetes-node)| -| [5.1.2](../../../ref_model/chapters/chapter05.md#512-virtual-storage) | infra.stg.cfg.002 | Storage Block | Must support | Must support | [ra2.stg.004](chapter04.md#46-storage-components)| -| [5.1.2](../../../ref_model/chapters/chapter05.md#512-virtual-storage) | infra.stg.cfg.003 | Storage with replication | Not required | Must support | **N/A** | -| [5.1.2](../../../ref_model/chapters/chapter05.md#512-virtual-storage) | infra.stg.cfg.004 | Storage with encryption | Must support | Must support | **N/A** | -| [5.1.2](../../../ref_model/chapters/chapter05.md#512-virtual-storage) | infra.stg.acc.cfg.001 | Storage IOPS oriented | Not required | Must support | **N/A** | -| [5.1.2](../../../ref_model/chapters/chapter05.md#512-virtual-storage) | infra.stg.acc.cfg.002 | Storage capacity oriented | Not required | Not required | N/A | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.cfg.001 | IO virtualisation using virtio1.1 | Must support(1) | Must support(1)| **N/A** | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.cfg.002 | The overlay network encapsulation protocol needs to enable ECMP in the underlay to take advantage of the scale-out features of the network fabric.(2) | Must support VXLAN, MPLSoUDP, GENEVE, other | *No requirement specified* | **N/A** | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.cfg.003 | Network Address Translation | Must support | Must support | **N/A** | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.cfg.004 | Security Groups | Must support | Must support | [ra2.k8s.014](chapter04.md#43-kubernetes) | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.cfg.005 | SFC support | Not required | Must support | **N/A** | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.cfg.006 | Traffic patterns symmetry | Must support | Must support | **N/A** | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.acc.cfg.001 | vSwitch optimisation | Not required | Must support DPDK(3)|[ra2.ntw.010](chapter04.md#45-networking-solutions)| -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.acc.cfg.002 | Support of HW offload | Not required | Optional, SmartNic | N/A | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.acc.cfg.003 | Crypto acceleration | Not required | Optional | N/A | -| [5.1.3](../../../ref_model/chapters/chapter05.md#513-virtual-networking) | infra.net.acc.cfg.004 | Crypto Acceleration Interface | Not required | Optional | N/A | - -

Table 2-3: Reference Model Requirements: Cloud Infrastructure Software Profile Requirements

- -**(1)** [Workload Transition Guidelines.](../chapters/appendix-a.md) might have other interfaces (such as SR-IOV VFs to be directly passed to a VM or a Pod) or NIC-specific drivers on guest machines transiently allowed until more mature solutions are available with an acceptable level of efficiency to support telecom workloads (for example regarding CPU and energy consumption).
-**(2)** In Kubernetes based infrastructures network separation is possible without an overlay (e.g.: with IPVLAN)
+| [5.1.1](../../../ref_model/chapters/chapter05.md#virtual-compute) | infra.com.cfg.001 | CPU allocation ratio | 1:1 | 1:1 | [ra2.ch.005](chapter04.md#kubernetes-node), [ra2.ch.006](chapter04.md#kubernetes-node)| +| [5.1.1](../../../ref_model/chapters/chapter05.md#virtual-compute) | infra.com.cfg.002 | NUMA awareness | Not required | Must support |[ra2.k8s.006](chapter04.md#kubernetes)| +| [5.1.1](../../../ref_model/chapters/chapter05.md#virtual-compute) | infra.com.cfg.003 | CPU pinning capability | Not required | Must support |[ra2.k8s.009](chapter04.md#kubernetes)| +| [5.1.1](../../../ref_model/chapters/chapter05.md#virtual-compute) | infra.com.cfg.004 | Huge pages | Not required | Must support | [ra2.ch.001](chapter04.md#kubernetes-node)| +| [5.1.2](../../../ref_model/chapters/chapter05.md#virtual-storage) | infra.stg.cfg.002 | Storage Block | Must support | Must support | [ra2.stg.004](chapter04.md#storage-components)| +| [5.1.2](../../../ref_model/chapters/chapter05.md#virtual-storage) | infra.stg.cfg.003 | Storage with replication | Not required | Must support | **N/A** | +| [5.1.2](../../../ref_model/chapters/chapter05.md#virtual-storage) | infra.stg.cfg.004 | Storage with encryption | Must support | Must support | **N/A** | +| [5.1.2](../../../ref_model/chapters/chapter05.md#virtual-storage) | infra.stg.acc.cfg.001 | Storage IOPS oriented | Not required | Must support | **N/A** | +| [5.1.2](../../../ref_model/chapters/chapter05.md#virtual-storage) | infra.stg.acc.cfg.002 | Storage capacity oriented | Not required | Not required | N/A | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.cfg.001 | IO virtualisation using virtio1.1 | Must support (1) | Must support (1) | **N/A** | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.cfg.002 | The overlay network encapsulation protocol needs to enable ECMP in the underlay to take advantage of the scale-out features of the network fabric. (2) | Must support VXLAN, MPLSoUDP, GENEVE, other | *No requirement specified* | **N/A** | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.cfg.003 | Network Address Translation | Must support | Must support | **N/A** | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.cfg.004 | Security Groups | Must support | Must support | [ra2.k8s.014](chapter04.md#kubernetes) | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.cfg.005 | SFC support | Not required | Must support | **N/A** | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.cfg.006 | Traffic patterns symmetry | Must support | Must support | **N/A** | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.acc.cfg.001 | vSwitch optimisation | Not required | Must support DPDK (3) |[ra2.ntw.010](chapter04.md#networking-solutions)| +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.acc.cfg.002 | Support of HW offload | Not required | Optional, SmartNic | N/A | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.acc.cfg.003 | Crypto acceleration | Not required | Optional | N/A | +| [5.1.3](../../../ref_model/chapters/chapter05.md#virtual-networking) | infra.net.acc.cfg.004 | Crypto Acceleration Interface | Not required | Optional | N/A | + +**Table 2-3:** Reference Model Requirements: Cloud Infrastructure Software Profile Requirements + +**(1)** [Workload Transition Guidelines.](../chapters/appendix-a.md) might have other interfaces (such as SR-IOV VFs to be directly passed to a VM or a Pod) or NIC-specific drivers on guest machines transiently allowed until more mature solutions are available with an acceptable level of efficiency to support telecom workloads (for example regarding CPU and energy consumption). + +**(2)** In Kubernetes based infrastructures network separation is possible without an overlay (e.g.: with IPVLAN) + **(3)** This feature is not applicable for Kubernetes based infrastructures due to lack of vSwitch however workloads need access to user space networking solutions. -### 2.2.4 Cloud Infrastructure Hardware Profile Requirements +### Cloud Infrastructure Hardware Profile Requirements | Reference Model Section | Reference | Description | Requirement for Basic Profile | Requirement for High-Performance Profile| Specification Reference | |---|---|---|---|---|---| -| [5.4.1](../../../ref_model/chapters/chapter05.md#541-compute-resources) | infra.hw.cpu.cfg.001 | Minimum number of CPU sockets | 2 | 2 |[ra2.ch.008](chapter04.md#42-kubernetes-node)| -| [5.4.1](../../../ref_model/chapters/chapter05.md#541-compute-resources) | infra.hw.cpu.cfg.002 | Minimum number of Cores per CPU | 20 | 20 |[ra2.ch.008](chapter04.md#42-kubernetes-node)| -| [5.4.1](../../../ref_model/chapters/chapter05.md#541-compute-resources) | infra.hw.cpu.cfg.003 | NUMA | Not required | Must support |[ra2.k8s.006](chapter04.md#43-kubernetes)| -| [5.4.1](../../../ref_model/chapters/chapter05.md#541-compute-resources) | infra.hw.cpu.cfg.004 | Simultaneous Multithreading/Symmetric Multiprocessing (SMT/SMP) | Must support | Optional |[ra2.ch.004](chapter04.md#42-kubernetes-node)| -| [5.4.1](../../../ref_model/chapters/chapter05.md#541-compute-resources) | infra.hw.cac.cfg.001 | GPU | Not required | Optional | N/A | -| [5.4.2](../../../ref_model/chapters/chapter05.md#542-storage-configurations) | infra.hw.stg.hdd.cfg.001 | Local Storage HDD | *No requirement specified* | *No requirement specified* | N/A | -| [5.4.2](../../../ref_model/chapters/chapter05.md#542-storage-configurations) | infra.hw.stg.ssd.cfg.002 | Local Storage SSD | Should support | Should support |[ra2.ch.009](chapter04.md#42-kubernetes-node)| -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.nic.cfg.001 | Total Number of NIC Ports available in the host | 4 | 4 |[ra2.ch.013](chapter04.md#42-kubernetes-node)| -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.nic.cfg.002 | Port speed specified in Gbps (minimum values) | 10 | 25 |[ra2.ch.014](chapter04.md#42-kubernetes-node)
[ra2.ch.015](chapter04.md#42-kubernetes-node)| -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.pci.cfg.001 | Number of PCIe slots available in the host | 8 | 8 | [ra2.ch.016](chapter04.md#42-kubernetes-node) | -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.pci.cfg.002 | PCIe speed | Gen 3 | Gen 3 |[ra2.ch.016](chapter04.md#42-kubernetes-node) | -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.pci.cfg.003 | PCIe Lanes | 8 | 8 |[ra2.ch.016](chapter04.md#42-kubernetes-node) | -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.nac.cfg.001 | Cryptographic Acceleration | Not required | Optional | N/A | -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.nac.cfg.002 | A SmartNIC that is used to offload vSwitch functionality to hardware | Not required | Optional(1) | N/A | -| [5.4.3](../../../ref_model/chapters/chapter05.md#543-network-resources) | infra.hw.nac.cfg.003 | Compression | Optional | Optional | N/A | - -

Table 2-4: Reference Model Requirements: Cloud Infrastructure Hardware Profile Requirements

+| [5.4.1](../../../ref_model/chapters/chapter05.md#compute-resources) | infra.hw.cpu.cfg.001 | Minimum number of CPU sockets | 2 | 2 |[ra2.ch.008](chapter04.md#kubernetes-node)| +| [5.4.1](../../../ref_model/chapters/chapter05.md#compute-resources) | infra.hw.cpu.cfg.002 | Minimum number of Cores per CPU | 20 | 20 |[ra2.ch.008](chapter04.md#kubernetes-node)| +| [5.4.1](../../../ref_model/chapters/chapter05.md#compute-resources) | infra.hw.cpu.cfg.003 | NUMA | Not required | Must support |[ra2.k8s.006](chapter04.md#kubernetes)| +| [5.4.1](../../../ref_model/chapters/chapter05.md#compute-resources) | infra.hw.cpu.cfg.004 | Simultaneous Multithreading/Symmetric Multiprocessing (SMT/SMP) | Must support | Optional |[ra2.ch.004](chapter04.md#kubernetes-node)| +| [5.4.1](../../../ref_model/chapters/chapter05.md#compute-resources) | infra.hw.cac.cfg.001 | GPU | Not required | Optional | N/A | +| [5.4.2](../../../ref_model/chapters/chapter05.md#storage-configurations) | infra.hw.stg.hdd.cfg.001 | Local Storage HDD | *No requirement specified* | *No requirement specified* | N/A | +| [5.4.2](../../../ref_model/chapters/chapter05.md#storage-configurations) | infra.hw.stg.ssd.cfg.002 | Local Storage SSD | Should support | Should support |[ra2.ch.009](chapter04.md#kubernetes-node)| +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.nic.cfg.001 | Total Number of NIC Ports available in the host | 4 | 4 |[ra2.ch.013](chapter04.md#kubernetes-node)| +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.nic.cfg.002 | Port speed specified in Gbps (minimum values) | 10 | 25 |[ra2.ch.014](chapter04.md#kubernetes-node), [ra2.ch.015](chapter04.md#kubernetes-node)| +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.pci.cfg.001 | Number of PCIe slots available in the host | 8 | 8 | [ra2.ch.016](chapter04.md#kubernetes-node) | +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.pci.cfg.002 | PCIe speed | Gen 3 | Gen 3 |[ra2.ch.016](chapter04.md#kubernetes-node) | +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.pci.cfg.003 | PCIe Lanes | 8 | 8 |[ra2.ch.016](chapter04.md#kubernetes-node) | +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.nac.cfg.001 | Cryptographic Acceleration | Not required | Optional | N/A | +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.nac.cfg.002 | A SmartNIC that is used to offload vSwitch functionality to hardware | Not required | Optional (1) | N/A | +| [5.4.3](../../../ref_model/chapters/chapter05.md#network-resources) | infra.hw.nac.cfg.003 | Compression | Optional | Optional | N/A | + +**Table 2-4:** Reference Model Requirements: Cloud Infrastructure Hardware Profile Requirements **(1)** There is no vSwitch in case of containers, but a SmartNIC can be used to offload any other network processing. -### 2.2.5 Cloud Infrastructure Management Requirements +### Cloud Infrastructure Management Requirements | Reference Model Section | Reference | Description | Requirement (common to all Profiles) | Specification Reference | |---|---|---|---|---| -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.001 | Capability to allocate virtual compute resources to a workload | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.002 | Capability to allocate virtual storage resources to a workload | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.003 | Capability to allocate virtual networking resources to a workload | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.004 | Capability to isolate resources between tenants | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.005 | Capability to manage workload software images | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.006 | Capability to provide information related to allocated virtualised resources per tenant | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.007 | Capability to notify state changes of allocated resources | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.008 | Capability to collect and expose performance information on virtualised resources allocated | Must support | **N/A** | -| [4.1.5](../../../ref_model/chapters/chapter04.md#415-cloud-infrastructure-management-capabilities) | e.man.009 | Capability to collect and notify fault information on virtualised resources | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.001 | Capability to allocate virtual compute resources to a workload | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.002 | Capability to allocate virtual storage resources to a workload | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.003 | Capability to allocate virtual networking resources to a workload | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.004 | Capability to isolate resources between tenants | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.005 | Capability to manage workload software images | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.006 | Capability to provide information related to allocated virtualised resources per tenant | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.007 | Capability to notify state changes of allocated resources | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.008 | Capability to collect and expose performance information on virtualised resources allocated | Must support | **N/A** | +| [4.1.5](../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities) | e.man.009 | Capability to collect and notify fault information on virtualised resources | Must support | **N/A** | -

Table 2-5: Reference Model Requirements: Cloud Infrastructure Management Requirements

+**Table 2-5:** Reference Model Requirements: Cloud Infrastructure Management Requirements -### 2.2.6 Cloud Infrastructure Security Requirements +### Cloud Infrastructure Security Requirements | Reference Model Section | Reference | Requirement (common to all Profiles) | Specification Reference | |---|---|---|---| -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.001 | The Platform **must** maintain the specified configuration. | | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.002 | All systems part of Cloud Infrastructure **must** support password hardening as defined in [CIS Password Policy Guide](https://www.cisecurity.org/white-papers/cis-password-policy-guide/). Hardening: CIS Password Policy Guide | [5.3.1 Node Hardening: Securing Kubernetes Hosts]( ./chapter05.md#531-node-hardening-securing-kubernetes-hosts) | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.003 | All servers part of Cloud Infrastructure **must** support a root of trust and secure boot. | | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.004 | The Operating Systems of all the servers part of Cloud Infrastructure **must** be hardened by removing or disabling unnecessary services, applications and network protocols, configuring operating system user authentication, configuring resource controls, installing and configuring additional security controls where needed, and testing the security of the Operating System. (NIST SP 800-123)| [5.2 Principles](./chapter05.md#52-principles) and [5.3 Node Hardening](./chapter05.md#53-node-hardening) | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.005 | The Platform **must** support Operating System level access control | [5.3 Node Hardening](./chapter05.md#53-node-hardening) | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.006 | The Platform **must** support Secure logging. Logging with root account must be prohibited when root privileges are not required. | [5.3.2 Restrict direct access to nodes](./chapter05.md#532-restrict-direct-access-to-nodes) | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.007 | All servers part of Cloud Infrastructure **must** be Time synchronized with authenticated Time service. | | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.008 | All servers part of Cloud Infrastructure **must** be regularly updated to address security vulnerabilities. | [5.3.3 Vulnerability assessment](./chapter05.md#533-vulnerability-assessment) | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.009 | The Platform **must** support Software integrity protection and verification and **must** scan source code and manifests. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.010 | The Cloud Infrastructure **must** support encrypted storage, for example, block, object and file storage, with access to encryption keys restricted based on a need to know. [Controlled Access Based on the Need to Know](https://www.cisecurity.org/controls/controlled-access-based-on-the-need-to-know/) | | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.011 | The Cloud Infrastructure **should** support Read and Write only storage partitions (write only permission to one or more authorized actors). | | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.012 | The Operator **must** ensure that only authorized actors have physical access to the underlying infrastructure. | | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.013 | The Platform **must** ensure that only authorized actors have logical access to the underlying infrastructure. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.014 | All servers part of Cloud Infrastructure **should** support measured boot and an attestation server that monitors the measurements of the servers. | | -| [7.9.1](../../../ref_model/chapters/chapter07.md#791-system-hardening) | sec.gen.015 | Any change to the Platform must be logged as a security event, and the logged event must include the identity of the entity making the change, the change, the date and the time of the change. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.001 | The Platform **must** support authenticated and secure access to API, GUI and command line interfaces. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.002 | The Platform **must** support Traffic Filtering for workloads (for example, Firewall). | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.003 | The Platform **must** support Secure and encrypted communications, and confidentiality and integrity of network traffic. | [5.4.3 Use Transport Layer Security and Service Mesh](./chapter05.md#543-use-transport-layer-security-and-service-mesh) | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.004 | The Cloud Infrastructure **must** support authentication, integrity and confidentiality on all network channels. | [5.4.3 Use Transport Layer Security and Service Mesh](./chapter05.md#543-use-transport-layer-security-and-service-mesh) | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.005 | The Cloud Infrastructure **must** segregate the underlay and overlay networks. | | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.006 | The Cloud Infrastructure must be able to utilise the Cloud Infrastructure Manager identity lifecycle management capabilities. | [5.2 Principles](./chapter05.md#52-principles) | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.007 | The Platform **must** implement controls enforcing separation of duties and privileges, least privilege use and least common mechanism (Role-Based Access Control). | [5.2 Principles](./chapter05.md#52-principles) and [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.008 | The Platform **must** be able to assign the Entities that comprise the tenant networks to different trust domains. Communication between different trust domains is not allowed, by default. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.009 | The Platform **must** support creation of Trust Relationships between trust domains. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.010 | For two or more domains without existing trust relationships, the Platform **must not** allow the effect of an attack on one domain to impact the other domains either directly or indirectly. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.011 | The Platform **must not** reuse the same authentication credential (e.g., key-pair) on different Platform components (e.g., on different hosts, or different services). | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.012 | The Platform **must** protect all secrets by using strong encryption techniques, and storing the protected secrets externally from the component | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.013 | The Platform **must** provide secrets dynamically as and when needed. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.014 | The Platform **should** use Linux Security Modules such as SELinux to control access to resources. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.015 | The Platform **must not** contain back door entries (unpublished access points, APIs, etc.). | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.016 | Login access to the platform's components **must** be through encrypted protocols such as SSH v2 or TLS v1.2 or higher. Note: Hardened jump servers isolated from external networks are recommended | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.017 | The Platform **must** provide the capability of using digital certificates that comply with X.509 standards issued by a trusted Certification Authority. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.018 | The Platform **must** provide the capability of allowing certificate renewal and revocation. | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.019 | The Platform **must** provide the capability of testing the validity of a digital certificate (CA signature, validity period, non revocation, identity). | | -| [7.9.2](../../../ref_model/chapters/chapter07.md#792-platform-and-access) | sec.sys.020 | The Cloud Infrastructure architecture **should** rely on Zero Trust principles to build a secure by design environment. | | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.001 | The Platform **must** support Confidentiality and Integrity of data at rest and in-transit. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.002 | The Platform **should** support self-encrypting storage devices. | | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.003 | The Platform **must** support Confidentiality and Integrity of data related metadata. | | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.004 | The Platform **must** support Confidentiality of processes and restrict information sharing with only the process owner (e.g., tenant). | | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.005 | The Platform **must** support Confidentiality and Integrity of process-related metadata and restrict information sharing with only the process owner (e.g., tenant). | | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.006 | The Platform **must** support Confidentiality and Integrity of workload resource utilization (RAM, CPU, Storage, Network I/O, cache, hardware offload) and restrict information sharing with only the workload owner (e.g., tenant). | | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.007 | The Platform **must not** allow Memory Inspection by any actor other than the authorized actors for the Entity to which Memory is assigned (e.g., tenants owning the workload), for Lawful Inspection, and by secure monitoring services. | | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.008 | The Cloud Infrastructure **must** support tenant networks segregation. | [5.7 Create and define Network Policies](./chapter05.md#57-create-and-define-network-policies) | -| [7.9.3](../../../ref_model/chapters/chapter07.md#793-confidentiality-and-integrity) | sec.ci.009 | For sensitive data encryption, the key management service **should** leverage a Hardware Security Module to manage and protect cryptographic keys. | | -| [7.9.4](../../../ref_model/chapters/chapter07.md#794-workload-security) | sec.wl.001 | The Platform **must** support Workload placement policy. | | -| [7.9.4](../../../ref_model/chapters/chapter07.md#794-workload-security) | sec.wl.002 | The Cloud Infrastructure **must** provide methods to ensure the platform’s trust status and integrity (e.g. remote attestation, Trusted Platform Module). | | -| [7.9.4](../../../ref_model/chapters/chapter07.md#794-workload-security) | sec.wl.003 | The Platform **must** support secure provisioning of workloads. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.4](../../../ref_model/chapters/chapter07.md#794-workload-security) | sec.wl.004 | The Platform **must** support Location assertion (for mandated in-country or location requirements). | | -| [7.9.4](../../../ref_model/chapters/chapter07.md#794-workload-security) | sec.wl.005 | The Platform **must** support the separation of production and non-production Workloads. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.4](../../../ref_model/chapters/chapter07.md#794-workload-security) | sec.wl.006 | The Platform **must** support the separation of Workloads based on their categorisation (for example, payment card information, healthcare, etc.). | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) and [5.6 Separate Sensitive Workload](./chapter05.md#56-separate-sensitive-workload) | -| [7.9.4](../../../ref_model/chapters/chapter07.md#794-workload-security) | sec.wl.007 | The Operator **must** implement processes and tools to verify VNF authenticity and integrity. | [5.13 Trusted Registry](./chapter05.md#513--trusted-registry) | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.001 | Images from untrusted sources **must not** be used. | [5.13 Trusted Registry](./chapter05.md#513--trusted-registry) | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.002 | Images **must** be scanned to be maintained free from known vulnerabilities. | [5.13 Trusted Registry](./chapter05.md#513--trusted-registry) | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.003 | Images **must not** be configured to run with privileges higher than the privileges of the actor authorized to run them. | [5.11 Run-Time Security](./chapter05.md#511--run-time-security) | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.004 | Images **must** only be accessible to authorized actors. | | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.005 | Image Registries **must** only be accessible to authorized actors. | | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.006 | Image Registries **must** only be accessible over secure networks that enforce authentication, integrity and confidentiality. | [5.13 Trusted Registry](./chapter05.md#513--trusted-registry) | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.007 | Image registries **must** be clear of vulnerable and out of date versions. | [5.13 Trusted Registry](./chapter05.md#513--trusted-registry) | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.008 | Images **must not** include any secrets. Secrets include passwords, cloud provider credentials, SSH keys, TLS certificate keys, etc. | [5.12 Secrets Management](./chapter05.md#512--secrets-management) | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.009 | CIS Hardened Images **should** be used whenever possible. | | -| [7.9.5](../../../ref_model/chapters/chapter07.md#795-image-security) | sec.img.010 | Minimalist base images **should** be used whenever possible. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.001 | The Platform **must** support Secure Provisioning, Availability, and Deprovisioning (Secure Clean-Up) of workload resources where Secure Clean-Up includes tear-down, defense against virus or other attacks. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.002 | Cloud operations staff and systems **must** use management protocols limiting security risk such as SNMPv3, SSH v2, ICMP, NTP, syslog and TLS v1.2 or higher. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.003 | The Cloud Operator **must** implement and strictly follow change management processes for Cloud Infrastructure, Cloud Infrastructure Manager and other components of the cloud, and Platform change control on hardware. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.004 | The Cloud Operator **should** support automated templated approved changes. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.005 | Platform **must** provide logs and these logs must be regularly monitored for anomalous behavior. | [5.10 Enable Logging and Monitoring](./chapter05.md#510--enable-logging-and-monitoring) | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.006 | The Platform **must** verify the integrity of all Resource management requests. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.007 | The Platform **must** be able to update newly instantiated, suspended, hibernated, migrated and restarted images with current time information. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#54-securing-kubernetes-orchestrator) | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.008 | The Platform **must** be able to update newly instantiated, suspended, hibernated, migrated and restarted images with relevant DNS information. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.009 | The Platform **must** be able to update the tag of newly instantiated, suspended, hibernated, migrated and restarted images with relevant geolocation (geographical) information. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.010 | The Platform **must** log all changes to geolocation along with the mechanisms and sources of location information (i.e. GPS, IP block, and timing). | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.011 | The Platform **must** implement Security life cycle management processes including the proactive update and patching of all deployed Cloud Infrastructure software. | | -| [7.9.6](../../../ref_model/chapters/chapter07.md#796-security-lcm) | sec.lcm.012 | The Platform **must** log any access privilege escalation. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.001 | Platform **must** provide logs and these logs must be regularly monitored for events of interest. The logs **must** contain the following fields: event type, date/time, protocol, service or program used for access, success/failure, login ID or process ID, IP address and ports (source and destination) involved. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.002 | Security logs **must** be time synchronised. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.003 | The Platform **must** log all changes to time server source, time, date and time zones. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.004 | The Platform **must** secure and protect Audit logs (containing sensitive information) both in-transit and at rest. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.005 | The Platform **must** Monitor and Audit various behaviours of connection and login attempts to detect access attacks and potential access attempts and take corrective actions accordingly. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.006 | The Platform **must** Monitor and Audit operations by authorized account access after login to detect malicious operational activity and take corrective actions accordingly. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.007 | The Platform **must** Monitor and Audit security parameter configurations for compliance with defined security policies. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.008 | The Platform **must** Monitor and Audit externally exposed interfaces for illegal access (attacks) and take corrective security hardening measures. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.009 | The Platform **must** Monitor and Audit service handling for various attacks (malformed messages, signalling flooding and replaying, etc.) and take corrective actions accordingly. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.010 | The Platform **must** Monitor and Audit running processes to detect unexpected or unauthorized processes and take corrective actions accordingly. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.011 | The Platform **must** Monitor and Audit logs from infrastructure elements and workloads to detected anomalies in the system components and take corrective actions accordingly. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.012 | The Platform **must** Monitor and Audit Traffic patterns and volumes to prevent malware download attempts. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.013 | The monitoring system **must not** affect the security (integrity and confidentiality) of the infrastructure, workloads, or the user data (through back door entries). | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.014 | The Monitoring systems **should not** impact IAAS, PAAS, and SAAS SLAs including availability SLAs. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.015 | The Platform **must** ensure that the Monitoring systems are never starved of resources and **must** activate alarms when resource utilisation exceeds a configurable threshold. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.016 | The Platform Monitoring components **should** follow security best practices for auditing, including secure logging and tracing. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.017 | The Platform **must** audit systems for any missing security patches and take appropriate actions. | [5.3.3 Vulnerability assessment](./chapter05.md#533-vulnerability-assessment) | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.018 | The Platform, starting from initialization, **must** collect and analyze logs to identify security events, and store these events in an external system. | [5.3.4 Patch management](./chapter05.md#534-patch-management) | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.019 | The Platform’s components **must not** include an authentication credential, e.g., password, in any logs, even if encrypted. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.020 | The Platform’s logging system **must** support the storage of security audit logs for a configurable period of time. | | -| [7.9.7](../../../ref_model/chapters/chapter07.md#797-monitoring-and-security-audit) | sec.mon.021 |The Platform **must** store security events locally if the external logging system is unavailable and shall periodically attempt to send these to the external logging system until successful. | | -| [7.9.8](../../../ref_model/chapters/chapter07.md#798-open-source-software) | sec.oss.001 | Open source code **must** be inspected by tools with various capabilities for static and dynamic code analysis. | [5.3.3 Vulnerability assessment](./chapter05.md#533-vulnerability-assessment) | -| [7.9.8](../../../ref_model/chapters/chapter07.md#798-open-source-software) | sec.oss.002 | The [CVE (Common Vulnerabilities and Exposures)]( https://cve.mitre.org/) **must** be used to identify vulnerabilities and their severity rating for open source code part of Cloud Infrastructure and workloads software. | | -| [7.9.8](../../../ref_model/chapters/chapter07.md#798-open-source-software) | sec.oss.003 | Critical and high severity rated vulnerabilities **must** be fixed in a timely manner. Refer to the [CVSS (Common Vulnerability Scoring System)](https://www.first.org/cvss/) to know a vulnerability score and its associated rate (low, medium, high, or critical). | | -| [7.9.8](../../../ref_model/chapters/chapter07.md#798-open-source-software) | sec.oss.004 | A dedicated internal isolated repository separated from the production environment **must** be used to store vetted open source content. | [5.13 Trusted Registry](./chapter05.md#513--trusted-registry) | -| [7.9.8](../../../ref_model/chapters/chapter07.md#798-open-source-software) | sec.oss.005 | A Software Bill of Materials ([SBOM](https://www.ntia.gov/SBOM)) **should** be provided or build, and maintained to identify the software components and their origins. | | -| [7.9.9](../../../ref_model/chapters/chapter07.md#799-iaac---secure-design-and-architecture-stage-requirements) | sec.arch.001 | Threat Modelling methodologies and tools **should** be used during the Secure Design and Architecture stage triggered by Software Feature Design trigger. It may be done manually or using tools like open source OWASP Threat Dragon | | -| [7.9.9](../../../ref_model/chapters/chapter07.md#799-iaac---secure-design-and-architecture-stage-requirements) | sec.arch.002 | Security Control Baseline Assessment **should** be performed during the Secure Design and Architecture stage triggered by Software Feature Design trigger. Typically done manually by internal or independent assessors. | | -| [7.9.10](../../../ref_model/chapters/chapter07.md#7910-iaac---secure-code-stage-requirements) | sec.code.001 | SAST -Static Application Security Testing **must** be applied during Secure Coding stage triggered by Pull, Clone or Comment trigger. Security testing that analyses application source code for software vulnerabilities and gaps against best practices. Example: open source OWASP range of tools. | | -| [7.9.10](../../../ref_model/chapters/chapter07.md#7910-iaac---secure-code-stage-requirements) | sec.code.002 | SCA – Software Composition Analysis **should** be applied during Secure Coding stage triggered by Pull, Clone or Comment trigger. Security testing that analyses application source code or compiled code for software components with known vulnerabilities. Example: open source OWASP range of tools. | | -| [7.9.10](../../../ref_model/chapters/chapter07.md#7910-iaac---secure-code-stage-requirements) | sec.code.003 | Source Code Review **should** be performed continuously during Secure Coding stage. Typically done manually. | | -| [7.9.10](../../../ref_model/chapters/chapter07.md#7910-iaac---secure-code-stage-requirements) | sec.code.004 | Integrated SAST via IDE Plugins **should** be used during Secure Coding stage triggered by Developer Code trigger. On the local machine: through the IDE or integrated test suites; triggered on completion of coding be developer. | | -| [7.9.10](../../../ref_model/chapters/chapter07.md#7910-iaac---secure-code-stage-requirements) | sec.code.005 | SAST of Source Code Repo **should** be performed during Secure Coding stage triggered by Developer Code trigger. Continuous delivery pre-deployment: scanning prior to deployment. | | -| [7.9.11](../../../ref_model/chapters/chapter07.md#7911-iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.001 | SAST -Static Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Build and Integrate trigger. Example: open source OWASP range of tools. | | -| [7.9.11](../../../ref_model/chapters/chapter07.md#7911-iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.002 | SCA – Software Composition Analysis **should** be applied during the Continuous Build, Integration and Testing stage triggered by Build and Integrate trigger. Example: open source OWASP range of tools. | | -| [7.9.11](../../../ref_model/chapters/chapter07.md#7911-iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.003 | Image Scan **must** be applied during the Continuous Build, Integration and Testing stage triggered by Package trigger. Example: A push of a container image to a container registry may trigger a vulnerability scan before the image becomes available in the registry. | | -| [7.9.11](../../../ref_model/chapters/chapter07.md#7911-iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.004 | DAST – Dynamic Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Stage & Test trigger. Security testing that analyses a running application by exercising application functionality and detecting vulnerabilities based on application behaviour and response. Example: OWASP ZAP. | | -| [7.9.11](../../../ref_model/chapters/chapter07.md#7911-iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.005 | Fuzzing **should** be applied during the Continuous Build, Integration and testing stage triggered by Stage & Test trigger. Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. Example: GitLab Open Sources Protocol Fuzzer Community Edition. | | -| [7.9.11](../../../ref_model/chapters/chapter07.md#7911-iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.006 | IAST – Interactive Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Stage & Test trigger. Software component deployed with an application that assesses application behaviour and detects presence of vulnerabilities on an application being exercised in realistic testing scenarios. Example: Contrast Community Edition. | | -| [7.9.12](../../../ref_model/chapters/chapter07.md#7912-iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.001 | Image Scan **must** be applied during the Continuous Delivery and Deployment stage triggered by Publish to Artifact and Image Repository trigger. Example: GitLab uses the open-source Clair engine for container image scanning.| | -| [7.9.12](../../../ref_model/chapters/chapter07.md#7912-iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.002 | Code Signing **must** be applied during the Continuous Delivery and Deployment stage triggered by Publish to Artifact and Image Repository trigger. Code Signing provides authentication to assure that downloaded files are form the publisher named on the certificate. | | -| [7.9.12](../../../ref_model/chapters/chapter07.md#7912-iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.003 | Artifact and Image Repository Scan **should** be continuously applied during the Continuous Delivery and Deployment stage. Example: GitLab uses the open source Clair engine for container scanning. | | -| [7.9.12](../../../ref_model/chapters/chapter07.md#7912-iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.004 | Component Vulnerability Scan **must** be applied during the Continuous Delivery and Deployment stage triggered by Instantiate Infrastructure trigger. The vulnerability scanning system is deployed on the cloud platform to detect security vulnerabilities of specified components through scanning and to provide timely security protection. Example: OWASP Zed Attack Proxy (ZAP). | | -| [7.9.13](../../../ref_model/chapters/chapter07.md#7913-iaac---runtime-defence-and-monitoring-requirements) | sec.run.001 | Component Vulnerability Monitoring **must** be continuously applied during the Runtime Defence and Monitoring stage and remediation actions **must** be applied for high severity rated vulnerabilities. Security technology that monitors components like virtual servers and assesses data, applications, and infrastructure for security risks. | | -| [7.9.13](../../../ref_model/chapters/chapter07.md#7913-iaac---runtime-defence-and-monitoring-requirements) | sec.run.002 | RASP – Runtime Application Self-Protection **should** be continuously applied during the Runtime Defence and Monitoring stage. Security technology deployed within the target application in production for detecting, alerting, and blocking attacks. | | -| [7.9.13](../../../ref_model/chapters/chapter07.md#7913-iaac---runtime-defence-and-monitoring-requirements) | sec.run.003 | Application testing and Fuzzing **should** be continuously applied during the Runtime Defence and Monitoring stage. Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. Example: GitLab Open Sources Protocol Fuzzer Community Edition. | | -| [7.9.13](../../../ref_model/chapters/chapter07.md#7913-iaac---runtime-defence-and-monitoring-requirements) | sec.run.004 | Penetration Testing **should** be continuously applied during the Runtime Defence and Monitoring stage. Typically done manually. | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.001 | The Cloud Operator **should** comply with Center for Internet Security CIS Controls ([https://www.cisecurity.org/](https://www.cisecurity.org/)) | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.002 | The Cloud Operator, Platform and Workloads **should** follow the guidance in the CSA Security Guidance for Critical Areas of Focus in Cloud Computing (latest version) [https://cloudsecurityalliance.org/](https://cloudsecurityalliance.org/) | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.003 | The Platform and Workloads **should** follow the guidance in the [OWASP Cheat Sheet Series (OCSS)](https://github.com/OWASP/CheatSheetSeries) | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.004 | The Cloud Operator, Platform and Workloads **should** ensure that their code is not vulnerable to the OWASP Top Ten Security Risks [https://owasp.org/www-project-top-ten/](https://owasp.org/www-project-top-ten/) | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.005 | The Cloud Operator, Platform and Workloads **should** strive to improve their maturity on the [OWASP Software Maturity Model (SAMM)](https://owaspsamm.org/blog/2019/12/20/version2-community-release/) | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.006 | The Cloud Operator, Platform and Workloads **should** utilize the [OWASP Web Security Testing Guide](https://github.com/OWASP/wstg/tree/master/document) | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.007 | The Cloud Operator, and Platform **should** satisfy the requirements for Information Management Systems specified in [ISO/IEC 27001](https://www.iso.org/obp/ui/#iso:std:iso-iec:27001:ed-2:v1:en). ISO/IEC 27002:2013 - ISO/IEC 27001 is the international Standard for best-practice information security management systems (ISMSs). | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.008 | The Cloud Operator, and Platform **should** implement the Code of practice for Security Controls specified [ISO/IEC 27002:2013 (or latest)](https://www.iso.org/obp/ui/#iso:std:iso-iec:27002:ed-2:v1:en) | | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.009 | The Cloud Operator, and Platform **should** implement the [ISO/IEC 27032:2012 (or latest)](https://www.iso.org/obp/ui/#iso:std:iso-iec:27032:ed-1:v1:en) Guidelines for Cybersecurity techniques. ISO/IEC 27032 - ISO/IEC 27032 is the international Standard focusing explicitly on cybersecurity. | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.010 | The Cloud Operator **should** conform to the ISO/IEC 27035 standard for incidence management. ISO/IEC 27035 - ISO/IEC 27035 is the international Standard for incident management. | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.011 | The Cloud Operator **should** conform to the ISO/IEC 27031 standard for business continuity. ISO/IEC 27031 - ISO/IEC 27031 is the international Standard for ICT readiness for business continuity. | | -| [7.9.14](../../../ref_model/chapters/chapter07.md#7914-compliance-with-standards) | sec.std.012 | The Public Cloud Operator **must**, and the Private Cloud Operator **may** be certified to be compliant with the International Standard on Awareness Engagements (ISAE) 3402 (in the US: SSAE 16). International Standard on Awareness Engagements (ISAE) 3402. US Equivalent: SSAE16. | | - -

Table 2-6: Reference Model Requirements: Cloud Infrastructure Security Requirements

- -## 2.3 Kubernetes Architecture Requirements +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.001 | The Platform **must** maintain the specified configuration. | | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.002 | All systems part of Cloud Infrastructure **must** support password hardening as defined in [CIS Password Policy Guide](https://www.cisecurity.org/white-papers/cis-password-policy-guide/). Hardening: CIS Password Policy Guide | [5.3.1 Node Hardening: Securing Kubernetes Hosts]( ./chapter05.md#node-hardening-securing-kubernetes-hosts) | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.003 | All servers part of Cloud Infrastructure **must** support a root of trust and secure boot. | | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.004 | The Operating Systems of all the servers part of Cloud Infrastructure **must** be hardened by removing or disabling unnecessary services, applications and network protocols, configuring operating system user authentication, configuring resource controls, installing and configuring additional security controls where needed, and testing the security of the Operating System. (NIST SP 800-123)| [5.2 Principles](./chapter05.md#principles) and [5.3 Node Hardening](./chapter05.md#node-hardening) | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.005 | The Platform **must** support Operating System level access control | [5.3 Node Hardening](./chapter05.md#node-hardening) | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.006 | The Platform **must** support Secure logging. Logging with root account must be prohibited when root privileges are not required. | [5.3.2 Restrict direct access to nodes](./chapter05.md#restrict-direct-access-to-nodes) | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.007 | All servers part of Cloud Infrastructure **must** be Time synchronized with authenticated Time service. | | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.008 | All servers part of Cloud Infrastructure **must** be regularly updated to address security vulnerabilities. | [5.3.3 Vulnerability assessment](./chapter05.md#vulnerability-assessment) | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.009 | The Platform **must** support Software integrity protection and verification and **must** scan source code and manifests. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.010 | The Cloud Infrastructure **must** support encrypted storage, for example, block, object and file storage, with access to encryption keys restricted based on a need to know. [Controlled Access Based on the Need to Know](https://www.cisecurity.org/controls/controlled-access-based-on-the-need-to-know/) | | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.011 | The Cloud Infrastructure **should** support Read and Write only storage partitions (write only permission to one or more authorized actors). | | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.012 | The Operator **must** ensure that only authorized actors have physical access to the underlying infrastructure. | | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.013 | The Platform **must** ensure that only authorized actors have logical access to the underlying infrastructure. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.014 | All servers part of Cloud Infrastructure **should** support measured boot and an attestation server that monitors the measurements of the servers. | | +| [7.9.1](../../../ref_model/chapters/chapter07.md#system-hardening) | sec.gen.015 | Any change to the Platform must be logged as a security event, and the logged event must include the identity of the entity making the change, the change, the date and the time of the change. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.001 | The Platform **must** support authenticated and secure access to API, GUI and command line interfaces. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.002 | The Platform **must** support Traffic Filtering for workloads (for example, Firewall). | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.003 | The Platform **must** support Secure and encrypted communications, and confidentiality and integrity of network traffic. | [5.4.3 Use Transport Layer Security and Service Mesh](./chapter05.md#use-transport-layer-security-and-service-mesh) | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.004 | The Cloud Infrastructure **must** support authentication, integrity and confidentiality on all network channels. | [5.4.3 Use Transport Layer Security and Service Mesh](./chapter05.md#use-transport-layer-security-and-service-mesh) | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.005 | The Cloud Infrastructure **must** segregate the underlay and overlay networks. | | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.006 | The Cloud Infrastructure must be able to utilise the Cloud Infrastructure Manager identity lifecycle management capabilities. | [5.2 Principles](./chapter05.md#principles) | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.007 | The Platform **must** implement controls enforcing separation of duties and privileges, least privilege use and least common mechanism (Role-Based Access Control). | [5.2 Principles](./chapter05.md#principles) and [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.008 | The Platform **must** be able to assign the Entities that comprise the tenant networks to different trust domains. Communication between different trust domains is not allowed, by default. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.009 | The Platform **must** support creation of Trust Relationships between trust domains. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.010 | For two or more domains without existing trust relationships, the Platform **must not** allow the effect of an attack on one domain to impact the other domains either directly or indirectly. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.011 | The Platform **must not** reuse the same authentication credential (e.g., key-pair) on different Platform components (e.g., on different hosts, or different services). | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.012 | The Platform **must** protect all secrets by using strong encryption techniques, and storing the protected secrets externally from the component | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.013 | The Platform **must** provide secrets dynamically as and when needed. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.014 | The Platform **should** use Linux Security Modules such as SELinux to control access to resources. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.015 | The Platform **must not** contain back door entries (unpublished access points, APIs, etc.). | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.016 | Login access to the platform's components **must** be through encrypted protocols such as SSH v2 or TLS v1.2 or higher. Note: Hardened jump servers isolated from external networks are recommended | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.017 | The Platform **must** provide the capability of using digital certificates that comply with X.509 standards issued by a trusted Certification Authority. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.018 | The Platform **must** provide the capability of allowing certificate renewal and revocation. | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.019 | The Platform **must** provide the capability of testing the validity of a digital certificate (CA signature, validity period, non revocation, identity). | | +| [7.9.2](../../../ref_model/chapters/chapter07.md#platform-and-access) | sec.sys.020 | The Cloud Infrastructure architecture **should** rely on Zero Trust principles to build a secure by design environment. | | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.001 | The Platform **must** support Confidentiality and Integrity of data at rest and in-transit. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.002 | The Platform **should** support self-encrypting storage devices. | | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.003 | The Platform **must** support Confidentiality and Integrity of data related metadata. | | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.004 | The Platform **must** support Confidentiality of processes and restrict information sharing with only the process owner (e.g., tenant). | | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.005 | The Platform **must** support Confidentiality and Integrity of process-related metadata and restrict information sharing with only the process owner (e.g., tenant). | | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.006 | The Platform **must** support Confidentiality and Integrity of workload resource utilization (RAM, CPU, Storage, Network I/O, cache, hardware offload) and restrict information sharing with only the workload owner (e.g., tenant). | | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.007 | The Platform **must not** allow Memory Inspection by any actor other than the authorized actors for the Entity to which Memory is assigned (e.g., tenants owning the workload), for Lawful Inspection, and by secure monitoring services. | | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.008 | The Cloud Infrastructure **must** support tenant networks segregation. | [5.7 Create and define Network Policies](./chapter05.md#create-and-define-network-policies) | +| [7.9.3](../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity) | sec.ci.009 | For sensitive data encryption, the key management service **should** leverage a Hardware Security Module to manage and protect cryptographic keys. | | +| [7.9.4](../../../ref_model/chapters/chapter07.md#workload-security) | sec.wl.001 | The Platform **must** support Workload placement policy. | | +| [7.9.4](../../../ref_model/chapters/chapter07.md#workload-security) | sec.wl.002 | The Cloud Infrastructure **must** provide methods to ensure the platform’s trust status and integrity (e.g. remote attestation, Trusted Platform Module). | | +| [7.9.4](../../../ref_model/chapters/chapter07.md#workload-security) | sec.wl.003 | The Platform **must** support secure provisioning of workloads. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.4](../../../ref_model/chapters/chapter07.md#workload-security) | sec.wl.004 | The Platform **must** support Location assertion (for mandated in-country or location requirements). | | +| [7.9.4](../../../ref_model/chapters/chapter07.md#workload-security) | sec.wl.005 | The Platform **must** support the separation of production and non-production Workloads. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.4](../../../ref_model/chapters/chapter07.md#workload-security) | sec.wl.006 | The Platform **must** support the separation of Workloads based on their categorisation (for example, payment card information, healthcare, etc.). | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) and [5.6 Separate Sensitive Workload](./chapter05.md#separate-sensitive-workload) | +| [7.9.4](../../../ref_model/chapters/chapter07.md#workload-security) | sec.wl.007 | The Operator **must** implement processes and tools to verify VNF authenticity and integrity. | [5.13 Trusted Registry](./chapter05.md#trusted-registry) | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.001 | Images from untrusted sources **must not** be used. | [5.13 Trusted Registry](./chapter05.md#trusted-registry) | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.002 | Images **must** be scanned to be maintained free from known vulnerabilities. | [5.13 Trusted Registry](./chapter05.md#trusted-registry) | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.003 | Images **must not** be configured to run with privileges higher than the privileges of the actor authorized to run them. | [5.11 Run-Time Security](./chapter05.md#run-time-security) | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.004 | Images **must** only be accessible to authorized actors. | | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.005 | Image Registries **must** only be accessible to authorized actors. | | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.006 | Image Registries **must** only be accessible over secure networks that enforce authentication, integrity and confidentiality. | [5.13 Trusted Registry](./chapter05.md#trusted-registry) | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.007 | Image registries **must** be clear of vulnerable and out of date versions. | [5.13 Trusted Registry](./chapter05.md#trusted-registry) | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.008 | Images **must not** include any secrets. Secrets include passwords, cloud provider credentials, SSH keys, TLS certificate keys, etc. | [5.12 Secrets Management](./chapter05.md#secrets-management) | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.009 | CIS Hardened Images **should** be used whenever possible. | | +| [7.9.5](../../../ref_model/chapters/chapter07.md#image-security) | sec.img.010 | Minimalist base images **should** be used whenever possible. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.001 | The Platform **must** support Secure Provisioning, Availability, and Deprovisioning (Secure Clean-Up) of workload resources where Secure Clean-Up includes tear-down, defense against virus or other attacks. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.002 | Cloud operations staff and systems **must** use management protocols limiting security risk such as SNMPv3, SSH v2, ICMP, NTP, syslog and TLS v1.2 or higher. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.003 | The Cloud Operator **must** implement and strictly follow change management processes for Cloud Infrastructure, Cloud Infrastructure Manager and other components of the cloud, and Platform change control on hardware. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.004 | The Cloud Operator **should** support automated templated approved changes. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.005 | Platform **must** provide logs and these logs must be regularly monitored for anomalous behavior. | [5.10 Enable Logging and Monitoring](./chapter05.md#enable-logging-and-monitoring) | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.006 | The Platform **must** verify the integrity of all Resource management requests. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.007 | The Platform **must** be able to update newly instantiated, suspended, hibernated, migrated and restarted images with current time information. | [5.4 Securing Kubernetes orchestrator](./chapter05.md#securing-kubernetes-orchestrator) | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.008 | The Platform **must** be able to update newly instantiated, suspended, hibernated, migrated and restarted images with relevant DNS information. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.009 | The Platform **must** be able to update the tag of newly instantiated, suspended, hibernated, migrated and restarted images with relevant geolocation (geographical) information. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.010 | The Platform **must** log all changes to geolocation along with the mechanisms and sources of location information (i.e. GPS, IP block, and timing). | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.011 | The Platform **must** implement Security life cycle management processes including the proactive update and patching of all deployed Cloud Infrastructure software. | | +| [7.9.6](../../../ref_model/chapters/chapter07.md#security-lcm) | sec.lcm.012 | The Platform **must** log any access privilege escalation. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.001 | Platform **must** provide logs and these logs must be regularly monitored for events of interest. The logs **must** contain the following fields: event type, date/time, protocol, service or program used for access, success/failure, login ID or process ID, IP address and ports (source and destination) involved. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.002 | Security logs **must** be time synchronised. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.003 | The Platform **must** log all changes to time server source, time, date and time zones. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.004 | The Platform **must** secure and protect Audit logs (containing sensitive information) both in-transit and at rest. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.005 | The Platform **must** Monitor and Audit various behaviours of connection and login attempts to detect access attacks and potential access attempts and take corrective actions accordingly. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.006 | The Platform **must** Monitor and Audit operations by authorized account access after login to detect malicious operational activity and take corrective actions accordingly. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.007 | The Platform **must** Monitor and Audit security parameter configurations for compliance with defined security policies. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.008 | The Platform **must** Monitor and Audit externally exposed interfaces for illegal access (attacks) and take corrective security hardening measures. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.009 | The Platform **must** Monitor and Audit service handling for various attacks (malformed messages, signalling flooding and replaying, etc.) and take corrective actions accordingly. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.010 | The Platform **must** Monitor and Audit running processes to detect unexpected or unauthorized processes and take corrective actions accordingly. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.011 | The Platform **must** Monitor and Audit logs from infrastructure elements and workloads to detected anomalies in the system components and take corrective actions accordingly. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.012 | The Platform **must** Monitor and Audit Traffic patterns and volumes to prevent malware download attempts. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.013 | The monitoring system **must not** affect the security (integrity and confidentiality) of the infrastructure, workloads, or the user data (through back door entries). | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.014 | The Monitoring systems **should not** impact IAAS, PAAS, and SAAS SLAs including availability SLAs. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.015 | The Platform **must** ensure that the Monitoring systems are never starved of resources and **must** activate alarms when resource utilisation exceeds a configurable threshold. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.016 | The Platform Monitoring components **should** follow security best practices for auditing, including secure logging and tracing. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.017 | The Platform **must** audit systems for any missing security patches and take appropriate actions. | [5.3.3 Vulnerability assessment](./chapter05.md#vulnerability-assessment) | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.018 | The Platform, starting from initialization, **must** collect and analyze logs to identify security events, and store these events in an external system. | [5.3.4 Patch management](./chapter05.md#patch-management) | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.019 | The Platform’s components **must not** include an authentication credential, e.g., password, in any logs, even if encrypted. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.020 | The Platform’s logging system **must** support the storage of security audit logs for a configurable period of time. | | +| [7.9.7](../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit) | sec.mon.021 |The Platform **must** store security events locally if the external logging system is unavailable and shall periodically attempt to send these to the external logging system until successful. | | +| [7.9.8](../../../ref_model/chapters/chapter07.md#open-source-software) | sec.oss.001 | Open source code **must** be inspected by tools with various capabilities for static and dynamic code analysis. | [5.3.3 Vulnerability assessment](./chapter05.md#vulnerability-assessment) | +| [7.9.8](../../../ref_model/chapters/chapter07.md#open-source-software) | sec.oss.002 | The [CVE (Common Vulnerabilities and Exposures)]( https://cve.mitre.org/) **must** be used to identify vulnerabilities and their severity rating for open source code part of Cloud Infrastructure and workloads software. | | +| [7.9.8](../../../ref_model/chapters/chapter07.md#open-source-software) | sec.oss.003 | Critical and high severity rated vulnerabilities **must** be fixed in a timely manner. Refer to the [CVSS (Common Vulnerability Scoring System)](https://www.first.org/cvss/) to know a vulnerability score and its associated rate (low, medium, high, or critical). | | +| [7.9.8](../../../ref_model/chapters/chapter07.md#open-source-software) | sec.oss.004 | A dedicated internal isolated repository separated from the production environment **must** be used to store vetted open source content. | [5.13 Trusted Registry](./chapter05.md#trusted-registry) | +| [7.9.8](../../../ref_model/chapters/chapter07.md#open-source-software) | sec.oss.005 | A Software Bill of Materials ([SBOM](https://www.ntia.gov/SBOM)) **should** be provided or build, and maintained to identify the software components and their origins. | | +| [7.9.9](../../../ref_model/chapters/chapter07.md#iaac---secure-design-and-architecture-stage-requirements) | sec.arch.001 | Threat Modelling methodologies and tools **should** be used during the Secure Design and Architecture stage triggered by Software Feature Design trigger. It may be done manually or using tools like open source OWASP Threat Dragon | | +| [7.9.9](../../../ref_model/chapters/chapter07.md#iaac---secure-design-and-architecture-stage-requirements) | sec.arch.002 | Security Control Baseline Assessment **should** be performed during the Secure Design and Architecture stage triggered by Software Feature Design trigger. Typically done manually by internal or independent assessors. | | +| [7.9.10](../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements) | sec.code.001 | SAST -Static Application Security Testing **must** be applied during Secure Coding stage triggered by Pull, Clone or Comment trigger. Security testing that analyses application source code for software vulnerabilities and gaps against best practices. Example: open source OWASP range of tools. | | +| [7.9.10](../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements) | sec.code.002 | SCA – Software Composition Analysis **should** be applied during Secure Coding stage triggered by Pull, Clone or Comment trigger. Security testing that analyses application source code or compiled code for software components with known vulnerabilities. Example: open source OWASP range of tools. | | +| [7.9.10](../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements) | sec.code.003 | Source Code Review **should** be performed continuously during Secure Coding stage. Typically done manually. | | +| [7.9.10](../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements) | sec.code.004 | Integrated SAST via IDE Plugins **should** be used during Secure Coding stage triggered by Developer Code trigger. On the local machine: through the IDE or integrated test suites; triggered on completion of coding be developer. | | +| [7.9.10](../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements) | sec.code.005 | SAST of Source Code Repo **should** be performed during Secure Coding stage triggered by Developer Code trigger. Continuous delivery pre-deployment: scanning prior to deployment. | | +| [7.9.11](../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.001 | SAST -Static Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Build and Integrate trigger. Example: open source OWASP range of tools. | | +| [7.9.11](../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.002 | SCA – Software Composition Analysis **should** be applied during the Continuous Build, Integration and Testing stage triggered by Build and Integrate trigger. Example: open source OWASP range of tools. | | +| [7.9.11](../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.003 | Image Scan **must** be applied during the Continuous Build, Integration and Testing stage triggered by Package trigger. Example: A push of a container image to a container registry may trigger a vulnerability scan before the image becomes available in the registry. | | +| [7.9.11](../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.004 | DAST – Dynamic Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Stage & Test trigger. Security testing that analyses a running application by exercising application functionality and detecting vulnerabilities based on application behaviour and response. Example: OWASP ZAP. | | +| [7.9.11](../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.005 | Fuzzing **should** be applied during the Continuous Build, Integration and testing stage triggered by Stage & Test trigger. Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. Example: GitLab Open Sources Protocol Fuzzer Community Edition. | | +| [7.9.11](../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements) | sec.bld.006 | IAST – Interactive Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Stage & Test trigger. Software component deployed with an application that assesses application behaviour and detects presence of vulnerabilities on an application being exercised in realistic testing scenarios. Example: Contrast Community Edition. | | +| [7.9.12](../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.001 | Image Scan **must** be applied during the Continuous Delivery and Deployment stage triggered by Publish to Artifact and Image Repository trigger. Example: GitLab uses the open-source Clair engine for container image scanning.| | +| [7.9.12](../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.002 | Code Signing **must** be applied during the Continuous Delivery and Deployment stage triggered by Publish to Artifact and Image Repository trigger. Code Signing provides authentication to assure that downloaded files are form the publisher named on the certificate. | | +| [7.9.12](../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.003 | Artifact and Image Repository Scan **should** be continuously applied during the Continuous Delivery and Deployment stage. Example: GitLab uses the open source Clair engine for container scanning. | | +| [7.9.12](../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements) | sec.del.004 | Component Vulnerability Scan **must** be applied during the Continuous Delivery and Deployment stage triggered by Instantiate Infrastructure trigger. The vulnerability scanning system is deployed on the cloud platform to detect security vulnerabilities of specified components through scanning and to provide timely security protection. Example: OWASP Zed Attack Proxy (ZAP). | | +| [7.9.13](../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements) | sec.run.001 | Component Vulnerability Monitoring **must** be continuously applied during the Runtime Defence and Monitoring stage and remediation actions **must** be applied for high severity rated vulnerabilities. Security technology that monitors components like virtual servers and assesses data, applications, and infrastructure for security risks. | | +| [7.9.13](../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements) | sec.run.002 | RASP – Runtime Application Self-Protection **should** be continuously applied during the Runtime Defence and Monitoring stage. Security technology deployed within the target application in production for detecting, alerting, and blocking attacks. | | +| [7.9.13](../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements) | sec.run.003 | Application testing and Fuzzing **should** be continuously applied during the Runtime Defence and Monitoring stage. Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. Example: GitLab Open Sources Protocol Fuzzer Community Edition. | | +| [7.9.13](../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements) | sec.run.004 | Penetration Testing **should** be continuously applied during the Runtime Defence and Monitoring stage. Typically done manually. | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.001 | The Cloud Operator **should** comply with Center for Internet Security CIS Controls ([https://www.cisecurity.org/](https://www.cisecurity.org/)) | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.002 | The Cloud Operator, Platform and Workloads **should** follow the guidance in the CSA Security Guidance for Critical Areas of Focus in Cloud Computing (latest version) [https://cloudsecurityalliance.org/](https://cloudsecurityalliance.org/) | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.003 | The Platform and Workloads **should** follow the guidance in the [OWASP Cheat Sheet Series (OCSS)](https://github.com/OWASP/CheatSheetSeries) | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.004 | The Cloud Operator, Platform and Workloads **should** ensure that their code is not vulnerable to the OWASP Top Ten Security Risks [https://owasp.org/www-project-top-ten/](https://owasp.org/www-project-top-ten/) | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.005 | The Cloud Operator, Platform and Workloads **should** strive to improve their maturity on the [OWASP Software Maturity Model (SAMM)](https://owaspsamm.org/blog/2019/12/20/version2-community-release/) | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.006 | The Cloud Operator, Platform and Workloads **should** utilize the [OWASP Web Security Testing Guide](https://github.com/OWASP/wstg/tree/master/document) | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.007 | The Cloud Operator, and Platform **should** satisfy the requirements for Information Management Systems specified in [ISO/IEC 27001](https://www.iso.org/obp/ui/#iso:std:iso-iec:27001:ed-2:v1:en). ISO/IEC 27002:2013 - ISO/IEC 27001 is the international Standard for best-practice information security management systems (ISMSs). | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.008 | The Cloud Operator, and Platform **should** implement the Code of practice for Security Controls specified [ISO/IEC 27002:2013 (or latest)](https://www.iso.org/obp/ui/#iso:std:iso-iec:27002:ed-2:v1:en) | | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.009 | The Cloud Operator, and Platform **should** implement the [ISO/IEC 27032:2012 (or latest)](https://www.iso.org/obp/ui/#iso:std:iso-iec:27032:ed-1:v1:en) Guidelines for Cybersecurity techniques. ISO/IEC 27032 - ISO/IEC 27032 is the international Standard focusing explicitly on cybersecurity. | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.010 | The Cloud Operator **should** conform to the ISO/IEC 27035 standard for incidence management. ISO/IEC 27035 - ISO/IEC 27035 is the international Standard for incident management. | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.011 | The Cloud Operator **should** conform to the ISO/IEC 27031 standard for business continuity. ISO/IEC 27031 - ISO/IEC 27031 is the international Standard for ICT readiness for business continuity. | | +| [7.9.14](../../../ref_model/chapters/chapter07.md#compliance-with-standards) | sec.std.012 | The Public Cloud Operator **must**, and the Private Cloud Operator **may** be certified to be compliant with the International Standard on Awareness Engagements (ISAE) 3402 (in the US: SSAE 16). International Standard on Awareness Engagements (ISAE) 3402. US Equivalent: SSAE16. | | + +**Table 2-6:** Reference Model Requirements: Cloud Infrastructure Security Requirements + +## Kubernetes Architecture Requirements The requirements in this section are to be delivered in addition to those in [section 2.2](#2.2), and have been created to support the Principles defined in [Chapter 1 of this Reference Architecture](./chapter01.md). @@ -312,40 +298,40 @@ With regards to containerisation platforms, the scope of the following Architect | Reference | Category | Sub-category | Description | Specification Reference | |---|---|---|---|---| -| gen.cnt.02 | General | Cloud nativeness | The Architecture **must** support immutable infrastructure. |[ra2.ch.017](chapter04.md#42-kubernetes-node)| -| gen.cnt.03 | General | Cloud nativeness | The Architecture **must** run conformant Kubernetes as defined by the [CNCF](https://github.com/cncf/k8s-conformance). |[ra2.k8s.001](chapter04.md#43-kubernetes)| +| gen.cnt.02 | General | Cloud nativeness | The Architecture **must** support immutable infrastructure. |[ra2.ch.017](chapter04.md#kubernetes-node)| +| gen.cnt.03 | General | Cloud nativeness | The Architecture **must** run conformant Kubernetes as defined by the [CNCF](https://github.com/cncf/k8s-conformance). |[ra2.k8s.001](chapter04.md#kubernetes)| | gen.cnt.04 | General | Cloud nativeness | The Architecture **must** support clearly defined abstraction layers. | | | gen.cnt.05 | General | Cloud nativeness | The Architecture **should** support configuration of all components in an automated manner using openly published API definitions. || | gen.scl.01 | General | Scalability | The Architecture **should** support policy driven horizontal auto-scaling of workloads. | | -| gen.rsl.01 | General | Resiliency | The Architecture **must** support resilient Kubernetes components that are required for the continued availability of running workloads. |[ra2.k8s.004](chapter04.md#43-kubernetes)| -| gen.rsl.02 | General | Resiliency | The Architecture **should** support resilient Kubernetes service components that are not subject to gen.rsl.01. |[ra2.k8s.002](chapter04.md#43-kubernetes)
[ra2.k8s.003](chapter04.md#43-kubernetes)| -| gen.avl.01 | General | Availability | The Architecture **must** provide High Availability for Kubernetes components. |[ra2.k8s.002](chapter04.md#43-kubernetes)
[ra2.k8s.003](chapter04.md#43-kubernetes)
[ra2.k8s.004](chapter04.md#43-kubernetes)| -| gen.ost.01 | General | Openness | The Architecture **should** embrace open-based standards and technologies. |[ra2.crt.001](chapter04.md#44-container-runtimes)
[ra2.crt.002](chapter04.md#44-container-runtimes)
[ra2.ntw.002](chapter04.md#45-networking-solutions)
[ra2.ntw.006](chapter04.md#45-networking-solutions)
[ra2.ntw.007](chapter04.md#45-networking-solutions)| -| inf.com.01 | Infrastructure | Compute | The Architecture **must** provide compute resources for Pods. |[ra2.k8s.004](chapter04.md#43-kubernetes)| -| inf.stg.01 | Infrastructure | Storage | The Architecture **must** support the ability for an operator to choose whether or not to deploy persistent storage for Pods. |[ra2.stg.004](chapter04.md#46-storage-components)| +| gen.rsl.01 | General | Resiliency | The Architecture **must** support resilient Kubernetes components that are required for the continued availability of running workloads. |[ra2.k8s.004](chapter04.md#kubernetes)| +| gen.rsl.02 | General | Resiliency | The Architecture **should** support resilient Kubernetes service components that are not subject to gen.rsl.01. |[ra2.k8s.002](chapter04.md#kubernetes), [ra2.k8s.003](chapter04.md#kubernetes)| +| gen.avl.01 | General | Availability | The Architecture **must** provide High Availability for Kubernetes components. |[ra2.k8s.002](chapter04.md#kubernetes), [ra2.k8s.003](chapter04.md#kubernetes), [ra2.k8s.004](chapter04.md#kubernetes)| +| gen.ost.01 | General | Openness | The Architecture **should** embrace open-based standards and technologies. |[ra2.crt.001](chapter04.md#container-runtimes), [ra2.crt.002](chapter04.md#container-runtimes), [ra2.ntw.002](chapter04.md#networking-solutions), [ra2.ntw.006](chapter04.md#networking-solutions), [ra2.ntw.007](chapter04.md#networking-solutions)| +| inf.com.01 | Infrastructure | Compute | The Architecture **must** provide compute resources for Pods. |[ra2.k8s.004](chapter04.md#kubernetes)| +| inf.stg.01 | Infrastructure | Storage | The Architecture **must** support the ability for an operator to choose whether or not to deploy persistent storage for Pods. |[ra2.stg.004](chapter04.md#storage-components)| | inf.ntw.01 | Infrastructure | Network | The Architecture **must** support network resiliency on the Kubernetes nodes. | | | inf.ntw.02 | Infrastructure | Network | The Architecture **must** support fully redundant network connectivity to the Kubernetes nodes, leveraging multiple network connections. | | -| inf.ntw.03 | Infrastructure | Network | The networking solution **should** be able to be centrally administrated and configured. |[ra2.ntw.001](chapter04.md#45-networking-solutions)
[ra2.ntw.004](chapter04.md#45-networking-solutions) | -| inf.ntw.04 | Infrastructure | Network | The Architecture **must** support dual stack IPv4 and IPv6 for Kubernetes workloads. |[ra2.ch.007](chapter04.md#42-kubernetes-node)
[ra2.k8s.010](chapter04.md#43-kubernetes)| +| inf.ntw.03 | Infrastructure | Network | The networking solution **should** be able to be centrally administrated and configured. |[ra2.ntw.001](chapter04.md#networking-solutions), [ra2.ntw.004](chapter04.md#networking-solutions) | +| inf.ntw.04 | Infrastructure | Network | The Architecture **must** support dual stack IPv4 and IPv6 for Kubernetes workloads. |[ra2.ch.007](chapter04.md#kubernetes-node), [ra2.k8s.010](chapter04.md#kubernetes)| | inf.ntw.05 | Infrastructure | Network | The Architecture **must** support capabilities for integrating SDN controllers. | | -| inf.ntw.06 | Infrastructure | Network | The Architecture **must** support more than one networking solution. |[ra2.ntw.005](chapter04.md#45-networking-solutions)
[ra2.ntw.007](chapter04.md#45-networking-solutions) | -| inf.ntw.07 | Infrastructure | Network | The Architecture **must** support the ability for an operator to choose whether or not to deploy more than one networking solution. |[ra2.ntw.005](chapter04.md#45-networking-solutions)| -| inf.ntw.08 | Infrastructure | Network | The Architecture **must** provide a default network which implements the Kubernetes network model. |[ra2.ntw.002](chapter04.md#45-networking-solutions) | +| inf.ntw.06 | Infrastructure | Network | The Architecture **must** support more than one networking solution. |[ra2.ntw.005](chapter04.md#networking-solutions), [ra2.ntw.007](chapter04.md#networking-solutions) | +| inf.ntw.07 | Infrastructure | Network | The Architecture **must** support the ability for an operator to choose whether or not to deploy more than one networking solution. |[ra2.ntw.005](chapter04.md#networking-solutions)| +| inf.ntw.08 | Infrastructure | Network | The Architecture **must** provide a default network which implements the Kubernetes network model. |[ra2.ntw.002](chapter04.md#networking-solutions) | | inf.ntw.09 | Infrastructure | Network | The networking solution **must not** interfere with or cause interference to any interface or network it does not own. | | | inf.ntw.10 | Infrastructure | Network | The Architecture **must** support Cluster wide coordination of IP address assignment. | | | inf.ntw.13 | Infrastructure | Network | The platform **must** allow specifying multiple separate IP pools. Tenants are required to select at least one IP pool that is different from the control infrastructure IP pool or other tenant IP pools. | | -| inf.ntw.14 | Infrastructure | Network | The platform **must** allow NATless traffic (i.e. exposing the pod IP address directly to the outside), allowing source and destination IP addresses to be preserved in the traffic headers from workloads to external networks. This is needed e.g. for signaling applications, using SIP and Diameter protocols.|[ra2.ntw.011](chapter04.md#45-networking-solutions)| +| inf.ntw.14 | Infrastructure | Network | The platform **must** allow NATless traffic (i.e. exposing the pod IP address directly to the outside), allowing source and destination IP addresses to be preserved in the traffic headers from workloads to external networks. This is needed e.g. for signaling applications, using SIP and Diameter protocols.|[ra2.ntw.011](chapter04.md#networking-solutions)| | inf.ntw.15 | Infrastructure | Network | The platform **must** support LoadBalancer [Publishing Service (ServiceType)](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types)|| | inf.ntw.16 | Infrastructure | Network | The platform **must** support [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/).|| | inf.ntw.17 | Infrastructure | Network | The platform **should** support NodePort [Publishing Service (ServiceTypes)](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types).|| | inf.ntw.18 | Infrastructure | Network | The platform **should** support ExternalName [Publishing Service (ServiceTypes)](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types).|| -| inf.vir.01 | Infrastructure | Virtual Infrastructure | The Architecture **must** support the capability for Containers to consume infrastructure resources abstracted by Host Operating Systems that are running within a virtual machine. |[ra2.ch.005](chapter04.md#42-kubernetes-node)
[ra2.ch.011](chapter04.md#42-kubernetes-node)| +| inf.vir.01 | Infrastructure | Virtual Infrastructure | The Architecture **must** support the capability for Containers to consume infrastructure resources abstracted by Host Operating Systems that are running within a virtual machine. |[ra2.ch.005](chapter04.md#kubernetes-node), [ra2.ch.011](chapter04.md#kubernetes-node)| | inf.phy.01 | Infrastructure | Physical Infrastructure | The Architecture **must** support the capability for Containers to consume infrastructure resources abstracted by Host Operating Systems that are running within a physical server.| ra2.ch.008 | | kcm.gen.01 | Kubernetes Cluster | General | The Architecture **must** support policy driven horizontal auto-scaling of Kubernetes Cluster. | **N/A** | -| kcm.gen.02 | Kubernetes Cluster | General | The Architecture **must** enable workload resiliency. |[ra2.k8s.004](chapter04.md#43-kubernetes)| -| int.api.01 | API | General | The Architecture **must** leverage the Kubernetes APIs to discover and declaratively manage compute (virtual and bare metal resources), network, and storage. |For Networking:
  • [ra2.ntw.001](chapter04.md#45-networking-solutions)
  • [ra2.ntw.008](chapter04.md#45-networking-solutions)
  • [ra2.app.006](chapter04.md#49-kubernetes-workloads)

Compute/storage not yet met. | -| int.api.02 | API | General | The Architecture **must** support the usage of a Kubernetes Application package manager using the Kubernetes API, like Helm v3. |[ra2.pkg.001](chapter04.md#48-kubernetes-application-package-manager)| +| kcm.gen.02 | Kubernetes Cluster | General | The Architecture **must** enable workload resiliency. |[ra2.k8s.004](chapter04.md#kubernetes)| +| int.api.01 | API | General | The Architecture **must** leverage the Kubernetes APIs to discover and declaratively manage compute (virtual and bare metal resources), network, and storage. |For Networking: [ra2.ntw.001](chapter04.md#networking-solutions), [ra2.ntw.008](chapter04.md#networking-solutions), [ra2.app.006](chapter04.md#kubernetes-workloads) Compute/storage not yet met. | +| int.api.02 | API | General | The Architecture **must** support the usage of a Kubernetes Application package manager using the Kubernetes API, like Helm v3. |[ra2.pkg.001](chapter04.md#kubernetes-application-package-manager)| | int.api.03 | API | General | The Architecture **must** support stable features in its APIs. || | int.api.04 | API | General | The Architecture **must** support limited backward compatibility in its APIs. Support for the whole API must not be dropped, but the schema or other details can change. || -

Table 2-7: Kubernetes Architecture Requirements

+**Table 2-7:** Kubernetes Architecture Requirements \ No newline at end of file diff --git a/doc/ref_arch/kubernetes/chapters/chapter02.rst b/doc/ref_arch/kubernetes/chapters/chapter02.rst new file mode 100644 index 0000000000..d73a2483b2 --- /dev/null +++ b/doc/ref_arch/kubernetes/chapters/chapter02.rst @@ -0,0 +1,360 @@ +Architecture Requirements +========================= + +Introduction +------------ + +This chapter will use the requirements defined in the overall Reference Model and only make additional entries in section `2.3 <#2.3>`__ if there are additional requirements needed for this Reference Architecture. + +Definitions +----------- + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in `RFC2119 `__. + +Reference Model Requirements +---------------------------- + +The tables below contain the requirements from the Reference Model to cover the Basic and High-Performance profiles. The table also includes a reference to the specification from `Chapter 04 - Component Level Architecture <./chapter04.md>`__ and from `Chapter 05 - Security Guidance `__ to ensure traceability. If the related Specification does not exist, the reference will read "N/A" (and in bold "**N/A**" for mandatory requirements). + +To ensure alignment with the infrastructure profile catalogue, the following requirements are referenced through: + +- Those relating to Cloud Infrastructure Software Profiles +- Those relating to Cloud Infrastructure Hardware Profiles +- Those relating to Cloud Infrastructure Management +- Those relating to Cloud Infrastructure Security + +Cloud Infrastructure Software Profile Capabilities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +=================================================================================================== =========== ======================================================================================================================================================================= ============================= ======================================== ================================================================================================================================================================================================================================================= +Reference Model Section Reference Description Requirement for Basic Profile Requirement for High-Performance Profile Specification Reference +=================================================================================================== =========== ======================================================================================================================================================================= ============================= ======================================== ================================================================================================================================================================================================================================================= +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.001 Max number of vCPU that can be assigned to a single Pod by the Cloud Infrastructure At least 16 At least 16 `ra2.ch.011 `__ +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.002 Max memory in MB that can be assigned to a single Pod by the Cloud Infrastructure at least 32 GB at least 32 GB `ra2.ch.012 `__ +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.003 Max storage in GB that can be assigned to a single Pod by the Cloud Infrastructure at least 320 GB at least 320 GB `ra2.ch.010 `__ +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.004 Max number of connection points that can be assigned to a single Pod by the Cloud Infrastructure 6 6 `ra2.ntw.003 `__ +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.005 Max storage in GB that can be attached / mounted to Pod by the Cloud Infrastructure Up to 16TB (1) Up to 16TB (1) N/A +`4.2.2 <../../../ref_model/chapters/chapter04.md#profiles-specifications--capability-mapping>`__ e.cap.006 CPU pinning support Not required Must support `ra2.k8s.009 `__ +`4.2.2 <../../../ref_model/chapters/chapter04.md#profiles-specifications--capability-mapping>`__ e.cap.007 NUMA support Not required Must support `ra2.k8s.006 `__ +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.008 IPSec Acceleration using the virtio-ipsec interface Not required Optional N/A +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.009 Crypto Acceleration using the virtio-crypto interface Not required Optional N/A +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.010 Transcoding Acceleration Not required Not required N/A +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.011 Programmable Acceleration Not required Not required N/A +`4.1.2 <../../../ref_model/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.012 Enhanced Cache Management: L=Lean; E=Equal; X=eXpanded E E N/A +`4.2.2 <../../../ref_model/chapters/chapter04.md#profiles-specifications--capability-mapping>`__ e.cap.013 SR-IOV over PCI-PT Not required Must support `ra2.ch.002 `__, `ra2.ch.003 `__, `ra2.k8s.007 `__, `ra2.ntw.004 `__, `ra2.ntw.008 `__ +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.014 Hardware coprocessor support (GPU/NPU) Not required Not required N/A +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.015 SmartNICs Not required Optional N/A +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ e.cap.016 FPGA/other Acceleration H/W Not required Optional `ra2.k8s.007 `__, `ra2.ntw.012 `__ +`4.1.2 <../../../ref_model/chapters/chapter04.md#exposed-infrastructure-capabilities>`__ *e.cap.017* *Ability to monitor L2-L7 data from workload* *n/a (2)* \*n/a (2) \* N/A +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.cap.014 Specifies the proportion of CPU cores consumed by the Cloud Infrastructure system on the worker nodes. If SMT is used, it indicates the number of consumed SMT threads. 2 2 `ra2.k8s.008 `__ +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.cap.015 Indicates the memory consumed by Cloud Infrastructure on the worker nodes 16 GB 16GB +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.cap.016 Number of virtual cores per physical core; also known as CPU overbooking ratio that is required 1:1 1:1 `ra2.ch.004 `__, `ra2.ch.005 `__ +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.cap.017 QoS enablement of the connection point (vNIC or interface) Not required Must support **N/A** +`4.1.4 <../../../ref_model/chapters/chapter04.chapter04.md#internal-infrastructure-capabilities>`__ i.cap.018 Support for huge pages Not required Must support `ra2.ch.001 `__ +`4.1.4 <../../../ref_model/chapters/chapter04.chapter04.md#internal-infrastructure-capabilities>`__ i.pm.001 Monitor worker node CPU usage, per nanosecond Must support Must support **N/A** +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.pm.002 Monitor pod CPU usage, per nanosecond Must support Must support **N/A** +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.pm.003 Monitor worker node CPU utilisation (%) Must support Must support **N/A** +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.pm.004 Monitor pod CPU utilisation Must support Must support **N/A** +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.pm.005 Measure external storage IOPs Must support Must support **N/A** +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.pm.006 Measure external storage throughput Must support Must support **N/A** +`4.1.4 <../../../ref_model/chapters/chapter04.md#internal-infrastructure-capabilities>`__ i.pm.007 Measure external storage capacity Must support Must support **N/A** +=================================================================================================== =========== ======================================================================================================================================================================= ============================= ======================================== ================================================================================================================================================================================================================================================= + +**Table 2-1:** Reference Model Requirements: Cloud Infrastructure Software Profile Capabilities + +**(1)** Defined in the ``.bronze`` configuration in section `4.2.6 Storage Extensions <../../../ref_model/chapters/chapter04.md#storage-extensions>`__ + +**(2)** In Kubernetes based infrastructures packet monitoring is out of the scope for the infrastructure. + +Virtual Network Interface Specifications +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The required number of connection points to a Pod is described in ``e.cap.004`` above. This section describes the required bandwidth of those connection points. + +============================================================================================= ================================== ================================= ============================= ======================================== ======================= +Reference Model Section Reference Description Requirement for Basic Profile Requirement for High-Performance Profile Specification Reference +============================================================================================= ================================== ================================= ============================= ======================================== ======================= +`4.2.5 <../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications>`__ n1, n2, n3, n4, n5, n6 1, 2, 3, 4, 5, 6 Gbps Must support Must support **N/A** +`4.2.5 <../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications>`__ nn10, n20, n30, n40, n50, n60 10, 20, 30, 40, 50, 60 Gbps Must support Must support **N/A** +`4.2.5 <../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications>`__ n25, n50, n75, n100, n125, n150 25, 50, 75, 100, 125, 150 Gbps Must support Must support **N/A** +`4.2.5 <../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications>`__ nn50, n100, n150, n200, n250, n300 50, 100, 150, 200, 250, 300 Gbps Must support Must support **N/A** +`4.2.5 <../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications>`__ n100, n200, n300, n400, n500, n600 100, 200, 300, 400, 500, 600 Gbps Must support Must support **N/A** +============================================================================================= ================================== ================================= ============================= ======================================== ======================= + +**Table 2-2:** Reference Model Requirements: Network Interface Specifications + +Cloud Infrastructure Software Profile Requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +======================================================================= ===================== ====================================================================================================================================================== =========================================== ======================================== ============================================================================================ +Reference Model Section Reference Description Requirement for Basic Profile Requirement for High-Performance Profile Specification Reference +======================================================================= ===================== ====================================================================================================================================================== =========================================== ======================================== ============================================================================================ +`5.1.1 <../../../ref_model/chapters/chapter05.md#virtual-compute>`__ infra.com.cfg.001 CPU allocation ratio 1:1 1:1 `ra2.ch.005 `__, `ra2.ch.006 `__ +`5.1.1 <../../../ref_model/chapters/chapter05.md#virtual-compute>`__ infra.com.cfg.002 NUMA awareness Not required Must support `ra2.k8s.006 `__ +`5.1.1 <../../../ref_model/chapters/chapter05.md#virtual-compute>`__ infra.com.cfg.003 CPU pinning capability Not required Must support `ra2.k8s.009 `__ +`5.1.1 <../../../ref_model/chapters/chapter05.md#virtual-compute>`__ infra.com.cfg.004 Huge pages Not required Must support `ra2.ch.001 `__ +`5.1.2 <../../../ref_model/chapters/chapter05.md#virtual-storage>`__ infra.stg.cfg.002 Storage Block Must support Must support `ra2.stg.004 `__ +`5.1.2 <../../../ref_model/chapters/chapter05.md#virtual-storage>`__ infra.stg.cfg.003 Storage with replication Not required Must support **N/A** +`5.1.2 <../../../ref_model/chapters/chapter05.md#virtual-storage>`__ infra.stg.cfg.004 Storage with encryption Must support Must support **N/A** +`5.1.2 <../../../ref_model/chapters/chapter05.md#virtual-storage>`__ infra.stg.acc.cfg.001 Storage IOPS oriented Not required Must support **N/A** +`5.1.2 <../../../ref_model/chapters/chapter05.md#virtual-storage>`__ infra.stg.acc.cfg.002 Storage capacity oriented Not required Not required N/A +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.cfg.001 IO virtualisation using virtio1.1 Must support (1) Must support (1) **N/A** +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.cfg.002 The overlay network encapsulation protocol needs to enable ECMP in the underlay to take advantage of the scale-out features of the network fabric. (2) Must support VXLAN, MPLSoUDP, GENEVE, other *No requirement specified* **N/A** +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.cfg.003 Network Address Translation Must support Must support **N/A** +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.cfg.004 Security Groups Must support Must support `ra2.k8s.014 `__ +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.cfg.005 SFC support Not required Must support **N/A** +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.cfg.006 Traffic patterns symmetry Must support Must support **N/A** +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.acc.cfg.001 vSwitch optimisation Not required Must support DPDK (3) `ra2.ntw.010 `__ +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.acc.cfg.002 Support of HW offload Not required Optional, SmartNic N/A +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.acc.cfg.003 Crypto acceleration Not required Optional N/A +`5.1.3 <../../../ref_model/chapters/chapter05.md#virtual-networking>`__ infra.net.acc.cfg.004 Crypto Acceleration Interface Not required Optional N/A +======================================================================= ===================== ====================================================================================================================================================== =========================================== ======================================== ============================================================================================ + +**Table 2-3:** Reference Model Requirements: Cloud Infrastructure Software Profile Requirements + +**(1)** `Workload Transition Guidelines. <../chapters/appendix-a.md>`__ might have other interfaces (such as SR-IOV VFs to be directly passed to a VM or a Pod) or NIC-specific drivers on guest machines transiently allowed until more mature solutions are available with an acceptable level of efficiency to support telecom workloads (for example regarding CPU and energy consumption). + +**(2)** In Kubernetes based infrastructures network separation is possible without an overlay (e.g.: with IPVLAN) + +**(3)** This feature is not applicable for Kubernetes based infrastructures due to lack of vSwitch however workloads need access to user space networking solutions. + +Cloud Infrastructure Hardware Profile Requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +=========================================================================== ======================== ==================================================================== ============================= ======================================== ============================================================================================ +Reference Model Section Reference Description Requirement for Basic Profile Requirement for High-Performance Profile Specification Reference +=========================================================================== ======================== ==================================================================== ============================= ======================================== ============================================================================================ +`5.4.1 <../../../ref_model/chapters/chapter05.md#compute-resources>`__ infra.hw.cpu.cfg.001 Minimum number of CPU sockets 2 2 `ra2.ch.008 `__ +`5.4.1 <../../../ref_model/chapters/chapter05.md#compute-resources>`__ infra.hw.cpu.cfg.002 Minimum number of Cores per CPU 20 20 `ra2.ch.008 `__ +`5.4.1 <../../../ref_model/chapters/chapter05.md#compute-resources>`__ infra.hw.cpu.cfg.003 NUMA Not required Must support `ra2.k8s.006 `__ +`5.4.1 <../../../ref_model/chapters/chapter05.md#compute-resources>`__ infra.hw.cpu.cfg.004 Simultaneous Multithreading/Symmetric Multiprocessing (SMT/SMP) Must support Optional `ra2.ch.004 `__ +`5.4.1 <../../../ref_model/chapters/chapter05.md#compute-resources>`__ infra.hw.cac.cfg.001 GPU Not required Optional N/A +`5.4.2 <../../../ref_model/chapters/chapter05.md#storage-configurations>`__ infra.hw.stg.hdd.cfg.001 Local Storage HDD *No requirement specified* *No requirement specified* N/A +`5.4.2 <../../../ref_model/chapters/chapter05.md#storage-configurations>`__ infra.hw.stg.ssd.cfg.002 Local Storage SSD Should support Should support `ra2.ch.009 `__ +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.nic.cfg.001 Total Number of NIC Ports available in the host 4 4 `ra2.ch.013 `__ +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.nic.cfg.002 Port speed specified in Gbps (minimum values) 10 25 `ra2.ch.014 `__, `ra2.ch.015 `__ +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.pci.cfg.001 Number of PCIe slots available in the host 8 8 `ra2.ch.016 `__ +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.pci.cfg.002 PCIe speed Gen 3 Gen 3 `ra2.ch.016 `__ +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.pci.cfg.003 PCIe Lanes 8 8 `ra2.ch.016 `__ +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.nac.cfg.001 Cryptographic Acceleration Not required Optional N/A +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.nac.cfg.002 A SmartNIC that is used to offload vSwitch functionality to hardware Not required Optional (1) N/A +`5.4.3 <../../../ref_model/chapters/chapter05.md#network-resources>`__ infra.hw.nac.cfg.003 Compression Optional Optional N/A +=========================================================================== ======================== ==================================================================== ============================= ======================================== ============================================================================================ + +**Table 2-4:** Reference Model Requirements: Cloud Infrastructure Hardware Profile Requirements + +**(1)** There is no vSwitch in case of containers, but a SmartNIC can be used to offload any other network processing. + +Cloud Infrastructure Management Requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +================================================================================================= ========= =========================================================================================== ==================================== ======================= +Reference Model Section Reference Description Requirement (common to all Profiles) Specification Reference +================================================================================================= ========= =========================================================================================== ==================================== ======================= +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.001 Capability to allocate virtual compute resources to a workload Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.002 Capability to allocate virtual storage resources to a workload Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.003 Capability to allocate virtual networking resources to a workload Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.004 Capability to isolate resources between tenants Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.005 Capability to manage workload software images Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.006 Capability to provide information related to allocated virtualised resources per tenant Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.007 Capability to notify state changes of allocated resources Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.008 Capability to collect and expose performance information on virtualised resources allocated Must support **N/A** +`4.1.5 <../../../ref_model/chapters/chapter04.md#cloud-infrastructure-management-capabilities>`__ e.man.009 Capability to collect and notify fault information on virtualised resources Must support **N/A** +================================================================================================= ========= =========================================================================================== ==================================== ======================= + +**Table 2-5:** Reference Model Requirements: Cloud Infrastructure Management Requirements + +Cloud Infrastructure Security Requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +======================================================================================================================== ============ ======================================================================================================================================================================================================================================================================================================================================================================================================== =============================================================================================================================================================================== +Reference Model Section Reference Requirement (common to all Profiles) Specification Reference +======================================================================================================================== ============ ======================================================================================================================================================================================================================================================================================================================================================================================================== =============================================================================================================================================================================== +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.001 The Platform **must** maintain the specified configuration. +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.002 All systems part of Cloud Infrastructure **must** support password hardening as defined in `CIS Password Policy Guide `__. Hardening: CIS Password Policy Guide `5.3.1 Node Hardening: Securing Kubernetes Hosts <./chapter05.md#node-hardening-securing-kubernetes-hosts>`__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.003 All servers part of Cloud Infrastructure **must** support a root of trust and secure boot. +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.004 The Operating Systems of all the servers part of Cloud Infrastructure **must** be hardened by removing or disabling unnecessary services, applications and network protocols, configuring operating system user authentication, configuring resource controls, installing and configuring additional security controls where needed, and testing the security of the Operating System. (NIST SP 800-123) `5.2 Principles <./chapter05.md#principles>`__ and `5.3 Node Hardening <./chapter05.md#node-hardening>`__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.005 The Platform **must** support Operating System level access control `5.3 Node Hardening <./chapter05.md#node-hardening>`__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.006 The Platform **must** support Secure logging. Logging with root account must be prohibited when root privileges are not required. `5.3.2 Restrict direct access to nodes <./chapter05.md#restrict-direct-access-to-nodes>`__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.007 All servers part of Cloud Infrastructure **must** be Time synchronized with authenticated Time service. +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.008 All servers part of Cloud Infrastructure **must** be regularly updated to address security vulnerabilities. `5.3.3 Vulnerability assessment <./chapter05.md#vulnerability-assessment>`__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.009 The Platform **must** support Software integrity protection and verification and **must** scan source code and manifests. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.010 The Cloud Infrastructure **must** support encrypted storage, for example, block, object and file storage, with access to encryption keys restricted based on a need to know. `Controlled Access Based on the Need to Know `__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.011 The Cloud Infrastructure **should** support Read and Write only storage partitions (write only permission to one or more authorized actors). +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.012 The Operator **must** ensure that only authorized actors have physical access to the underlying infrastructure. +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.013 The Platform **must** ensure that only authorized actors have logical access to the underlying infrastructure. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.014 All servers part of Cloud Infrastructure **should** support measured boot and an attestation server that monitors the measurements of the servers. +`7.9.1 <../../../ref_model/chapters/chapter07.md#system-hardening>`__ sec.gen.015 Any change to the Platform must be logged as a security event, and the logged event must include the identity of the entity making the change, the change, the date and the time of the change. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.001 The Platform **must** support authenticated and secure access to API, GUI and command line interfaces. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.002 The Platform **must** support Traffic Filtering for workloads (for example, Firewall). +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.003 The Platform **must** support Secure and encrypted communications, and confidentiality and integrity of network traffic. `5.4.3 Use Transport Layer Security and Service Mesh <./chapter05.md#use-transport-layer-security-and-service-mesh>`__ +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.004 The Cloud Infrastructure **must** support authentication, integrity and confidentiality on all network channels. `5.4.3 Use Transport Layer Security and Service Mesh <./chapter05.md#use-transport-layer-security-and-service-mesh>`__ +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.005 The Cloud Infrastructure **must** segregate the underlay and overlay networks. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.006 The Cloud Infrastructure must be able to utilise the Cloud Infrastructure Manager identity lifecycle management capabilities. `5.2 Principles <./chapter05.md#principles>`__ +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.007 The Platform **must** implement controls enforcing separation of duties and privileges, least privilege use and least common mechanism (Role-Based Access Control). `5.2 Principles <./chapter05.md#principles>`__ and `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.008 The Platform **must** be able to assign the Entities that comprise the tenant networks to different trust domains. Communication between different trust domains is not allowed, by default. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.009 The Platform **must** support creation of Trust Relationships between trust domains. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.010 For two or more domains without existing trust relationships, the Platform **must not** allow the effect of an attack on one domain to impact the other domains either directly or indirectly. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.011 The Platform **must not** reuse the same authentication credential (e.g., key-pair) on different Platform components (e.g., on different hosts, or different services). +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.012 The Platform **must** protect all secrets by using strong encryption techniques, and storing the protected secrets externally from the component +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.013 The Platform **must** provide secrets dynamically as and when needed. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.014 The Platform **should** use Linux Security Modules such as SELinux to control access to resources. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.015 The Platform **must not** contain back door entries (unpublished access points, APIs, etc.). +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.016 Login access to the platform's components **must** be through encrypted protocols such as SSH v2 or TLS v1.2 or higher. Note: Hardened jump servers isolated from external networks are recommended `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.017 The Platform **must** provide the capability of using digital certificates that comply with X.509 standards issued by a trusted Certification Authority. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.018 The Platform **must** provide the capability of allowing certificate renewal and revocation. +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.019 The Platform **must** provide the capability of testing the validity of a digital certificate (CA signature, validity period, non revocation, identity). +`7.9.2 <../../../ref_model/chapters/chapter07.md#platform-and-access>`__ sec.sys.020 The Cloud Infrastructure architecture **should** rely on Zero Trust principles to build a secure by design environment. +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.001 The Platform **must** support Confidentiality and Integrity of data at rest and in-transit. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.002 The Platform **should** support self-encrypting storage devices. +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.003 The Platform **must** support Confidentiality and Integrity of data related metadata. +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.004 The Platform **must** support Confidentiality of processes and restrict information sharing with only the process owner (e.g., tenant). +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.005 The Platform **must** support Confidentiality and Integrity of process-related metadata and restrict information sharing with only the process owner (e.g., tenant). +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.006 The Platform **must** support Confidentiality and Integrity of workload resource utilization (RAM, CPU, Storage, Network I/O, cache, hardware offload) and restrict information sharing with only the workload owner (e.g., tenant). +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.007 The Platform **must not** allow Memory Inspection by any actor other than the authorized actors for the Entity to which Memory is assigned (e.g., tenants owning the workload), for Lawful Inspection, and by secure monitoring services. +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.008 The Cloud Infrastructure **must** support tenant networks segregation. `5.7 Create and define Network Policies <./chapter05.md#create-and-define-network-policies>`__ +`7.9.3 <../../../ref_model/chapters/chapter07.md#confidentiality-and-integrity>`__ sec.ci.009 For sensitive data encryption, the key management service **should** leverage a Hardware Security Module to manage and protect cryptographic keys. +`7.9.4 <../../../ref_model/chapters/chapter07.md#workload-security>`__ sec.wl.001 The Platform **must** support Workload placement policy. +`7.9.4 <../../../ref_model/chapters/chapter07.md#workload-security>`__ sec.wl.002 The Cloud Infrastructure **must** provide methods to ensure the platform’s trust status and integrity (e.g. remote attestation, Trusted Platform Module). +`7.9.4 <../../../ref_model/chapters/chapter07.md#workload-security>`__ sec.wl.003 The Platform **must** support secure provisioning of workloads. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.4 <../../../ref_model/chapters/chapter07.md#workload-security>`__ sec.wl.004 The Platform **must** support Location assertion (for mandated in-country or location requirements). +`7.9.4 <../../../ref_model/chapters/chapter07.md#workload-security>`__ sec.wl.005 The Platform **must** support the separation of production and non-production Workloads. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.4 <../../../ref_model/chapters/chapter07.md#workload-security>`__ sec.wl.006 The Platform **must** support the separation of Workloads based on their categorisation (for example, payment card information, healthcare, etc.). `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ and `5.6 Separate Sensitive Workload <./chapter05.md#separate-sensitive-workload>`__ +`7.9.4 <../../../ref_model/chapters/chapter07.md#workload-security>`__ sec.wl.007 The Operator **must** implement processes and tools to verify VNF authenticity and integrity. `5.13 Trusted Registry <./chapter05.md#trusted-registry>`__ +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.001 Images from untrusted sources **must not** be used. `5.13 Trusted Registry <./chapter05.md#trusted-registry>`__ +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.002 Images **must** be scanned to be maintained free from known vulnerabilities. `5.13 Trusted Registry <./chapter05.md#trusted-registry>`__ +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.003 Images **must not** be configured to run with privileges higher than the privileges of the actor authorized to run them. `5.11 Run-Time Security <./chapter05.md#run-time-security>`__ +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.004 Images **must** only be accessible to authorized actors. +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.005 Image Registries **must** only be accessible to authorized actors. +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.006 Image Registries **must** only be accessible over secure networks that enforce authentication, integrity and confidentiality. `5.13 Trusted Registry <./chapter05.md#trusted-registry>`__ +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.007 Image registries **must** be clear of vulnerable and out of date versions. `5.13 Trusted Registry <./chapter05.md#trusted-registry>`__ +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.008 Images **must not** include any secrets. Secrets include passwords, cloud provider credentials, SSH keys, TLS certificate keys, etc. `5.12 Secrets Management <./chapter05.md#secrets-management>`__ +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.009 CIS Hardened Images **should** be used whenever possible. +`7.9.5 <../../../ref_model/chapters/chapter07.md#image-security>`__ sec.img.010 Minimalist base images **should** be used whenever possible. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.001 The Platform **must** support Secure Provisioning, Availability, and Deprovisioning (Secure Clean-Up) of workload resources where Secure Clean-Up includes tear-down, defense against virus or other attacks. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.002 Cloud operations staff and systems **must** use management protocols limiting security risk such as SNMPv3, SSH v2, ICMP, NTP, syslog and TLS v1.2 or higher. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.003 The Cloud Operator **must** implement and strictly follow change management processes for Cloud Infrastructure, Cloud Infrastructure Manager and other components of the cloud, and Platform change control on hardware. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.004 The Cloud Operator **should** support automated templated approved changes. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.005 Platform **must** provide logs and these logs must be regularly monitored for anomalous behavior. `5.10 Enable Logging and Monitoring <./chapter05.md#enable-logging-and-monitoring>`__ +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.006 The Platform **must** verify the integrity of all Resource management requests. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.007 The Platform **must** be able to update newly instantiated, suspended, hibernated, migrated and restarted images with current time information. `5.4 Securing Kubernetes orchestrator <./chapter05.md#securing-kubernetes-orchestrator>`__ +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.008 The Platform **must** be able to update newly instantiated, suspended, hibernated, migrated and restarted images with relevant DNS information. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.009 The Platform **must** be able to update the tag of newly instantiated, suspended, hibernated, migrated and restarted images with relevant geolocation (geographical) information. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.010 The Platform **must** log all changes to geolocation along with the mechanisms and sources of location information (i.e. GPS, IP block, and timing). +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.011 The Platform **must** implement Security life cycle management processes including the proactive update and patching of all deployed Cloud Infrastructure software. +`7.9.6 <../../../ref_model/chapters/chapter07.md#security-lcm>`__ sec.lcm.012 The Platform **must** log any access privilege escalation. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.001 Platform **must** provide logs and these logs must be regularly monitored for events of interest. The logs **must** contain the following fields: event type, date/time, protocol, service or program used for access, success/failure, login ID or process ID, IP address and ports (source and destination) involved. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.002 Security logs **must** be time synchronised. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.003 The Platform **must** log all changes to time server source, time, date and time zones. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.004 The Platform **must** secure and protect Audit logs (containing sensitive information) both in-transit and at rest. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.005 The Platform **must** Monitor and Audit various behaviours of connection and login attempts to detect access attacks and potential access attempts and take corrective actions accordingly. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.006 The Platform **must** Monitor and Audit operations by authorized account access after login to detect malicious operational activity and take corrective actions accordingly. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.007 The Platform **must** Monitor and Audit security parameter configurations for compliance with defined security policies. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.008 The Platform **must** Monitor and Audit externally exposed interfaces for illegal access (attacks) and take corrective security hardening measures. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.009 The Platform **must** Monitor and Audit service handling for various attacks (malformed messages, signalling flooding and replaying, etc.) and take corrective actions accordingly. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.010 The Platform **must** Monitor and Audit running processes to detect unexpected or unauthorized processes and take corrective actions accordingly. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.011 The Platform **must** Monitor and Audit logs from infrastructure elements and workloads to detected anomalies in the system components and take corrective actions accordingly. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.012 The Platform **must** Monitor and Audit Traffic patterns and volumes to prevent malware download attempts. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.013 The monitoring system **must not** affect the security (integrity and confidentiality) of the infrastructure, workloads, or the user data (through back door entries). +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.014 The Monitoring systems **should not** impact IAAS, PAAS, and SAAS SLAs including availability SLAs. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.015 The Platform **must** ensure that the Monitoring systems are never starved of resources and **must** activate alarms when resource utilisation exceeds a configurable threshold. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.016 The Platform Monitoring components **should** follow security best practices for auditing, including secure logging and tracing. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.017 The Platform **must** audit systems for any missing security patches and take appropriate actions. `5.3.3 Vulnerability assessment <./chapter05.md#vulnerability-assessment>`__ +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.018 The Platform, starting from initialization, **must** collect and analyze logs to identify security events, and store these events in an external system. `5.3.4 Patch management <./chapter05.md#patch-management>`__ +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.019 The Platform’s components **must not** include an authentication credential, e.g., password, in any logs, even if encrypted. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.020 The Platform’s logging system **must** support the storage of security audit logs for a configurable period of time. +`7.9.7 <../../../ref_model/chapters/chapter07.md#monitoring-and-security-audit>`__ sec.mon.021 The Platform **must** store security events locally if the external logging system is unavailable and shall periodically attempt to send these to the external logging system until successful. +`7.9.8 <../../../ref_model/chapters/chapter07.md#open-source-software>`__ sec.oss.001 Open source code **must** be inspected by tools with various capabilities for static and dynamic code analysis. `5.3.3 Vulnerability assessment <./chapter05.md#vulnerability-assessment>`__ +`7.9.8 <../../../ref_model/chapters/chapter07.md#open-source-software>`__ sec.oss.002 The `CVE (Common Vulnerabilities and Exposures) `__ **must** be used to identify vulnerabilities and their severity rating for open source code part of Cloud Infrastructure and workloads software. +`7.9.8 <../../../ref_model/chapters/chapter07.md#open-source-software>`__ sec.oss.003 Critical and high severity rated vulnerabilities **must** be fixed in a timely manner. Refer to the `CVSS (Common Vulnerability Scoring System) `__ to know a vulnerability score and its associated rate (low, medium, high, or critical). +`7.9.8 <../../../ref_model/chapters/chapter07.md#open-source-software>`__ sec.oss.004 A dedicated internal isolated repository separated from the production environment **must** be used to store vetted open source content. `5.13 Trusted Registry <./chapter05.md#trusted-registry>`__ +`7.9.8 <../../../ref_model/chapters/chapter07.md#open-source-software>`__ sec.oss.005 A Software Bill of Materials (`SBOM `__) **should** be provided or build, and maintained to identify the software components and their origins. +`7.9.9 <../../../ref_model/chapters/chapter07.md#iaac---secure-design-and-architecture-stage-requirements>`__ sec.arch.001 Threat Modelling methodologies and tools **should** be used during the Secure Design and Architecture stage triggered by Software Feature Design trigger. It may be done manually or using tools like open source OWASP Threat Dragon +`7.9.9 <../../../ref_model/chapters/chapter07.md#iaac---secure-design-and-architecture-stage-requirements>`__ sec.arch.002 Security Control Baseline Assessment **should** be performed during the Secure Design and Architecture stage triggered by Software Feature Design trigger. Typically done manually by internal or independent assessors. +`7.9.10 <../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements>`__ sec.code.001 SAST -Static Application Security Testing **must** be applied during Secure Coding stage triggered by Pull, Clone or Comment trigger. Security testing that analyses application source code for software vulnerabilities and gaps against best practices. Example: open source OWASP range of tools. +`7.9.10 <../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements>`__ sec.code.002 SCA – Software Composition Analysis **should** be applied during Secure Coding stage triggered by Pull, Clone or Comment trigger. Security testing that analyses application source code or compiled code for software components with known vulnerabilities. Example: open source OWASP range of tools. +`7.9.10 <../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements>`__ sec.code.003 Source Code Review **should** be performed continuously during Secure Coding stage. Typically done manually. +`7.9.10 <../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements>`__ sec.code.004 Integrated SAST via IDE Plugins **should** be used during Secure Coding stage triggered by Developer Code trigger. On the local machine: through the IDE or integrated test suites; triggered on completion of coding be developer. +`7.9.10 <../../../ref_model/chapters/chapter07.md#iaac---secure-code-stage-requirements>`__ sec.code.005 SAST of Source Code Repo **should** be performed during Secure Coding stage triggered by Developer Code trigger. Continuous delivery pre-deployment: scanning prior to deployment. +`7.9.11 <../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements>`__ sec.bld.001 SAST -Static Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Build and Integrate trigger. Example: open source OWASP range of tools. +`7.9.11 <../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements>`__ sec.bld.002 SCA – Software Composition Analysis **should** be applied during the Continuous Build, Integration and Testing stage triggered by Build and Integrate trigger. Example: open source OWASP range of tools. +`7.9.11 <../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements>`__ sec.bld.003 Image Scan **must** be applied during the Continuous Build, Integration and Testing stage triggered by Package trigger. Example: A push of a container image to a container registry may trigger a vulnerability scan before the image becomes available in the registry. +`7.9.11 <../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements>`__ sec.bld.004 DAST – Dynamic Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Stage & Test trigger. Security testing that analyses a running application by exercising application functionality and detecting vulnerabilities based on application behaviour and response. Example: OWASP ZAP. +`7.9.11 <../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements>`__ sec.bld.005 Fuzzing **should** be applied during the Continuous Build, Integration and testing stage triggered by Stage & Test trigger. Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. Example: GitLab Open Sources Protocol Fuzzer Community Edition. +`7.9.11 <../../../ref_model/chapters/chapter07.md#iaac---continuous-build-integration-and-testing-stage-requirements>`__ sec.bld.006 IAST – Interactive Application Security Testing **should** be applied during the Continuous Build, Integration and Testing stage triggered by Stage & Test trigger. Software component deployed with an application that assesses application behaviour and detects presence of vulnerabilities on an application being exercised in realistic testing scenarios. Example: Contrast Community Edition. +`7.9.12 <../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements>`__ sec.del.001 Image Scan **must** be applied during the Continuous Delivery and Deployment stage triggered by Publish to Artifact and Image Repository trigger. Example: GitLab uses the open-source Clair engine for container image scanning. +`7.9.12 <../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements>`__ sec.del.002 Code Signing **must** be applied during the Continuous Delivery and Deployment stage triggered by Publish to Artifact and Image Repository trigger. Code Signing provides authentication to assure that downloaded files are form the publisher named on the certificate. +`7.9.12 <../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements>`__ sec.del.003 Artifact and Image Repository Scan **should** be continuously applied during the Continuous Delivery and Deployment stage. Example: GitLab uses the open source Clair engine for container scanning. +`7.9.12 <../../../ref_model/chapters/chapter07.md#iaac---continuous-delivery-and-deployment-stage-requirements>`__ sec.del.004 Component Vulnerability Scan **must** be applied during the Continuous Delivery and Deployment stage triggered by Instantiate Infrastructure trigger. The vulnerability scanning system is deployed on the cloud platform to detect security vulnerabilities of specified components through scanning and to provide timely security protection. Example: OWASP Zed Attack Proxy (ZAP). +`7.9.13 <../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements>`__ sec.run.001 Component Vulnerability Monitoring **must** be continuously applied during the Runtime Defence and Monitoring stage and remediation actions **must** be applied for high severity rated vulnerabilities. Security technology that monitors components like virtual servers and assesses data, applications, and infrastructure for security risks. +`7.9.13 <../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements>`__ sec.run.002 RASP – Runtime Application Self-Protection **should** be continuously applied during the Runtime Defence and Monitoring stage. Security technology deployed within the target application in production for detecting, alerting, and blocking attacks. +`7.9.13 <../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements>`__ sec.run.003 Application testing and Fuzzing **should** be continuously applied during the Runtime Defence and Monitoring stage. Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. Example: GitLab Open Sources Protocol Fuzzer Community Edition. +`7.9.13 <../../../ref_model/chapters/chapter07.md#iaac---runtime-defence-and-monitoring-requirements>`__ sec.run.004 Penetration Testing **should** be continuously applied during the Runtime Defence and Monitoring stage. Typically done manually. +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.001 The Cloud Operator **should** comply with Center for Internet Security CIS Controls (`https://www.cisecurity.org/ `__) +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.002 The Cloud Operator, Platform and Workloads **should** follow the guidance in the CSA Security Guidance for Critical Areas of Focus in Cloud Computing (latest version) `https://cloudsecurityalliance.org/ `__ +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.003 The Platform and Workloads **should** follow the guidance in the `OWASP Cheat Sheet Series (OCSS) `__ +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.004 The Cloud Operator, Platform and Workloads **should** ensure that their code is not vulnerable to the OWASP Top Ten Security Risks `https://owasp.org/www-project-top-ten/ `__ +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.005 The Cloud Operator, Platform and Workloads **should** strive to improve their maturity on the `OWASP Software Maturity Model (SAMM) `__ +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.006 The Cloud Operator, Platform and Workloads **should** utilize the `OWASP Web Security Testing Guide `__ +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.007 The Cloud Operator, and Platform **should** satisfy the requirements for Information Management Systems specified in `ISO/IEC 27001 `__. ISO/IEC 27002:2013 - ISO/IEC 27001 is the international Standard for best-practice information security management systems (ISMSs). +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.008 The Cloud Operator, and Platform **should** implement the Code of practice for Security Controls specified `ISO/IEC 27002:2013 (or latest) `__ +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.009 The Cloud Operator, and Platform **should** implement the `ISO/IEC 27032:2012 (or latest) `__ Guidelines for Cybersecurity techniques. ISO/IEC 27032 - ISO/IEC 27032 is the international Standard focusing explicitly on cybersecurity. +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.010 The Cloud Operator **should** conform to the ISO/IEC 27035 standard for incidence management. ISO/IEC 27035 - ISO/IEC 27035 is the international Standard for incident management. +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.011 The Cloud Operator **should** conform to the ISO/IEC 27031 standard for business continuity. ISO/IEC 27031 - ISO/IEC 27031 is the international Standard for ICT readiness for business continuity. +`7.9.14 <../../../ref_model/chapters/chapter07.md#compliance-with-standards>`__ sec.std.012 The Public Cloud Operator **must**, and the Private Cloud Operator **may** be certified to be compliant with the International Standard on Awareness Engagements (ISAE) 3402 (in the US: SSAE 16). International Standard on Awareness Engagements (ISAE) 3402. US Equivalent: SSAE16. +======================================================================================================================== ============ ======================================================================================================================================================================================================================================================================================================================================================================================================== =============================================================================================================================================================================== + +**Table 2-6:** Reference Model Requirements: Cloud Infrastructure Security Requirements + +Kubernetes Architecture Requirements +------------------------------------ + +The requirements in this section are to be delivered in addition to those in `section 2.2 <#2.2>`__, and have been created to support the Principles defined in `Chapter 1 of this Reference Architecture <./chapter01.md>`__. + +The Reference Model (RM) defines the Cloud Infrastructure, which consists of the physical resources, virtualised resources and a software management system. + +In virtualisation platforms, the Cloud Infrastructure consists of the Guest Operating System, Hypervisor and, if needed, other software such as libvirt. The Cloud Infrastructure Management component is responsible for, among others, tenant management, resources management, inventory, scheduling, and access management. + +With regards to containerisation platforms, the scope of the following Architecture requirements include the Cloud Infrastructure Hardware (e.g. physical resources), Cloud Infrastructure Software (e.g. Hypervisor (optional), Container Runtime, virtual or container Orchestrator(s), Operating System), and infrastructure resources consumed by virtual machines or containers. + +========== ================== ======================= ================================================================================================================================================================================================================================================================================================================== =================================================================================================================================================================================================================================================================== +Reference Category Sub-category Description Specification Reference +========== ================== ======================= ================================================================================================================================================================================================================================================================================================================== =================================================================================================================================================================================================================================================================== +gen.cnt.02 General Cloud nativeness The Architecture **must** support immutable infrastructure. `ra2.ch.017 `__ +gen.cnt.03 General Cloud nativeness The Architecture **must** run conformant Kubernetes as defined by the `CNCF `__. `ra2.k8s.001 `__ +gen.cnt.04 General Cloud nativeness The Architecture **must** support clearly defined abstraction layers. +gen.cnt.05 General Cloud nativeness The Architecture **should** support configuration of all components in an automated manner using openly published API definitions. +gen.scl.01 General Scalability The Architecture **should** support policy driven horizontal auto-scaling of workloads. +gen.rsl.01 General Resiliency The Architecture **must** support resilient Kubernetes components that are required for the continued availability of running workloads. `ra2.k8s.004 `__ +gen.rsl.02 General Resiliency The Architecture **should** support resilient Kubernetes service components that are not subject to gen.rsl.01. `ra2.k8s.002 `__, `ra2.k8s.003 `__ +gen.avl.01 General Availability The Architecture **must** provide High Availability for Kubernetes components. `ra2.k8s.002 `__, `ra2.k8s.003 `__, `ra2.k8s.004 `__ +gen.ost.01 General Openness The Architecture **should** embrace open-based standards and technologies. `ra2.crt.001 `__, `ra2.crt.002 `__, `ra2.ntw.002 `__, `ra2.ntw.006 `__, `ra2.ntw.007 `__ +inf.com.01 Infrastructure Compute The Architecture **must** provide compute resources for Pods. `ra2.k8s.004 `__ +inf.stg.01 Infrastructure Storage The Architecture **must** support the ability for an operator to choose whether or not to deploy persistent storage for Pods. `ra2.stg.004 `__ +inf.ntw.01 Infrastructure Network The Architecture **must** support network resiliency on the Kubernetes nodes. +inf.ntw.02 Infrastructure Network The Architecture **must** support fully redundant network connectivity to the Kubernetes nodes, leveraging multiple network connections. +inf.ntw.03 Infrastructure Network The networking solution **should** be able to be centrally administrated and configured. `ra2.ntw.001 `__, `ra2.ntw.004 `__ +inf.ntw.04 Infrastructure Network The Architecture **must** support dual stack IPv4 and IPv6 for Kubernetes workloads. `ra2.ch.007 `__, `ra2.k8s.010 `__ +inf.ntw.05 Infrastructure Network The Architecture **must** support capabilities for integrating SDN controllers. +inf.ntw.06 Infrastructure Network The Architecture **must** support more than one networking solution. `ra2.ntw.005 `__, `ra2.ntw.007 `__ +inf.ntw.07 Infrastructure Network The Architecture **must** support the ability for an operator to choose whether or not to deploy more than one networking solution. `ra2.ntw.005 `__ +inf.ntw.08 Infrastructure Network The Architecture **must** provide a default network which implements the Kubernetes network model. `ra2.ntw.002 `__ +inf.ntw.09 Infrastructure Network The networking solution **must not** interfere with or cause interference to any interface or network it does not own. +inf.ntw.10 Infrastructure Network The Architecture **must** support Cluster wide coordination of IP address assignment. +inf.ntw.13 Infrastructure Network The platform **must** allow specifying multiple separate IP pools. Tenants are required to select at least one IP pool that is different from the control infrastructure IP pool or other tenant IP pools. +inf.ntw.14 Infrastructure Network The platform **must** allow NATless traffic (i.e. exposing the pod IP address directly to the outside), allowing source and destination IP addresses to be preserved in the traffic headers from workloads to external networks. This is needed e.g. for signaling applications, using SIP and Diameter protocols. `ra2.ntw.011 `__ +inf.ntw.15 Infrastructure Network The platform **must** support LoadBalancer `Publishing Service (ServiceType) `__ +inf.ntw.16 Infrastructure Network The platform **must** support `Ingress `__. +inf.ntw.17 Infrastructure Network The platform **should** support NodePort `Publishing Service (ServiceTypes) `__. +inf.ntw.18 Infrastructure Network The platform **should** support ExternalName `Publishing Service (ServiceTypes) `__. +inf.vir.01 Infrastructure Virtual Infrastructure The Architecture **must** support the capability for Containers to consume infrastructure resources abstracted by Host Operating Systems that are running within a virtual machine. `ra2.ch.005 `__, `ra2.ch.011 `__ +inf.phy.01 Infrastructure Physical Infrastructure The Architecture **must** support the capability for Containers to consume infrastructure resources abstracted by Host Operating Systems that are running within a physical server. ra2.ch.008 +kcm.gen.01 Kubernetes Cluster General The Architecture **must** support policy driven horizontal auto-scaling of Kubernetes Cluster. **N/A** +kcm.gen.02 Kubernetes Cluster General The Architecture **must** enable workload resiliency. `ra2.k8s.004 `__ +int.api.01 API General The Architecture **must** leverage the Kubernetes APIs to discover and declaratively manage compute (virtual and bare metal resources), network, and storage. For Networking: `ra2.ntw.001 `__, `ra2.ntw.008 `__, `ra2.app.006 `__ Compute/storage not yet met. +int.api.02 API General The Architecture **must** support the usage of a Kubernetes Application package manager using the Kubernetes API, like Helm v3. `ra2.pkg.001 `__ +int.api.03 API General The Architecture **must** support stable features in its APIs. +int.api.04 API General The Architecture **must** support limited backward compatibility in its APIs. Support for the whole API must not be dropped, but the schema or other details can change. +========== ================== ======================= ================================================================================================================================================================================================================================================================================================================== =================================================================================================================================================================================================================================================================== + +**Table 2-7:** Kubernetes Architecture Requirements diff --git a/doc/ref_arch/kubernetes/chapters/chapter03.md b/doc/ref_arch/kubernetes/chapters/chapter03.md index e280e9bd45..1b1ed69aac 100644 --- a/doc/ref_arch/kubernetes/chapters/chapter03.md +++ b/doc/ref_arch/kubernetes/chapters/chapter03.md @@ -1,34 +1,7 @@ -[<< Back](../../kubernetes) - -# 3. High Level Architecture -

scope

- -## Table of Contents - -* [3.1 Introduction](#31-introduction) -* [3.2 Infrastructure Services](#32-infrastructure-services) - * [3.2.1 Container Compute Services](#321-container-compute-services) - * [3.2.1.1 Container Runtime Services](#3211-container-runtime-services) - * [3.2.1.2 CPU Management](#3212-cpu-management) - * [3.2.1.3 Memory and Huge Pages Resources Management](#3213-memory-and-huge-pages-resources-management) - * [3.2.1.4 Hardware Topology Management](#3214-hardware-topology-management) - * [3.2.1.5 Node Feature Discovery](#3215-node-feature-discovery) - * [3.2.1.6 Device Plugin Framework](#3216-device-plugin-framework) - * [3.2.1.7 Hardware Acceleration](#3217-hardware-acceleration) - * [3.2.1.8 Scheduling Pods with Non-resilient Applications](#3218-scheduling-pods-with-non-resilient-applications) - * [3.2.1.9 Virtual Machine based Clusters](#3219-virtual-machine-based-clusters) - * [3.2.2 Container Networking Services](#322-container-networking-services) - * [3.2.2.1 Kubernetes Networking Semantics](#3.2.2.1) - * [3.2.2.1.1 Built in Kubernetes Network Functionality](#3.2.2.1.1) - * [3.2.2.1.2 Multiple Networks and Advanced Configurations](#3.2.2.1.2) - * [3.2.3 Container Storage Services](#323-container-storage-services) - * [3.2.4 Kubernetes Application package manager](#324-kubernetes-application-package-managers) - * [3.2.5 Custom Resources](#325-custom-resources) - * [3.2.5.1 Operator Pattern](#3251-operator-pattern) -* [3.3 CaaS Manager - Cluster Lifecycle Management](#33-caas-manager---cluster-lifecycle-management) - - -## 3.1 Introduction + +# High Level Architecture + +## Introduction The Anuket Kubernetes Reference Architecture (RA) is intended to be an industry standard independent Kubernetes reference architecture that is not tied to any @@ -65,27 +38,25 @@ instance profile (Basic and Network Intensive). For more information on the instance profiles please refer to [RM Chapter 4, section 4.2.4](../../../ref_model/chapters/chapter04.md#4.2.4). -

Figure 5-3 (from RM): NFVI software -profiles

+![**Figure 5-3 (from RM):** NFVI softwareprofiles](../../../ref_model/figures/RM_chap5_fig_5_3_SW_profile.png) + +**Figure 5-3 (from RM):** NFVI softwareprofiles In addition, the RM Figure 5-4 (shown below) depicts the hardware profile features that apply to each instance profile. -

Figure 5-4 (from RM): NFVI hardware -profiles and host associated capabilities

+![**Figure 5-4 (from RM):** NFVI hardwareprofiles and host associated capabilities](../../../ref_model/figures/RM_chap5_fig_5_4_HW_profile.png) + +**Figure 5-4 (from RM):** NFVI hardwareprofiles and host associated capabilities The features and capabilities described in the software and hardware profiles are considered throughout this RA, with the RA requirements traceability to the RM requirements formally documented in [chapter 2, section 2.2](./chapter02.md#2.2) of this RA. -## 3.2 Infrastructure Services +## Infrastructure Services -### 3.2.1 Container Compute Services +### Container Compute Services The primary interface between the Physical / Virtual Infrastructure and any container-relevant components is the Kubernetes Node Operating System. This is @@ -93,13 +64,14 @@ the OS within which the container runtime exists, and within which the containers run (and therefore, the OS whose kernel is shared by the referenced containers). This is shown in Figure 3-1 below. -

Kubernetes Host -Operating System

-

Figure 3-1: Kubernetes Node Operating System

+![**Figure 3-1:** Kubernetes Node Operating System](../figures/ch03_hostOS.png) + +**Figure 3-1:** Kubernetes Node Operating System The Kubernetes Node OS (as with any OS) consists of two main components: -- Kernel space -- User space + +* Kernel space +* User space The Kernel is the tightly controlled space that provides an API to applications running in the user space (which usually have their own southbound interface in @@ -124,8 +96,7 @@ set of processes that are used to manage these containers (pull, run, delete, etc.), and the kernel features required to provide the isolation mechanisms (cgroups, namespaces, filesystems, etc.) between the components. - -#### 3.2.1.1 Container Runtime Services +#### Container Runtime Services The Container Runtime is the component that runs within a Kubernetes Node Operating System (OS) and manages the underlying OS functionality, such as @@ -190,20 +161,16 @@ Pod and workloads | Description [CronJob:](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) | A CronJob manages time-based Jobs, namely: once at a specified point in time and repeatedly at a specified point in time [StatefulSet:](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) | StatefulSet represents a set of pods with consistent identities. Identities are defined as: network, storage. - - -#### 3.2.1.2 CPU Management +#### CPU Management CPU management has policies to determine placement preferences to use for workloads that are sensitive to cache affinity or latency, and so the workloads must not be moved by OS scheduler or throttled by kubelet. Additionally, some workloads are sensitive to differences between physical cores and SMT, while others (like DPDK-based workloads) are designed to run on isolated CPUs (like on Linux with cpuset-based selection of CPUs and isolcpus kernel parameter specifying cores isolated from general SMP balancing and scheduler algorithms). Kubernetes [CPU Manager](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/) works with Topology Manager. Special care needs to be taken of: -• Supporting isolated CPUs: Using kubelet [Reserved CPUs](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#explicitly-reserved-cpu-list) and Linux isolcpus allows configuration where only isolcpus are allocatable to pods. Scheduling pods to such nodes can be influenced with taints, tolerations and node affinity. - -• Differentiating between physical cores and SMT: When requesting even number of CPU cores for pods, scheduling can be influenced with taints, tolerations, and node affinity. +* Supporting isolated CPUs: Using kubelet [Reserved CPUs](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#explicitly-reserved-cpu-list) and Linux isolcpus allows configuration where only isolcpus are allocatable to pods. Scheduling pods to such nodes can be influenced with taints, tolerations and node affinity. +* Differentiating between physical cores and SMT: When requesting even number of CPU cores for pods, scheduling can be influenced with taints, tolerations, and node affinity. - -#### 3.2.1.3 Memory and Huge Pages Resources Management +#### Memory and Huge Pages Resources Management The Reference Model requires the support of huge pages in i.cap.018 which is supported by upstream Kubernetes ([documentation](https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/)). @@ -213,8 +180,7 @@ For some applications, huge pages should be allocated to account for consideration of the underlying HW topology. [The Memory Manager](https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/) (added to Kubernetes v1.21 as alpha feature) enables the feature of guaranteed memory and huge pages allocation for pods in the Guaranteed QoS class. The Memory Manager feeds the Topology Manager with hints for most suitable NUMA affinity. - -#### 3.2.1.4 Hardware Topology Management +#### Hardware Topology Management Scheduling pods across NUMA boundaries can result in lower performance and higher latencies. This would be an issue for applications that require optimisations of CPU isolation, memory and device locality. @@ -222,40 +188,36 @@ Kubernetes supports Topology policy per node as beta feature ([documentation](ht If case that memory or huge pages are not considered by the Topology Manager, it can be done by the operating system providing best-effort local page allocation for containers as long as there is sufficient free local memory on the node, or with Control Groups (cgroups) cpuset subsystem that can isolate memory to single NUMA node. - -#### 3.2.1.5 Node Feature Discovery +#### Node Feature Discovery [Node Feature Discovery](https://kubernetes-sigs.github.io/node-feature-discovery/stable/get-started/index.html) (NFD) can run on every node as a daemon or as a job. NFD detects detailed hardware and software capabilities of each node and then advertises those capabilities as node labels. Those node labels can be used in scheduling pods by using Node Selector or Node Affinity for pods that require such capabilities. - -#### 3.2.1.6 Device Plugin Framework +#### Device Plugin Framework [Device Plugin Framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) advertises device hardware resources to kubelet with which vendors can implement plugins for devices that may require vendor-specific activation and life cycle management, and securely maps these devices to containers. Figure 3-2 shows in four steps how device plugins operate on a Kubernetes node: -* 1: During setup, the cluster administrator (more in [3.2.5.1 Operator Pattern](chapter03.md#3251-operator-pattern)) knows or discovers (as per [3.2.1.5 Node Feature Discovery](chapter03.md#3215-node-feature-discovery)) what kind of devices are present on the different nodes, selects which devices to enable and deploys the associated device plugins. +* 1: During setup, the cluster administrator (more in [3.2.5.1 Operator Pattern](chapter03.md#operator-pattern)) knows or discovers (as per [3.2.1.5 Node Feature Discovery](chapter03.md#node-feature-discovery)) what kind of devices are present on the different nodes, selects which devices to enable and deploys the associated device plugins. * 2: The plugin reports the devices it found on the node to the Kubelet device manager and starts its gRPC server to monitor the devices. * 3: A user submits a pod specification (workload manifest file) requesting a certain type of device. * 4: The scheduler determines a suitable node based on device availability and the local kubelet assigns a specific device to the pod's containers. -

Device Plugin Operation

-

Figure 3-2: Device Plugin Operation

+![**Figure 3-2:** Device Plugin Operation](../figures/Ch3_Figure_Device_Plugin_operation.png) -An example of often used device plugin is the [SR-IOV Network Device Plugin](https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin), that discovers and advertises SR-IOV Virtual Functions (VFs) available on a Kubernetes node, and is used to map VFs to scheduled pods. To use it, the SR-IOV CNI is required, as well as a CNI multiplexer plugin (such as [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) or [DANM](https://github.com/nokia/danm)), to provision additional secondary network interfaces for VFs (beyond the primary network interface). The SR-IOV CNI during pod creation allocates a SR-IOV VF to a pod's network namespace using the VF information given by the meta plugin, and on pod deletion releases the VF from the pod. +**Figure 3-2:** Device Plugin Operation +An example of often used device plugin is the [SR-IOV Network Device Plugin](https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin), that discovers and advertises SR-IOV Virtual Functions (VFs) available on a Kubernetes node, and is used to map VFs to scheduled pods. To use it, the SR-IOV CNI is required, as well as a CNI multiplexer plugin (such as [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) or [DANM](https://github.com/nokia/danm)), to provision additional secondary network interfaces for VFs (beyond the primary network interface). The SR-IOV CNI during pod creation allocates a SR-IOV VF to a pod's network namespace using the VF information given by the meta plugin, and on pod deletion releases the VF from the pod. -#### 3.2.1.7 Hardware Acceleration +#### Hardware Acceleration Hardware Acceleration Abstraction in RM [3.8 Hardware Acceleration Abstraction](../../../ref_model/chapters/chapter03.md#3.8) describes types of hardware acceleration (CPU instructions, Fixed function accelerators, Firmware-programmable adapters, SmartNICs and SmartSwitches), and usage for Infrastructure Level Acceleration and Application Level Acceleration. Scheduling pods that require or prefer to run on nodes with hardware accelerators will depend on type of accelerator used: -• CPU instructions can be found with Node Feature Discovery - -• Fixed function accelerators, Firmware-programmable network adapters and SmartNICs can be found and mapped to pods by using Device Plugin. +* CPU instructions can be found with Node Feature Discovery +* Fixed function accelerators, Firmware-programmable network adapters and SmartNICs can be found and mapped to pods by using Device Plugin. - -#### 3.2.1.8 Scheduling Pods with Non-resilient Applications +#### Scheduling Pods with Non-resilient Applications Non-resilient applications are sensitive to platform impairments on Compute like pausing CPU cycles (for example because of OS scheduler) or Networking like packet drops, reordering or latencies. Such applications need to be carefully scheduled on nodes and preferably still decoupled from infrastructure details of those nodes. @@ -267,9 +229,9 @@ Non-resilient applications are sensitive to platform impairments on Compute like | 4 | Networking (dataplane) | | No, or Fixed function acceleration, Firmware-programmable network adapters or SmartNICs | Huge pages (for DPDK-based applications); CPU Manager with configuration for isolcpus and SMT; Multiple interfaces; NUMA topology; Device Plugin | | 5 | Networking (dataplane) | | CPU instructions | Huge pages (for DPDK-based applications); CPU Manager with configuration for isolcpus and SMT; Multiple interfaces; NUMA topology; Device Plugin; NFD | -

Table 3-1: Categories of applications, requirements for scheduling pods and Kubernetes features

+**Table 3-1:** Categories of applications, requirements for scheduling pods and Kubernetes features -#### 3.2.1.9 Virtual Machine based Clusters +#### Virtual Machine based Clusters Kubernetes clusters using above enhancements can implement worker nodes with "bare metal" servers (running Container Runtime in Linux host Operating System) or with virtual machines (VMs, on hypervisor). @@ -279,8 +241,7 @@ When running in VMs, the following list of configurations shows what is needed f * Hardware Topology Management with NUMA enabled in hypervisor, mapped into VM, if needed enabled in guest OS, and mapped into pod. * If Node Feature Discovery and Device Plugin Framework are required, the required CPU instructions must be enabled in the VM virtual hardware, and the required devices must be virtualised in the hypervisor or passed through to the Node VM, and mapped into the pods. - -### 3.2.2 Container Networking Services +### Container Networking Services Kubernetes considers networking as a key component, with a number of distinct solutions. By default, Kubernetes networking is considered an "extension" to the @@ -311,6 +272,7 @@ these options. It is important to note that different networking solutions requi different descriptors from the Kubernetes workloads (specifically, the deployment artefacts such as YAML files, etc.), therefore the networking solution should be agreed between the CNF vendors and the CNF operators: + - The **Default CNI Plugin** through the use of deployment specific configuration (e.g. [Tungsten Fabric](https://tungstenfabric.github.io/website/Tungsten-Fabric-Architecture.html#vrouter-deployment-options)) - A **multiplexer/meta-plugin** that integrates with the Kubernetes control plane @@ -339,8 +301,7 @@ Mesh](https://networkservicemesh.io/docs/concepts/what-is-nsm/)) | Cluster wide coordination of IP address assignment (`req.inf.ntw.10`) | Supported via IPAM CNI plugin | Supported | Supported | Supported via IPAM CNI plugin | -

Table 3-1: Comparison of example networking solutions

- +**Table 3-1:** Comparison of example networking solutions For hardware resources that are needed by Kubernetes applications, [Device Plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) @@ -351,15 +312,16 @@ resources that may require vendor specific initialisation and setup" to be managed and consumed via standard interfaces. Figure 3-3 below shows the main building blocks of a Kubernetes networking solution: -- **Kubernetes Control Plane**: this is the core of a Kubernetes Cluster - the + +* **Kubernetes Control Plane**: this is the core of a Kubernetes Cluster - the apiserver, etcd cluster, kube-scheduler and the various controller-managers. The control plane (in particular the apiserver) provide a centralised point by which the networking solution is managed using a centralised management API. -- **Default CNI Plugin (Cluster Network)**: this is the default Cluster network plugin +* **Default CNI Plugin (Cluster Network)**: this is the default Cluster network plugin that has been deployed within the Cluster to provide IP addresses to Pods. Note that support for IPv6 requires not only changes in the Kubernetes control plane, but also requires the use of a CNI Plugin that support dual-stack networking. -- **CNI multiplexer/meta-plugin**: as described above, this is an optional component +* **CNI multiplexer/meta-plugin**: as described above, this is an optional component that integrates with the Kubernetes control plane via CNI, but allows for the use of multiple CNI plugins and the provision of multiple network connections to each Pod, as shown by the use of additional CNI Plugin and `net0` connection in @@ -368,57 +330,61 @@ require different networking technologies, which would potentially require different CNI plugins. Also note that this is only required for the Network Intensive profile. Example CNI implementations which meet these requirements include Multus and DANM. -- **CNI Plugin (Additional)**: this is a CNI plugin that is used to provide +* **CNI Plugin (Additional)**: this is a CNI plugin that is used to provide additional networking needs to Pods, that aren't provided by the default CNI plugin. This can include connectivity to underlay networks via accelerated hardware devices. -- **Device Plugin**: this is a Kubernetes extension that allows for the management +* **Device Plugin**: this is a Kubernetes extension that allows for the management and advertisement of vendor hardware devices. In particular, devices such as FPGA, SR-IOV NICs, SmartNICs, etc. can be made available to Pods by using Device Plugins. Note that alignment of these devices, CPU topology and huge pages will need the use of the [Topology Manager](https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/). -- **External / Application Load Balancing**: As Kubernetes Ingress, Egress and +* **External / Application Load Balancing**: As Kubernetes Ingress, Egress and Services have no support for all the protocols needed in telecommunication environments (Diameter, SIP, LDAP, etc) and their capacity is limited, the architecture includes the use of alternative load balancers, including external or ones built into the application. Management of external load balancers must be possible via Kubernetes API objects. -- **Other Features**: these additional features that are required by the +* **Other Features**: these additional features that are required by the networking solution as a whole, may be delivered by the **"Default CNI Plugin"**, or the **"CNI multiplexer/meta-plugin"** if it is deployed. For example: - - The integration of SDN solutions required by `req.inf.ntw.05` is enabled + + * The integration of SDN solutions required by `req.inf.ntw.05` is enabled via CNI integration. - - IP Address Management (**IPAM**) of the various networks can be provided + * IP Address Management (**IPAM**) of the various networks can be provided by one or more IPAM plugins, which can be part of a CNI plugin, or some other component (i.e. external SDN solution) - it is key that there are no overlapping IP addresses within a Cluster, and if multiple IPAM solutions are used that they are co-ordinated in some way (as required by `req.inf.ntw.10`). -- **Service Mesh**: The well known service meshes are "application service meshes" + +* **Service Mesh**: The well known service meshes are "application service meshes" that address and interact with the application layer 7 protocols (eg.: HTTP) only. Therefore, their support is not required in this architecture, as these service meshes are outside the scope of the infrastructure layer of this architecture. -

Kubernetes Networking Architecture

-

Figure 3-3: Kubernetes Networking Architecture

+![**Figure 3-3:** Kubernetes Networking Architecture](../figures/ch03_networking.png) +**Figure 3-3:** Kubernetes Networking Architecture + There are a number of different methods involved in managing, configuring and consuming networking resources in Kubernetes, including: -- The Default Cluster Network can be installed and managed by config files, + +* The Default Cluster Network can be installed and managed by config files, Kubernetes API Server (e.g. Custom Resource Definitions) or a combination of the two. -- Additional networking management plane (e.g. CNI multiplexer/meta-plugin or +* Additional networking management plane (e.g. CNI multiplexer/meta-plugin or federated networking manager) can be installed and managed by config files, Kubernetes API Server (e.g. Custom Resource Definitions) or a combination of the two. -- The connecting of Pods to the Default Cluster Network is handled by the Default +* The connecting of Pods to the Default Cluster Network is handled by the Default CNI Plugin (Cluster Network). -- The connecting of Pods to the additional networks is handled by the additional +* The connecting of Pods to the additional networks is handled by the additional networking management plane through the Kubernetes API (e.g. Custom Resource Definitions, Device Plugin API). -- Configuration of these additional network connections to Pods (i.e. provision of +* Configuration of these additional network connections to Pods (i.e. provision of an IP address to a Pod) can either be managed through the Kubernetes API (e.g. Custom Resource Definitions) or an external management plane (e.g. dynamic address assignment from a VPN server). @@ -442,15 +408,12 @@ an additional feature and still be conformant with Anuket. [here](../../../ref_model/chapters/chapter05.md#5.1) and hardware profile features [here](../../../ref_model/chapters/chapter05.md#5.4). - -### 3.2.2.1 Kubernetes Networking Semantics +### Kubernetes Networking Semantics The support for advanced network configuration management doesn't exist in core Kubernetes. Kubernetes is missing the advanced networking configuration component of Infrastructure as a Service (IaaS). For example, there is no network configuration API, there is no way to create L2 networks, instantiate network services such as L3aaS and LBaaS and then connect them all together. Kubernetes networking can be divided into two parts, built in network functionality available through the pod's mandatory primary interface and network functionality available through the pod's optional secondary interfaces. - - -#### 3.2.2.1.1 Built in Kubernetes Network Functionality +#### Built in Kubernetes Network Functionality Kubernetes currently only allows for one network, the *cluster* network, and one network attachment for each pod. All pods and containers have an *eth0* interface, this interface is created by Kubernetes at pod creation and attached to the cluster network. All communication to and from the pod is done through this interface. To only allow for one interface in a pod removes the need for traditional networking tools such as *VRFs* and additional routes and routing tables inside the pod network namespace. The basic semantics of Kubernetes, and the information found in manifests, defines the connectivity rules and behavior without any references to IP addresses. This has many advantages, it makes it easy to create portable, scalable SW services and network policies for them that are not location aware and therefore can be executed more or less anywhere. @@ -505,8 +468,7 @@ This is all that is needed to deploy 4 pods/containers that are fronted by a ser None of this is of much help however when implementing network service functions such as VNFs/CNFs that operate on multiple networks and require advanced networking configurations. - -#### 3.2.2.1.2 Multiple Networks and Advanced Configurations +#### Multiple Networks and Advanced Configurations Kubernetes does currently not in itself support multiple networks, pod multiple network attachments or advanced network configurations. This is supported by using a [*Container Network Interface*](https://github.com/containernetworking/cni) multiplexer such as [Multus](https://github.com/k8snetworkplumbingwg/multus-cni). A considerable effort is being invested to add better network support to Kubernetes, all such activities are coordinated through the kubernetes [*Network Special Interest Group*](https://github.com/kubernetes/community/tree/master/sig-network) and it's sub groups. One such group, the [*Network Plumbing Working Group*](https://github.com/k8snetworkplumbingwg/community) has produced the [Kubernetes Network Custom Resource Definition De-facto Standard](https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit). This document describes how secondary networks can be defined and attached to pods. @@ -574,7 +536,7 @@ metadata: This is enough to support basic network configuration management, it is possible to map up L2 networks from an external network infrastructure into a Kubernetes system and attach pods to these networks. The support for IPv4 and IPv6 address management is however limited. The address must be assigned by the CNI plugin as part of the pod creation process. -### 3.2.3 Container Storage Services +### Container Storage Services Since its 1.13 version Kubernetes supports Container Storage Interface (CSI) in production and in-tree volume plugins are moved out from the Kubernetes @@ -623,13 +585,13 @@ Kubernetes-based workloads. There are no restrictions or constraints that Kubernetes places on the storage that can be consumed by a workload, in terms of the requirements that are defined in RM sections -[5.2.2](../../../ref_model/chapters/chapter05.md#522-virtual-storage) (software) -and [5.4.2](../../../ref_model/chapters/chapter05.md#542-storage-configurations) +[5.2.2](../../../ref_model/chapters/chapter05.md#virtual-storage) (software) +and [5.4.2](../../../ref_model/chapters/chapter05.md#storage-configurations) (hardware). The only point of difference is that Kubernetes does not have a native object storage offering, and addressing this capability gap directly is outside of the scope of this RA. -### 3.2.4 Kubernetes Application package manager +### Kubernetes Application package manager To manage complex applications consisting of several Pods the Reference Architecture must provide support for a Kubernetes Application package manager. @@ -641,26 +603,27 @@ lifecycle management of the applications they manage. The Reference Architecture does not recommend the usage of a Kubernetes Application package manager with a server side component installed to the Kubernetes Cluster (e.g.: Tiller). -### 3.2.5 Custom Resources +### Custom Resources [Custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) are extensions of the Kubernetes API that represent customizations of Kubernetes installation. Core Kubernetes functions are also built using custom resources which makes Kubernetes more modular. Two ways to add custom resources are: * [Custom Resource Definitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) (CRDs): Defining CRD object creates new custom resource with a name and schema that are easy to use. * [API Server Aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/): Additional API that in flexible way extends Kubernetes beyond core Kubernetes API. -#### 3.2.5.1 Operator Pattern +#### Operator Pattern A [custom controller](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#custom-controllers) is a control loop that watches a custom resource for changes and tries to keep the current state of the resource in sync with the desired state. [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) combines custom resources and custom controllers. Operators are software extensions to Kubernetes that capture operational knowledge and automate usage of custom resources to manage applications, their components and cloud infrastructure. Operators can have different capability levels. As per repository [OperatorHub.io](https://operatorhub.io/), an operator can have different capability levels ([picture](https://operatorhub.io/static/images/capability-level-diagram.svg)): + * Basic install: Automated application provisioning and configuration management. * Seamless upgrades: Patch and minor version upgrades supported. * Full lifecycle: Application lifecycle, storage lifecycle (backup, failure recovery). * Deep insights: Metrics, alerts, log processing and workload analysis. * Auto pilot: Horizontal/vertical scaling, automated configuration tuning, abnormality detection, scheduling tuning. -## 3.3 CaaS Manager - Cluster Lifecycle Management +## CaaS Manager - Cluster Lifecycle Management > Note: *detailed requirements and component specification of cluster LCM are out of scope for this release.* @@ -669,12 +632,14 @@ Architecture provides support for a **CaaS Manager**, a component responsible fo This component is responsible for delivering an end-to-end life cycle management (creation and installation, scaling, updating, deleting, etc., of entire clusters), visibility and control of CaaS clusters, along with verification of security and compliance of Kubernetes clusters across multiple data centres and clouds. Specifically, the scope of the CaaS Manager includes: -- Infrastructure (Kubernetes Clusters) provisioning - - LCM of control/worker VM nodes - via IaaS API - - or Baremetal provisioning for physical nodes -- Control plane installation (i.e. Kubernetes control plane components on the nodes) -- Node Host OS customisation (e.g. Kernel customisation) -- Management of Cluster add-ons (eg CNIs, CSIs, Service Meshes) +* Infrastructure (Kubernetes Clusters) provisioning + + * LCM of control/worker VM nodes - via IaaS API + * or Baremetal provisioning for physical nodes + +* Control plane installation (i.e. Kubernetes control plane components on the nodes) +* Node Host OS customisation (e.g. Kernel customisation) +* Management of Cluster add-ons (eg CNIs, CSIs, Service Meshes) The CaaS Manager maintains a catalogue of **clusters templates**, used to create clusters specific to the requirements of workloads, the underlying virtualisation provider and/or the specific server hardware to be used for the cluster. diff --git a/doc/ref_arch/kubernetes/chapters/chapter03.rst b/doc/ref_arch/kubernetes/chapters/chapter03.rst new file mode 100644 index 0000000000..6e68ccdbbb --- /dev/null +++ b/doc/ref_arch/kubernetes/chapters/chapter03.rst @@ -0,0 +1,729 @@ +High Level Architecture +======================= + +Introduction +------------ + +The Anuket Kubernetes Reference Architecture (RA) is intended to be an industry +standard independent Kubernetes reference architecture that is not tied to any +specific offering or distribution. No vendor-specific enhancements are required +in order to achieve conformance to the principles of Anuket specifications; conformance is achieved by +using upstream components or features that are developed by the open source +community. This allows operators to have a common Kubernetes-based architecture +that supports any conformant VNF or CNF deployed on it to operate as expected. +The purpose of this chapter is to outline all the components required to provide +Kubernetes in a consistent and reliable way. The specification of how to use +these components is detailed in `Chapter 04 `__. + +Kubernetes is already a well documented and widely deployed Open Source project +managed by the Cloud Native Computing Foundation (CNCF). Full documentation of +the Kubernetes code and project can be found at +`https://kubernetes.io/docs/home/ `__. The +following chapters will only describe the specific features required by the Anuket +Reference Architecture, and how they would be expected to be implemented. For +any information related to standard Kubernetes features and capabilities, refer +back to the standard Kubernetes documentation. + +While this reference architecture provides options for pluggable components such +as service mesh and other plugins that might be used, the focus of the +reference architecture is on the abstracted interfaces and features that are +required for telco type workload management and execution. + +Chapter 5 of the Reference Model (RM) describes the +`hardware <../../../ref_model/chapters/chapter05.md#5.3>`__ and +`software <../../../ref_model/chapters/chapter05.md#5.1>`__ profiles that are +descriptions of the capabilities and features that the Cloud Infrastructure +provide to the workloads. As of v2.0, Figure 5-3 in the RM (also shown below) +depicts a high level view of the software profile features that apply to each +instance profile (Basic and Network Intensive). For more information on the +instance profiles please refer to `RM Chapter 4, section +4.2.4 <../../../ref_model/chapters/chapter04.md#4.2.4>`__. + +.. image:: ../../../ref_model/figures/RM_chap5_fig_5_3_SW_profile.png + :alt: "Figure 5-3 (from RM): NFVI softwareprofiles" + + +**Figure 5-3 (from RM):** NFVI softwareprofiles + +In addition, the RM Figure 5-4 (shown below) depicts the hardware profile features +that apply to each instance profile. + +.. image:: ../../../ref_model/figures/RM_chap5_fig_5_4_HW_profile.png + :alt: "Figure 5-4 (from RM): NFVI hardwareprofiles and host associated capabilities" + + +**Figure 5-4 (from RM):** NFVI hardwareprofiles and host associated capabilities + +The features and capabilities described in the software and hardware profiles +are considered throughout this RA, with the RA requirements traceability to the +RM requirements formally documented in `chapter 2, section +2.2 <./chapter02.md#2.2>`__ of this RA. + +Infrastructure Services +----------------------- + +Container Compute Services +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The primary interface between the Physical / Virtual Infrastructure and any +container-relevant components is the Kubernetes Node Operating System. This is +the OS within which the container runtime exists, and within which the +containers run (and therefore, the OS whose kernel is shared by the referenced +containers). This is shown in Figure 3-1 below. + +.. image:: ../figures/ch03_hostOS.png + :alt: "Figure 3-1: Kubernetes Node Operating System" + + +**Figure 3-1:** Kubernetes Node Operating System + +The Kubernetes Node OS (as with any OS) consists of two main components: + +- Kernel space +- User space + +The Kernel is the tightly controlled space that provides an API to applications +running in the user space (which usually have their own southbound interface in +an interpreter or libraries). Key containerisation capabilities such as Control +Groups (cgroups) and namespaces are kernel features, and are used and managed by +the container runtime in order to provide isolation between the user space +processes, which would also include the container itself as well as any +processes running within it. The security of the Kubernetes Node OS and its +relationship to the container and the applications running within the container +or containers is essential to the overall security posture of the entire system, +and must be appropriately secured to ensure processes running in one container +cannot escalate their privileges or otherwise affect processes running in an +adjacent container. An example and more details of this concept can be found in +`chapter 6 <./chapter06.md>`__. + +It is important to note that the container runtime itself is also a set of +processes that run in user space, and therefore also interact with the kernel +via system calls. Many diagrams will show containers as running on top of the +runtime, or inside the runtime. More accurately, the containers themselves are +simply processes running within an OS, the container runtime is simply another +set of processes that are used to manage these containers (pull, run, delete, +etc.), and the kernel features required to provide the isolation mechanisms +(cgroups, namespaces, filesystems, etc.) between the components. + +Container Runtime Services +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Container Runtime is the component that runs within a Kubernetes Node +Operating System (OS) and manages the underlying OS functionality, such as +cgroups and namespaces (in Linux), in order to provide a service within which +container images can be executed and make use of the infrastructure resources +(compute, storage, networking and other I/O devices) abstracted by the Container +Host OS, based on API instructions from the kubelet. + +There are a number of different container runtimes. The simplest form, low-level +container runtimes, just manage the OS capabilities such as cgroups and +namespaces, and then run commands from within those cgroups and namespaces. An +example of this type of runtime is runc, which underpins many of the +higher-level runtimes and is considered a reference implementation of the `Open +Container Initiative (OCI) runtime +spec `__. This specification +includes details on how an implementation (i.e. an actual container runtime such +as runc) must, for example, configure resource shares and limits (e.g. CPU, +Memory, IOPS) for the containers that Kubernetes (via the kubelet) schedules on +that host. This is important to ensure that the features and capabilities +described in `chapter 5 of the RM <../../../ref_model/chapters/chapter05.md>`__ are +supported by this RA and delivered by any downstream Reference Implementations +(RIs) to the instance types defined in the RM. + +Where low-level runtimes are used for the execution of a container within an OS, +the more complex/complete high-level container runtimes are used for the general +management of container images - moving them to where they need to be executed, +unpacking them, and then passing them to the low-level runtime, which then +executes the container. These high-level runtimes also include a comprehensive +API that other applications (e.g. Kubernetes) can use to interact and manage the +containers. An example of this type of runtime is containerd, which provides the +features described above, before passing off the unpacked container image to +runc for execution. + +For Kubernetes the important interface to consider for container management is +the `Kubernetes Container Runtime Interface +(CRI) `__. +This is an interface specification for any container runtime so that it is able +to integrate with the kubelet on a Kubernetes Node. The CRI decouples the +kubelet from the runtime that is running in the Host OS, meaning that the code +required to integrate kubelet with a container runtime is not part of the +kubelet itself (i.e. if a new container runtime is needed and it uses CRI, it +will work with kubelet). Examples of this type of runtime include containerd +(with CRI plugin) and cri-o, which is built specifically to work with +Kubernetes. + +To fulfil ``req.inf.vir.01`` the architecture should support a container runtime +which provides the isolation of Operating System kernels. + +The architecture must support a way to isolate the compute resources of the +infrastructure itself from the workloads compute resources. + +The basic semantics of Kubernetes, and the information found in manifests, defines the built-in Kubernetes objects and their desired state. + +Kubernetes built in objects + ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ +|Pod and workloads | Description | ++==========================================================================================+===============================================================================================================================================+ +|`Pod: `__ | Pod is a collection of containers that can run on a node. This resource is created by clients and scheduled onto nodes. | ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ +|`ReplicaSet: `__ | ReplicaSet ensures that a specified number of pod replicas are running at any given time. | ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ +|`Deployment: `__ | Deployment enables declarative updates for Pods and ReplicaSets. | ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ +|`DaemonSet: `__ | A Daemon set ensures that the correct nodes run a copy of a Pod. | ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ +|`Job: `__ | A Job represent a task, it creates one or more Pods and will continue to retry until the expected number of successful completions is reached.| ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ +|`CronJob: `__ | A CronJob manages time-based Jobs, namely: once at a specified point in time and repeatedly at a specified point in time | ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ +|`StatefulSet: `__ | StatefulSet represents a set of pods with consistent identities. Identities are defined as: network, storage. | ++------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------+ + +CPU Management +^^^^^^^^^^^^^^ + +CPU management has policies to determine placement preferences to use for workloads that are sensitive to cache affinity or latency, and so the workloads must not be moved by OS scheduler or throttled by kubelet. Additionally, some workloads are sensitive to differences between physical cores and SMT, while others (like DPDK-based workloads) are designed to run on isolated CPUs (like on Linux with cpuset-based selection of CPUs and isolcpus kernel parameter specifying cores isolated from general SMP balancing and scheduler algorithms). + +Kubernetes `CPU Manager `__ works with Topology Manager. Special care needs to be taken of: + +- Supporting isolated CPUs: Using kubelet `Reserved CPUs `__ and Linux isolcpus allows configuration where only isolcpus are allocatable to pods. Scheduling pods to such nodes can be influenced with taints, tolerations and node affinity. +- Differentiating between physical cores and SMT: When requesting even number of CPU cores for pods, scheduling can be influenced with taints, tolerations, and node affinity. + +Memory and Huge Pages Resources Management +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Reference Model requires the support of huge pages in i.cap.018 which is supported by upstream Kubernetes (`documentation `__). + +For proper mapping of huge pages to scheduled pods, both need to have huge pages enabled in the operating system (configured in kernel and mounted with correct permissions) and kubelet configuration. Multiple sizes of huge pages can be enabled like 2 MiB and 1 GiB. + +For some applications, huge pages +should be allocated to account for consideration of the underlying HW topology. +`The Memory Manager `__ (added to Kubernetes v1.21 as alpha feature) enables the feature of guaranteed memory and huge pages allocation for pods in the Guaranteed QoS class. The Memory Manager feeds the Topology Manager with hints for most suitable NUMA affinity. + +Hardware Topology Management +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Scheduling pods across NUMA boundaries can result in lower performance and higher latencies. This would be an issue for applications that require optimisations of CPU isolation, memory and device locality. + +Kubernetes supports Topology policy per node as beta feature (`documentation `__) and not per pod. The Topology Manager receives Topology information from Hint Providers which identify NUMA nodes (defined as server system architecture divisions of CPU sockets) and preferred scheduling. In the case of the pod with Guaranteed QoS class having integer CPU requests, the static CPU Manager policy would return topology hints relating to the exclusive CPU and the Device Manager would provide hints for the requested device. + +If case that memory or huge pages are not considered by the Topology Manager, it can be done by the operating system providing best-effort local page allocation for containers as long as there is sufficient free local memory on the node, or with Control Groups (cgroups) cpuset subsystem that can isolate memory to single NUMA node. + +Node Feature Discovery +^^^^^^^^^^^^^^^^^^^^^^ + +`Node Feature Discovery `__ (NFD) can run on every node as a daemon or as a job. NFD detects detailed hardware and software capabilities of each node and then advertises those capabilities as node labels. Those node labels can be used in scheduling pods by using Node Selector or Node Affinity for pods that require such capabilities. + +Device Plugin Framework +^^^^^^^^^^^^^^^^^^^^^^^ + +`Device Plugin Framework `__ advertises device hardware resources to kubelet with which vendors can implement plugins for devices that may require vendor-specific activation and life cycle management, and securely maps these devices to containers. + +Figure 3-2 shows in four steps how device plugins operate on a Kubernetes node: + +- 1: During setup, the cluster administrator (more in `3.2.5.1 Operator Pattern `__) knows or discovers (as per `3.2.1.5 Node Feature Discovery `__) what kind of devices are present on the different nodes, selects which devices to enable and deploys the associated device plugins. +- 2: The plugin reports the devices it found on the node to the Kubelet device manager and starts its gRPC server to monitor the devices. +- 3: A user submits a pod specification (workload manifest file) requesting a certain type of device. +- 4: The scheduler determines a suitable node based on device availability and the local kubelet assigns a specific device to the pod's containers. + +.. image:: ../figures/Ch3_Figure_Device_Plugin_operation.png + :alt: "Figure 3-2: Device Plugin Operation" + + +**Figure 3-2:** Device Plugin Operation + +An example of often used device plugin is the `SR-IOV Network Device Plugin `__, that discovers and advertises SR-IOV Virtual Functions (VFs) available on a Kubernetes node, and is used to map VFs to scheduled pods. To use it, the SR-IOV CNI is required, as well as a CNI multiplexer plugin (such as `Multus CNI `__ or `DANM `__), to provision additional secondary network interfaces for VFs (beyond the primary network interface). The SR-IOV CNI during pod creation allocates a SR-IOV VF to a pod's network namespace using the VF information given by the meta plugin, and on pod deletion releases the VF from the pod. + +Hardware Acceleration +^^^^^^^^^^^^^^^^^^^^^ + +Hardware Acceleration Abstraction in RM `3.8 Hardware Acceleration Abstraction <../../../ref_model/chapters/chapter03.md#3.8>`__ describes types of hardware acceleration (CPU instructions, Fixed function accelerators, Firmware-programmable adapters, SmartNICs and SmartSwitches), and usage for Infrastructure Level Acceleration and Application Level Acceleration. + +Scheduling pods that require or prefer to run on nodes with hardware accelerators will depend on type of accelerator used: + +- CPU instructions can be found with Node Feature Discovery +- Fixed function accelerators, Firmware-programmable network adapters and SmartNICs can be found and mapped to pods by using Device Plugin. + +Scheduling Pods with Non-resilient Applications +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Non-resilient applications are sensitive to platform impairments on Compute like pausing CPU cycles (for example because of OS scheduler) or Networking like packet drops, reordering or latencies. Such applications need to be carefully scheduled on nodes and preferably still decoupled from infrastructure details of those nodes. + += ====================== ====================== ======================================================================================= ===================================================================================================================================================== +# Intensive on Not intensive on Using hardware acceleration Requirements for optimised pod scheduling += ====================== ====================== ======================================================================================= ===================================================================================================================================================== +1 Compute Networking (dataplane) No CPU Manager +2 Compute Networking (dataplane) CPU instructions CPU Manager, NFD +3 Compute Networking (dataplane) Fixed function acceleration, Firmware-programmable network adapters or SmartNICs CPU Manager, Device Plugin +4 Networking (dataplane) No, or Fixed function acceleration, Firmware-programmable network adapters or SmartNICs Huge pages (for DPDK-based applications); CPU Manager with configuration for isolcpus and SMT; Multiple interfaces; NUMA topology; Device Plugin +5 Networking (dataplane) CPU instructions Huge pages (for DPDK-based applications); CPU Manager with configuration for isolcpus and SMT; Multiple interfaces; NUMA topology; Device Plugin; NFD += ====================== ====================== ======================================================================================= ===================================================================================================================================================== + +**Table 3-1:** Categories of applications, requirements for scheduling pods and Kubernetes features + +Virtual Machine based Clusters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Kubernetes clusters using above enhancements can implement worker nodes with "bare metal" servers (running Container Runtime in Linux host Operating System) or with virtual machines (VMs, on hypervisor). + +When running in VMs, the following list of configurations shows what is needed for non-resilient applications: + +- CPU Manager managing vCPUs that hypervisor provides to VMs. +- Huge pages enabled in hypervisor, mapped to VM, enabled in guest OS, and mapped to pod. +- Hardware Topology Management with NUMA enabled in hypervisor, mapped into VM, if needed enabled in guest OS, and mapped into pod. +- If Node Feature Discovery and Device Plugin Framework are required, the required CPU instructions must be enabled in the VM virtual hardware, and the required devices must be virtualised in the hypervisor or passed through to the Node VM, and mapped into the pods. + +Container Networking Services +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Kubernetes considers networking as a key component, with a number of distinct +solutions. By default, Kubernetes networking is considered an "extension" to the +core functionality, and is managed through the use of `Network +Plugins `__, +which can be categorised based on the topology of the networks they manage, and +the integration with the switching (e.g. vlan vs tunnels) and routing (e.g. +virtual vs physical gateways) infrastructure outside of the Cluster: + +- **Layer 2 underlay** plugins provide east/west ethernet connectivity between + pods and north/south connectivity between pods and external networks by using + the network underlay (eg VLANs on DC switches). When using the underlay for + layer 2 segments, configuration is required on the DC network for every network. +- **Layer 2 overlay** plugins provide east/west pod-to-pod connectivity by creating + overlay tunnels (eg VXLAN/GENEVE tunnels) between the nodes, without requiring + creation of per-application layer 2 segments on the underlay. North-south + connectivity cannot be provided. +- **Layer 3** plugins create a virtual router (eg BPF, iptables, kubeproxy) in + each node, and can route traffic between multiple layer 2 overlays via them. + North-south traffic is managed by peering (eg with BGP) virtual routers on the + nodes with the DC network underlay, allowing each pod or service IP to be + announced independently. + +However, for more complex requirements such as providing connectivity through +acceleration hardware, there are three approaches that can be taken, with Table 3-1 +showing some of the differences between networking solutions that consist of +these options. It is important to note that different networking solutions require +different descriptors from the Kubernetes workloads (specifically, the deployment +artefacts such as YAML files, etc.), therefore the networking solution should be +agreed between the CNF vendors and the CNF operators: + +- The **Default CNI Plugin** through the use of deployment specific configuration + (e.g. `Tungsten Fabric `__) +- A **multiplexer/meta-plugin** that integrates with the Kubernetes control plane + via CNI (Container Network Interface) and allows for use of multiple CNI plugins + in order to provide this specific connectivity that the default Network Plugin may + not be able to provide (e.g. `Multus `__, + `DANM `__) +- An external, **federated networking manager** that uses the Kubernetes API Server + to create and manage additional connections for Pods (e.g. `Network Service + Mesh `__) + +============================================================================================================ ================================================================= ================================================================= ================================================================= ================================================================= +Requirement Networking Solution with Multus Networking Solution with DANM Networking Solution with Tungsten Fabric Networking Solution with NSM +============================================================================================================ ================================================================= ================================================================= ================================================================= ================================================================= +Additional network connections provider Multiplexer/meta-plugin Multiplexer/meta-plugin Federated networking manager Default CNI Plugin +The overlay network encapsulation protocol needs to enable ECMP in the underlay (``infra.net.cfg.002``) Supported via the additional CNI plugin Supported via the additional CNI plugin Supported TBC +NAT (``infra.net.cfg.003``) Supported via the additional CNI plugin Supported Supported TBC +Network Policies (Security Groups) (``infra.net.cfg.004``) Supported via a CNI Network Plugin that supports Network Policies Supported via a CNI Network Plugin that supports Network Policies Supported via a CNI Network Plugin that supports Network Policies Supported via a CNI Network Plugin that supports Network Policies +Traffic patterns symmetry (``infra.net.cfg.006``) Depends on CNI plugin used Depends on CNI plugin used Depends on CNI plugin used Depends on CNI plugin used +Centrally administrated and configured (``req.inf.ntw.03``) Supported via Kubernetes API Server Supported via Kubernetes API Server Supported via Kubernetes API Server Supported via Kubernetes API Server +Dual stack IPv4 and IPv6 for Kubernetes workloads (``req.inf.ntw.04``) Supported via the additional CNI plugin Supported Supported Supported +Integrating SDN controllers (``req.inf.ntw.05``) Supported via the additional CNI plugin Supported via the additional CNI plugin TF is an SDN controller TBC +More than one networking solution (``req.inf.ntw.06``) Supported Supported Supported Supported +Choose whether or not to deploy more than one networking solution (``req.inf.ntw.07``) Supported Supported Supported Supported +Kubernetes network model (``req.inf.ntw.08``) Supported via the additional CNI plugin Supported via the additional CNI plugin Supported Supported via the default CNI plugin +Do not interfere with or cause interference to any interface or network it does not own (``req.inf.ntw.09``) Supported Supported Supported Supported +Cluster wide coordination of IP address assignment (``req.inf.ntw.10``) Supported via IPAM CNI plugin Supported Supported Supported via IPAM CNI plugin +============================================================================================================ ================================================================= ================================================================= ================================================================= ================================================================= + +**Table 3-1:** Comparison of example networking solutions + +For hardware resources that are needed by Kubernetes applications, `Device +Plugins `__ +can be used to manage those resources and advertise them to the kubelet for use +by the Kubernetes applications. This allows resources such as "GPUs, +high-performance NICs, FPGAs, InfiniBand adapters, and other similar computing +resources that may require vendor specific initialisation and setup" to be +managed and consumed via standard interfaces. + +Figure 3-3 below shows the main building blocks of a Kubernetes networking solution: + +- **Kubernetes Control Plane**: this is the core of a Kubernetes Cluster - the + apiserver, etcd cluster, kube-scheduler and the various controller-managers. The + control plane (in particular the apiserver) provide a centralised point by which + the networking solution is managed using a centralised management API. + +- **Default CNI Plugin (Cluster Network)**: this is the default Cluster network plugin + that has been deployed within the Cluster to provide IP addresses to Pods. Note that + support for IPv6 requires not only changes in the Kubernetes control plane, but + also requires the use of a CNI Plugin that support dual-stack networking. + +- **CNI multiplexer/meta-plugin**: as described above, this is an optional component + that integrates with the Kubernetes control plane via CNI, but allows for the + use of multiple CNI plugins and the provision of multiple network connections to + each Pod, as shown by the use of additional CNI Plugin and ``net0`` connection in + the Pod. Note that the different network characteristics of the interfaces might + require different networking technologies, which would potentially require + different CNI plugins. Also note that this is only required for the Network + Intensive profile. Example CNI implementations which meet these requirements + include Multus and DANM. + +- **CNI Plugin (Additional)**: this is a CNI plugin that is used to provide + additional networking needs to Pods, that aren't provided by the default CNI plugin. + This can include connectivity to underlay networks via accelerated hardware devices. + +- **Device Plugin**: this is a Kubernetes extension that allows for the management + and advertisement of vendor hardware devices. In particular, devices such as + FPGA, SR-IOV NICs, SmartNICs, etc. can be made available to Pods by using Device Plugins. + Note that alignment of these devices, CPU topology and huge pages will need the use + of the `Topology Manager `__. + +- **External / Application Load Balancing**: As Kubernetes Ingress, Egress and + Services have no support for all the protocols needed in telecommunication + environments (Diameter, SIP, LDAP, etc) and their capacity is limited, the + architecture includes the use of alternative load balancers, including external + or ones built into the application. Management of external load balancers must + be possible via Kubernetes API objects. + +- **Other Features**: these additional features that are required by the + networking solution as a whole, may be delivered by the **"Default CNI Plugin"**, + or the **"CNI multiplexer/meta-plugin"** if it is deployed. For example: + + - The integration of SDN solutions required by ``req.inf.ntw.05`` is enabled + via CNI integration. + - IP Address Management (**IPAM**) of the various networks can be provided + by one or more IPAM plugins, which can be part of a CNI plugin, or some other + component (i.e. external SDN solution) - it is key that there are no overlapping + IP addresses within a Cluster, and if multiple IPAM solutions are used that + they are co-ordinated in some way (as required by ``req.inf.ntw.10``). + +- **Service Mesh**: The well known service meshes are "application service meshes" + that address and interact with the application layer 7 protocols (eg.: HTTP) + only. Therefore, their support is not required in this architecture, as these + service meshes are outside the scope of the infrastructure layer of this + architecture. + +.. image:: ../figures/ch03_networking.png + :alt: "Figure 3-3: Kubernetes Networking Architecture" + + +.. raw:: html + + + +**Figure 3-3:** Kubernetes Networking Architecture + +There are a number of different methods involved in managing, configuring and +consuming networking resources in Kubernetes, including: + +- The Default Cluster Network can be installed and managed by config files, + Kubernetes API Server (e.g. Custom Resource Definitions) or a combination of the + two. +- Additional networking management plane (e.g. CNI multiplexer/meta-plugin or + federated networking manager) can be installed and managed by config files, + Kubernetes API Server (e.g. Custom Resource Definitions) or a combination of the + two. +- The connecting of Pods to the Default Cluster Network is handled by the Default + CNI Plugin (Cluster Network). +- The connecting of Pods to the additional networks is handled by the additional + networking management plane through the Kubernetes API (e.g. Custom Resource + Definitions, Device Plugin API). +- Configuration of these additional network connections to Pods (i.e. provision of + an IP address to a Pod) can either be managed through the Kubernetes API (e.g. + Custom Resource Definitions) or an external management plane (e.g. dynamic + address assignment from a VPN server). + +There are several types of low latency and high throughput networks required by +telco workloads: signalling traffic workloads and user plane traffic workloads. +Networks used for signalling traffic are more demanding than what a standard +overlay network can handle, but still do not need the use of user space +networking. Due to the nature of the signalling protocols used, these type of +networks require NAT-less communication documented in ``infra.net.cfg.003`` and will +need to be served by a CNI plugin with IPVLAN or MACVLAN support. On the other +hand, the low latency, high throughput networks used for handling the user plane +traffic require the capability to use a user space networking technology. + + Note: An infrastructure can provide the possibility to use SR-IOV with DPDK as + an additional feature and still be conformant with Anuket. + +.. + + Editors note: The possibility to SR-IOV for DPDK is under discussion. + + Refer to software profile features + `here <../../../ref_model/chapters/chapter05.md#5.1>`__ and hardware profile + features `here <../../../ref_model/chapters/chapter05.md#5.4>`__. + +Kubernetes Networking Semantics +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The support for advanced network configuration management doesn't exist in core Kubernetes. Kubernetes is missing the advanced networking configuration component of Infrastructure as a Service (IaaS). For example, there is no network configuration API, there is no way to create L2 networks, instantiate network services such as L3aaS and LBaaS and then connect them all together. + +Kubernetes networking can be divided into two parts, built in network functionality available through the pod's mandatory primary interface and network functionality available through the pod's optional secondary interfaces. + +Built in Kubernetes Network Functionality +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Kubernetes currently only allows for one network, the *cluster* network, and one network attachment for each pod. All pods and containers have an *eth0* interface, this interface is created by Kubernetes at pod creation and attached to the cluster network. All communication to and from the pod is done through this interface. To only allow for one interface in a pod removes the need for traditional networking tools such as *VRFs* and additional routes and routing tables inside the pod network namespace. + +The basic semantics of Kubernetes, and the information found in manifests, defines the connectivity rules and behavior without any references to IP addresses. This has many advantages, it makes it easy to create portable, scalable SW services and network policies for them that are not location aware and therefore can be executed more or less anywhere. + +============================================================================================== ================================================================================================================================================================================================================================================================ +Network objects Description +============================================================================================== ================================================================================================================================================================================================================================================================ +`Ingress: `__ Ingress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend. An Ingress can be configured to give services externally reachable URLs, load balance traffic, terminate SSL, offer name based virtual hosting etc. +`Service: `__ Service is a named abstraction of an application running on a set of pods consisting of a local port (for example 3306) that the proxy listens on, and the selector that determines which pods will answer requests sent through the proxy. +`EndpointSlices: `__ Endpoints and Endpointslices are a collection of objects that contain the ip address, v4 and v6, of the pods that represents a service. +`Network Policy: `__ Network Policy defines which network traffic is allowed to ingress and egress from a set of pods. +============================================================================================== ================================================================================================================================================================================================================================================================ + +There is no need to explicitly define internal load balancers, server pools, service monitors, firewalls and so on. The Kubernetes semantics and relation between the different objects defined in the object manifests contains all the information needed. + +Example: The manifests for service *my-service* and the *deployment* with the four load balanced pods of type *my-app* + +Service: + +:: + + apiVersion: v1 + kind: Service + metadata: + name: my-service + spec: + selector: + app: my-app + ports: + - protocol: TCP + port: 123 + +Deployment: + +:: + + apiVersion: apps/v1 + kind: Deployment + metadata: name: my-app-deployment + spec: + selector: + matchLabels: + app: my-app + replicas: 4 + template: + metadata: + labels: + app: my-app + spec: + containers: + - name: my-app + image: my-app-1.2.3 + ports: + - containerPort: 123 + +This is all that is needed to deploy 4 pods/containers that are fronted by a service that performes load balancing. The *Deployment* will ensure that there are always four pods of type *my-app* available. the *Deployment* is responsible for the full lifecycle management of the pods, this includes in service update/upgrade. + +None of this is of much help however when implementing network service functions such as VNFs/CNFs that operate on multiple networks and require advanced networking configurations. + +Multiple Networks and Advanced Configurations +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Kubernetes does currently not in itself support multiple networks, pod multiple network attachments or advanced network configurations. This is supported by using a `Container Network Interface `__ multiplexer such as `Multus `__. +A considerable effort is being invested to add better network support to Kubernetes, all such activities are coordinated through the kubernetes `Network Special Interest Group `__ and it's sub groups. One such group, the `Network Plumbing Working Group `__ has produced the `Kubernetes Network Custom Resource Definition De-facto Standard `__. This document describes how secondary networks can be defined and attached to pods. + +This defacto standard defines among other things + ++-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Definition | Description | ++=========================================+==========================================================================================================================================================================+ +| Kubernetes Cluster-Wide default network | A network to which all pods are attached following the current behavior and requirements of Kubernetes, this done by attaching the *eth0* interface to the pod namespace.| ++-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Network Attachment | A means of allowing a pod to directly communicate with a given logical or physical network. Typically (but not necessarily) each attachment takes the form of a kernel | +| | network interface placed into the pod’s network namespace. Each attachment may result in zero or more IP addresses being assigned to the pod. | ++-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| NetworkAttachmentDefinition object | This defines resource object that describes how to attach a pod to a logical or physical network, the annotation name is *"k8s.v1.cni.cncf.io/networks"* | ++-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +| Network Attachment Selection Annotation | Selects one or more networks that a pod should be attached to. | ++-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + +Example: Define three network attachments and attach the three networks to a pod. + +Green network + +:: + + apiVersion: "k8s.cni.cncf.io/v1" + kind: NetworkAttachmentDefinition + metadata: +   name:green-network + spec: +   config: '{ +     "cniVersion": "0.3.0", +     "type": "plugin-A", + "vlan": "1234" +   }' + ) + +Blue network + +:: + + apiVersion: "k8s.cni.cncf.io/v1" + kind: NetworkAttachmentDefinition + metadata: +   name:blue-network + spec: +   config: '{ +     "cniVersion": "0.3.0", +     "type": "plugin-A", + "vlan": "3456" +   }' + ) + +Red network + +:: + + apiVersion: "k8s.cni.cncf.io/v1" + kind: NetworkAttachmentDefinition + metadata: +   name:red-network + spec: +   config: '{ +     "cniVersion": "0.3.0", +     "type": "plugin-B", + "knid": "123456789" +   }' + ) + +Pod my-pod + +:: + + kind: Pod + metadata: +   name: my-pod +   namespace: my-namespace +   annotations: +     k8s.v1.cni.cncf.io/networks: blue-network, green-network, red-network + +This is enough to support basic network configuration management, it is possible to map up L2 networks from an external network infrastructure into a Kubernetes system and attach pods to these networks. The support for IPv4 and IPv6 address management is however limited. The address must be assigned by the CNI plugin as part of the pod creation process. + +Container Storage Services +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since its 1.13 version Kubernetes supports Container Storage Interface (CSI) in +production and in-tree volume plugins are moved out from the Kubernetes +repository (see a list of CSI drivers +`here `__). + +Running containers will require ephemeral storage on which to run themselves +(i.e. storage on which the unpacked container image is stored and executed +from). This ephemeral storage lives and dies with the container and is a +directory on the worker node on which the container is running. Note, this +means that the ephemeral storage is mounted locally in the worker node +filesystem. The filesystem can be physically external to the worker node +(e.g. iSCSI, NFS, FC) but the container will still reference it as part of the +local filesystem. + +Additional storage might also be attached to a container through the use of +Kubernetes Volumes - this can be storage from the worker node filesystem +(through hostPaths - not recommended), or it can be external storage that is +accessed through the use of a Volume Plugin. Volume Plugins allow the use of a +storage protocol (e.g. iSCSI, NFS) or management API (e.g. Cinder, EBS) for the +attaching and mounting of storage into a Pod. This additional storage, that is +attached to a container using a Kubernetes Volume, does not live and die with +the container but instead follows the lifecycle of the Pod that the container is +a part of. This means the Volume persists across container restarts, as long as +the Pod itself is still running. However it does not necessarily persist when a +Pod is destroyed, and therefore cannot be considered suitable for any scenario +requiring persistent data. The lifecycle of the actual data depends on the +Volume Plugin used, and sometimes the configuration of the Volume Plugin as +well. + +For those scenarios where data persistence is required, Persistent Volumes (PV) +are used in Kubernetes. PVs are resources in a Kubernetes Cluster that are +consumed by Persistent Volume Claims (PVCs) and have a lifecycle that is +independent of any Pod that uses the PV. A Pod will use a PVC as the volume in +the Pod spec; a PVC is a request for persistent storage (a PV) by a Pod. By +default, PVs and PVCs are manually created and deleted. + +Kubernetes also provides an object called Storage Class, which is created by +Cluster administrators and maps to storage attributes such as +quality-of-service, encryption, data resilience, etc. Storage Classes also +enable the dynamic provisioning of Persistent Volumes (as opposed to the default +manual creation). This can be beneficial for organisations where the +administration of storage is performed separately from the administration of +Kubernetes-based workloads. + +There are no restrictions or constraints that Kubernetes places on the storage +that can be consumed by a workload, in terms of the requirements that are +defined in RM sections +`5.2.2 <../../../ref_model/chapters/chapter05.md#virtual-storage>`__ (software) +and `5.4.2 <../../../ref_model/chapters/chapter05.md#storage-configurations>`__ +(hardware). The only point of difference is that Kubernetes does not have a +native object storage offering, and addressing this capability gap directly is +outside of the scope of this RA. + +Kubernetes Application package manager +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To manage complex applications consisting of several Pods the Reference +Architecture must provide support for a Kubernetes Application package manager. +The Package Manager would be able to manage the lifecycle of a set of Pods and +provide a framework to customise a set of parameters for their deployment. The +requirement for this Reference Architecture is to provide a Kubernetes API that +complies with the CNCF Conformance test for the package managers to use in the +lifecycle management of the applications they manage. The Reference Architecture +does not recommend the usage of a Kubernetes Application package manager with a +server side component installed to the Kubernetes Cluster (e.g.: Tiller). + +Custom Resources +~~~~~~~~~~~~~~~~ + +`Custom resources `__ are extensions of the Kubernetes API that represent customizations of Kubernetes installation. Core Kubernetes functions are also built using custom resources which makes Kubernetes more modular. +Two ways to add custom resources are: + +- `Custom Resource Definitions `__ (CRDs): Defining CRD object creates new custom resource with a name and schema that are easy to use. +- `API Server Aggregation `__: Additional API that in flexible way extends Kubernetes beyond core Kubernetes API. + +Operator Pattern +^^^^^^^^^^^^^^^^ + +A `custom controller `__ is a control loop that watches a custom resource for changes and tries to keep the current state of the resource in sync with the desired state. + +`Operator pattern `__ combines custom resources and custom controllers. Operators are software extensions to Kubernetes that capture operational knowledge and automate usage of custom resources to manage applications, their components and cloud infrastructure. +Operators can have different capability levels. As per repository `OperatorHub.io `__, an operator can have different capability levels (`picture `__): + +- Basic install: Automated application provisioning and configuration management. +- Seamless upgrades: Patch and minor version upgrades supported. +- Full lifecycle: Application lifecycle, storage lifecycle (backup, failure recovery). +- Deep insights: Metrics, alerts, log processing and workload analysis. +- Auto pilot: Horizontal/vertical scaling, automated configuration tuning, abnormality detection, scheduling tuning. + +CaaS Manager - Cluster Lifecycle Management +------------------------------------------- + + Note: *detailed requirements and component specification of cluster LCM are out of scope for this release.* + +In order to provision multiple Kubernetes Clusters, which is a common scenario where workloads and network functions require dedicated, single-tenant Clusters, the Reference +Architecture provides support for a **CaaS Manager**, a component responsible for the Lifecycle Management of multiple Kubernetes Clusters. +This component is responsible for delivering an end-to-end life cycle management (creation and installation, scaling, updating, deleting, etc., of entire clusters), visibility and control of CaaS clusters, along with verification of security and compliance of Kubernetes clusters across multiple data centres and clouds. +Specifically, the scope of the CaaS Manager includes: + +- Infrastructure (Kubernetes Clusters) provisioning + + - LCM of control/worker VM nodes - via IaaS API + - or Baremetal provisioning for physical nodes + +- Control plane installation (i.e. Kubernetes control plane components on the nodes) + +- Node Host OS customisation (e.g. Kernel customisation) + +- Management of Cluster add-ons (eg CNIs, CSIs, Service Meshes) + +The CaaS Manager maintains a catalogue of **clusters templates**, used to create clusters specific to the requirements of workloads, the underlying virtualisation provider and/or the specific server hardware to be used for the cluster. + +The CaaS manager works by integrating with an underlying virtualisation provider for VM-based clusters, or with Bare Metal management APIs for physical clusters, to create Cluster nodes and provide other capabilities such as node scaling (e.g. provisioning a new node and attaching it to a cluster). + +A CaaS Manager leverages the closed-loop desired state configuration management concept that Kubernetes itself enables. Meaning, the CaaS Manager takes the desired state of a CaaS Cluster as input and the controller must be able to maintain that desired state through a series of closed loops. + diff --git a/doc/ref_arch/kubernetes/chapters/chapter04.md b/doc/ref_arch/kubernetes/chapters/chapter04.md index de5e4350e6..983f85d720 100644 --- a/doc/ref_arch/kubernetes/chapters/chapter04.md +++ b/doc/ref_arch/kubernetes/chapters/chapter04.md @@ -1,24 +1,7 @@ -[<< Back](../../kubernetes) -# 4. Component Level Architecture +# Component Level Architecture -

scope

- -## Table of Contents - -- [4. Component Level Architecture](#4-component-level-architecture) - - [4.1 Introduction](#41-introduction) - - [4.2 Kubernetes Node](#42-kubernetes-node) - - [4.3 Kubernetes](#43-kubernetes) - - [4.4 Container runtimes](#44-container-runtimes) - - [4.5 Networking solutions](#45-networking-solutions) - - [4.6 Storage components](#46-storage-components) - - [4.7 Service meshes](#47-service-meshes) - - [4.8 Kubernetes Application package manager](#48-kubernetes-application-package-manager) - - [4.9 Kubernetes workloads](#49-kubernetes-workloads) - - [4.10 Additional required components](#410-additional-required-components) - -## 4.1 Introduction +## Introduction This chapter describes in detail the Kubernetes Reference Architecture in terms of the functional capabilities and how they relate to the Reference Model @@ -34,12 +17,11 @@ Reference Implementation and any vendor or community implementations. Figure 4-1 below shows the architectural components that are described in the subsequent sections of this chapter. -

-

Figure 4-1: Kubernetes Reference Architecture

+![**Figure 4-1:** Kubernetes Reference Architecture](../figures/ch04_k8s_architecture.png) + +**Figure 4-1:** Kubernetes Reference Architecture -## 4.2 Kubernetes Node +## Kubernetes Node This section describes the configuration that will be applied to the physical or virtual machine and an installed Operating System. In order for a Kubernetes Node @@ -48,110 +30,110 @@ the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| -|`ra2.ch.001`|Huge pages|When hosting workloads matching the Network Intensive profile, it **must** be possible to enable Huge pages (2048KiB and 1048576KiB) within the Kubernetes Node OS, exposing schedulable resources `hugepages-2Mi` and `hugepages-1Gi`.|[infra.com.cfg.004](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ch.002`|SR-IOV capable NICs|When hosting workloads matching the Network Intensive profile, the physical machines on which the Kubernetes Nodes run **must** be equipped with NICs that are SR-IOV capable.|[e.cap.013](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.003`|SR-IOV Virtual Functions|When hosting workloads matching the Network Intensive profile, SR-IOV virtual functions (VFs) **must** be configured within the Kubernetes Node OS, as the SR-IOV Device Plugin does not manage the creation of these VFs.|[e.cap.013](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ch.004`|CPU Simultaneous Multi-Threading (SMT)|SMT **must** be enabled in the BIOS on the physical machine on which the Kubernetes Node runs.|[infra.hw.cpu.cfg.004](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.005`|CPU Allocation Ratio - VMs|For Kubernetes nodes running as Virtual Machines, the CPU allocation ratio between vCPU and physical CPU core **must** be 1:1.|[infra.com.cfg.001](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|| -|`ra2.ch.006`|CPU Allocation Ratio - Pods|To ensure the CPU allocation ratio between vCPU and physical CPU core is 1:1, the sum of CPU requests and limits by containers in Pod specifications **must** remain less than the allocatable quantity of CPU resources (i.e. `requests.cpu < allocatable.cpu` and `limits.cpu < allocatable.cpu`).|[infra.com.cfg.001](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.007`|IPv6DualStack|To support IPv4/IPv6 dual stack networking, the Kubernetes Node OS **must** support and be allocated routable IPv4 and IPv6 addresses.|[req.inf.ntw.04](./chapter02.md#23-kubernetes-architecture-requirements)|| -|`ra2.ch.008`|Physical CPU Quantity|The physical machines on which the Kubernetes Nodes run **must** be equipped with at least 2 physical sockets, each with at least 20 CPU cores.|[infra.hw.cpu.cfg.001](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)
[infra.hw.cpu.cfg.002](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.009`|Physical Storage|The physical machines on which the Kubernetes Nodes run **should** be equipped with Sold State Drives (SSDs).|[infra.hw.stg.ssd.cfg.002](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.010`|Local Filesystem Storage Quantity|The Kubernetes Nodes **must** be equipped with local filesystem capacity of at least 320GB for unpacking and executing containers. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc.|[e.cap.003](./chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.011`|Virtual Node CPU Quantity|If using VMs, the Kubernetes Nodes **must** be equipped with at least 16 vCPUs. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc.|[e.cap.001](./chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|| -|`ra2.ch.012`|Kubernetes Node RAM Quantity|The Kubernetes Nodes **must** be equipped with at least 32GB of RAM. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc.|[e.cap.002](./chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.013`|Physical NIC Quantity|The physical machines on which the Kubernetes Nodes run **must** be equipped with at least four (4) Network Interface Card (NIC) ports.|[infra.hw.nic.cfg.001](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.014`|Physical NIC Speed - Basic Profile|The speed of NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Basic Profile **must** be at least 10Gbps.|[infra.hw.nic.cfg.002](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| -|`ra2.ch.015`|Physical NIC Speed - Network Intensive Profile|The speed of NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Network Intensive profile **must** be at least 25Gbps.|[infra.hw.nic.cfg.002](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#33-infrastructure-requirements)| +|`ra2.ch.001`|Huge pages|When hosting workloads matching the Network Intensive profile, it **must** be possible to enable Huge pages (2048KiB and 1048576KiB) within the Kubernetes Node OS, exposing schedulable resources `hugepages-2Mi` and `hugepages-1Gi`.|[infra.com.cfg.004](./chapter02.md#cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ch.002`|SR-IOV capable NICs|When hosting workloads matching the Network Intensive profile, the physical machines on which the Kubernetes Nodes run **must** be equipped with NICs that are SR-IOV capable.|[e.cap.013](./chapter02.md#cloud-infrastructure-software-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.003`|SR-IOV Virtual Functions|When hosting workloads matching the Network Intensive profile, SR-IOV virtual functions (VFs) **must** be configured within the Kubernetes Node OS, as the SR-IOV Device Plugin does not manage the creation of these VFs.|[e.cap.013](./chapter02.md#cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ch.004`|CPU Simultaneous Multi-Threading (SMT)|SMT **must** be enabled in the BIOS on the physical machine on which the Kubernetes Node runs.|[infra.hw.cpu.cfg.004](./chapter02.md#cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.005`|CPU Allocation Ratio - VMs|For Kubernetes nodes running as Virtual Machines, the CPU allocation ratio between vCPU and physical CPU core **must** be 1:1.|[infra.com.cfg.001](./chapter02.md#cloud-infrastructure-software-profile-requirements)|| +|`ra2.ch.006`|CPU Allocation Ratio - Pods|To ensure the CPU allocation ratio between vCPU and physical CPU core is 1:1, the sum of CPU requests and limits by containers in Pod specifications **must** remain less than the allocatable quantity of CPU resources (i.e. `requests.cpu < allocatable.cpu` and `limits.cpu < allocatable.cpu`).|[infra.com.cfg.001](./chapter02.md#cloud-infrastructure-software-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.007`|IPv6DualStack|To support IPv4/IPv6 dual stack networking, the Kubernetes Node OS **must** support and be allocated routable IPv4 and IPv6 addresses.|[req.inf.ntw.04](./chapter02.md#kubernetes-architecture-requirements)|| +|`ra2.ch.008`|Physical CPU Quantity|The physical machines on which the Kubernetes Nodes run **must** be equipped with at least 2 physical sockets, each with at least 20 CPU cores.|[infra.hw.cpu.cfg.001](./chapter02.md#cloud-infrastructure-hardware-profile-requirements), [infra.hw.cpu.cfg.002](./chapter02.md#cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.009`|Physical Storage|The physical machines on which the Kubernetes Nodes run **should** be equipped with Sold State Drives (SSDs).|[infra.hw.stg.ssd.cfg.002](./chapter02.md#cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.010`|Local Filesystem Storage Quantity|The Kubernetes Nodes **must** be equipped with local filesystem capacity of at least 320GB for unpacking and executing containers. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc.|[e.cap.003](./chapter02.md#cloud-infrastructure-software-profile-capabilities)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.011`|Virtual Node CPU Quantity|If using VMs, the Kubernetes Nodes **must** be equipped with at least 16 vCPUs. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc.|[e.cap.001](./chapter02.md#cloud-infrastructure-software-profile-capabilities)|| +|`ra2.ch.012`|Kubernetes Node RAM Quantity|The Kubernetes Nodes **must** be equipped with at least 32GB of RAM. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc.|[e.cap.002](./chapter02.md#cloud-infrastructure-software-profile-capabilities)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.013`|Physical NIC Quantity|The physical machines on which the Kubernetes Nodes run **must** be equipped with at least four (4) Network Interface Card (NIC) ports.|[infra.hw.nic.cfg.001](./chapter02.md#cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.014`|Physical NIC Speed - Basic Profile|The speed of NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Basic Profile **must** be at least 10Gbps.|[infra.hw.nic.cfg.002](./chapter02.md#cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| +|`ra2.ch.015`|Physical NIC Speed - Network Intensive Profile|The speed of NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Network Intensive profile **must** be at least 25Gbps.|[infra.hw.nic.cfg.002](./chapter02.md#cloud-infrastructure-hardware-profile-requirements)|[3.3](../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements)| |`ra2.ch.016`|Physical PCIe slots|The physical machines on which the Kubernetes Nodes run **must** be equipped with at least eight (8) Gen3.0 PCIe slots, each with at least eight (8) lanes.| -|`ra2.ch.017`|Immutable infrastructure|Whether physical or virtual machines are used, the Kubernetes Node **must not** be changed after it is instantiated. New changes to the Kubernetes Node must be implemented as new Node instances. This covers any changes from BIOS through Operating System to running processes and all associated configurations.|[req.gen.cnt.02](./chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ch.018`|NFD|[Node Feature Discovery](https://kubernetes-sigs.github.io/node-feature-discovery/stable/get-started/index.html) **must** be used to advertise the detailed software and hardware capabilities of each node in the Kubernetes Cluster.|TBD|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| +|`ra2.ch.017`|Immutable infrastructure|Whether physical or virtual machines are used, the Kubernetes Node **must not** be changed after it is instantiated. New changes to the Kubernetes Node must be implemented as new Node instances. This covers any changes from BIOS through Operating System to running processes and all associated configurations.|[req.gen.cnt.02](./chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ch.018`|NFD|[Node Feature Discovery](https://kubernetes-sigs.github.io/node-feature-discovery/stable/get-started/index.html) **must** be used to advertise the detailed software and hardware capabilities of each node in the Kubernetes Cluster.|TBD|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| -

Table 4-1: Node Specifications

+**Table 4-1:** Node Specifications -## 4.3 Kubernetes +## Kubernetes In order for the Kubernetes components to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| -|`ra2.k8s.001`|Kubernetes Conformance|The Kubernetes distribution, product, or installer used in the implementation **must** be listed in the [Kubernetes Distributions and Platforms document](https://docs.google.com/spreadsheets/d/1uF9BoDzzisHSQemXHIKegMhuythuq_GL3N1mlUUK2h0/edit#gid=0) and marked (X) as conformant for the Kubernetes version defined in [README](../README.md#required-versions-of-most-important-components).|[req.gen.cnt.03](./chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.k8s.002`|Highly available etcd|An implementation **must** consist of either three, five or seven nodes running the etcd service (can be colocated on the master nodes, or can run on separate nodes, but not on worker nodes).|[req.gen.rsl.02 req.gen.avl.01](./chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.k8s.003`|Highly available control plane|An implementation **must** consist of at least one master node per availability zone or fault domain to ensure the high availability and resilience of the Kubernetes control plane services.|[req.gen.rsl.02](./chapter02.md#23-kubernetes-architecture-requirements)
[req.gen.avl.01](./chapter02.md#23-kubernetes-architecture-requirements)| -|`ra2.k8s.012`|Control plane services|A master node **must** run at least the following Kubernetes control plane services: `kube-apiserver`, `kube-scheduler` and `kube-controller-manager`.|[req.gen.rsl.02](./chapter02.md#23-kubernetes-architecture-requirements)
[req.gen.avl.01](./chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.k8s.004`|Highly available worker nodes|An implementation **must** consist of at least one worker node per availability zone or fault domain to ensure the high availability and resilience of workloads managed by Kubernetes|[req.gen.rsl.01](./chapter02.md#23-kubernetes-architecture-requirements)
[req.gen.avl.01](./chapter02.md#23-kubernetes-architecture-requirements)
[req.kcm.gen.02](./chapter02.md#23-kubernetes-architecture-requirements)
[req.inf.com.01](./chapter02.md#23-kubernetes-architecture-requirements)| +|`ra2.k8s.001`|Kubernetes Conformance|The Kubernetes distribution, product, or installer used in the implementation **must** be listed in the [Kubernetes Distributions and Platforms document](https://docs.google.com/spreadsheets/d/1uF9BoDzzisHSQemXHIKegMhuythuq_GL3N1mlUUK2h0/edit#gid=0) and marked (X) as conformant for the Kubernetes version defined in [README](../README.md#required-versions-of-most-important-components).|[req.gen.cnt.03](./chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.k8s.002`|Highly available etcd|An implementation **must** consist of either three, five or seven nodes running the etcd service (can be colocated on the master nodes, or can run on separate nodes, but not on worker nodes).|[req.gen.rsl.02 req.gen.avl.01](./chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.k8s.003`|Highly available control plane|An implementation **must** consist of at least one master node per availability zone or fault domain to ensure the high availability and resilience of the Kubernetes control plane services.|[req.gen.rsl.02](./chapter02.md#kubernetes-architecture-requirements), [req.gen.avl.01](./chapter02.md#kubernetes-architecture-requirements)| +|`ra2.k8s.012`|Control plane services|A master node **must** run at least the following Kubernetes control plane services: `kube-apiserver`, `kube-scheduler` and `kube-controller-manager`.|[req.gen.rsl.02](./chapter02.md#kubernetes-architecture-requirements), [req.gen.avl.01](./chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.k8s.004`|Highly available worker nodes|An implementation **must** consist of at least one worker node per availability zone or fault domain to ensure the high availability and resilience of workloads managed by Kubernetes|[req.gen.rsl.01](./chapter02.md#kubernetes-architecture-requirements), [req.gen.avl.01](./chapter02.md#kubernetes-architecture-requirements), [req.kcm.gen.02](./chapter02.md#kubernetes-architecture-requirements), [req.inf.com.01](./chapter02.md#kubernetes-architecture-requirements)| |`ra2.k8s.005`|Kubernetes API Version|In alignment with the [Kubernetes version support policy](https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions), an implementation **must** use a Kubernetes version as per the subcomponent versions table in [README](../README.md#required-versions-of-most-important-components).|TBC|| -|`ra2.k8s.006`|NUMA Support|When hosting workloads matching the Network Intensive profile, the `TopologyManager` and `CPUManager` feature gates **must** be enabled and configured on the kubelet (note, TopologyManager is enabled by default in Kubernetes v1.18 and later, with CPUManager enabled by default in Kubernetes v1.10 and later). `--feature-gates="...,TopologyManager=true,CPUManager=true" --topology-manager-policy=single-numa-node --cpu-manager-policy=static`|[e.cap.007](chapter02.md#221-cloud-infrastructure-software-profile-capabilities) [infra.com.cfg.002](./chapter02.md#223-cloud-infrastructure-software-profile-requirements) [infra.hw.cpu.cfg.003](./chapter02.md#224-cloud-infrastructure-hardware-profile-requirements)| -|`ra2.k8s.007`|DevicePlugins Feature Gate|When hosting workloads matching the Network Intensive profile, the DevicePlugins feature gate **must** be enabled (note, this is enabled by default in Kubernetes v1.10 or later). `--feature-gates="...,DevicePlugins=true,..."`|Various, e.g. [e.cap.013](chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.k8s.008`|System Resource Reservations|To avoid resource starvation issues on nodes, the implementation of the architecture **must** reserve compute resources for system daemons and Kubernetes system daemons such as kubelet, container runtime, etc. Use the following kubelet flags: `--reserved-cpus=[a-z]`, using two of `a-z` to reserve 2 SMT threads.|[i.cap.014](chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|| -|`ra2.k8s.009`|CPU Pinning|When hosting workloads matching the Network Intensive profile, in order to support CPU Pinning, the kubelet **must** be started with the `--cpu-manager-policy=static` option. (Note, only containers in `Guaranteed` pods - where CPU resource `requests` and `limits` are identical - and configured with positive-integer CPU `requests` will take advantage of this. All other Pods will run on CPUs in the remaining shared pool.)|[infra.com.cfg.003](./chapter02.md#223-cloud-infrastructure-software-profile-requirements)| -|`ra2.k8s.010`|IPv6DualStack|To support IPv6 and IPv4, the `IPv6DualStack` feature gate **must** be enabled on various components (requires Kubernetes v1.16 or later). kube-apiserver: `--feature-gates="IPv6DualStack=true"`. kube-controller-manager: `--feature-gates="IPv6DualStack=true" --cluster-cidr=, --service-cluster-ip-range=, --node-cidr-mask-size-ipv4 ¦ --node-cidr-mask-size-ipv6` defaults to /24 for IPv4 and /64 for IPv6. kubelet: `--feature-gates="IPv6DualStack=true"`. kube-proxy: `--cluster-cidr=, --feature-gates="IPv6DualStack=true"`|[req.inf.ntw.04](./chapter02.md#23-kubernetes-architecture-requirements)| -|`ra2.k8s.011`|Anuket profile labels|To clearly identify which worker nodes are compliant with the different profiles defined by Anuket the worker nodes **must** be labelled according to the following pattern: an `anuket.io/profile/basic` label must be set to `true` on the worker node if it can fulfil the requirements of the basic profile and an `anuket.io/profile/network-intensive` label must be set to `true` on the worker node if it can fulfil the requirements of the network intensive profile. The requirements for both profiles can be found in [chapter 2](./chapter02.md#22-reference-model-requirements)||| -|`ra2.k8s.012`|Kubernetes APIs|Kubernetes [Alpha API](https://kubernetes.io/docs/reference/using-api/#api-versioning) are recommended only for testing, therefore all Alpha APIs **must** be disabled.|[req.int.api.03](./chapter02.md#22-reference-model-requirements)|| -|`ra2.k8s.013`|Kubernetes APIs|Backward compatibility of all supported GA APIs of Kubernetes **must** be supported. |[req.int.api.04](./chapter02.md#23-kubernetes-architecture-requirements)|| -|`ra2.k8s.014`|Security Groups|Kubernetes **must** support NetworkPolicy feature. |[infra.net.cfg.004](chapter02.md#23-kubernetes-architecture-requirements)|| +|`ra2.k8s.006`|NUMA Support|When hosting workloads matching the Network Intensive profile, the `TopologyManager` and `CPUManager` feature gates **must** be enabled and configured on the kubelet (note, TopologyManager is enabled by default in Kubernetes v1.18 and later, with CPUManager enabled by default in Kubernetes v1.10 and later). `--feature-gates="...,TopologyManager=true,CPUManager=true" --topology-manager-policy=single-numa-node --cpu-manager-policy=static`|[e.cap.007](chapter02.md#cloud-infrastructure-software-profile-capabilities) [infra.com.cfg.002](./chapter02.md#cloud-infrastructure-software-profile-requirements) [infra.hw.cpu.cfg.003](./chapter02.md#cloud-infrastructure-hardware-profile-requirements)| +|`ra2.k8s.007`|DevicePlugins Feature Gate|When hosting workloads matching the Network Intensive profile, the DevicePlugins feature gate **must** be enabled (note, this is enabled by default in Kubernetes v1.10 or later). `--feature-gates="...,DevicePlugins=true,..."`|Various, e.g. [e.cap.013](chapter02.md#cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.k8s.008`|System Resource Reservations|To avoid resource starvation issues on nodes, the implementation of the architecture **must** reserve compute resources for system daemons and Kubernetes system daemons such as kubelet, container runtime, etc. Use the following kubelet flags: `--reserved-cpus=[a-z]`, using two of `a-z` to reserve 2 SMT threads.|[i.cap.014](chapter02.md#cloud-infrastructure-software-profile-capabilities)|| +|`ra2.k8s.009`|CPU Pinning|When hosting workloads matching the Network Intensive profile, in order to support CPU Pinning, the kubelet **must** be started with the `--cpu-manager-policy=static` option. (Note, only containers in `Guaranteed` pods - where CPU resource `requests` and `limits` are identical - and configured with positive-integer CPU `requests` will take advantage of this. All other Pods will run on CPUs in the remaining shared pool.)|[infra.com.cfg.003](./chapter02.md#cloud-infrastructure-software-profile-requirements)| +|`ra2.k8s.010`|IPv6DualStack|To support IPv6 and IPv4, the `IPv6DualStack` feature gate **must** be enabled on various components (requires Kubernetes v1.16 or later). kube-apiserver: `--feature-gates="IPv6DualStack=true"`. kube-controller-manager: `--feature-gates="IPv6DualStack=true" --cluster-cidr=, --service-cluster-ip-range=, --node-cidr-mask-size-ipv4 ¦ --node-cidr-mask-size-ipv6` defaults to /24 for IPv4 and /64 for IPv6. kubelet: `--feature-gates="IPv6DualStack=true"`. kube-proxy: `--cluster-cidr=, --feature-gates="IPv6DualStack=true"`|[req.inf.ntw.04](./chapter02.md#kubernetes-architecture-requirements)| +|`ra2.k8s.011`|Anuket profile labels|To clearly identify which worker nodes are compliant with the different profiles defined by Anuket the worker nodes **must** be labelled according to the following pattern: an `anuket.io/profile/basic` label must be set to `true` on the worker node if it can fulfil the requirements of the basic profile and an `anuket.io/profile/network-intensive` label must be set to `true` on the worker node if it can fulfil the requirements of the network intensive profile. The requirements for both profiles can be found in [chapter 2](./chapter02.md#reference-model-requirements)||| +|`ra2.k8s.012`|Kubernetes APIs|Kubernetes [Alpha API](https://kubernetes.io/docs/reference/using-api/#api-versioning) are recommended only for testing, therefore all Alpha APIs **must** be disabled.|[req.int.api.03](./chapter02.md#reference-model-requirements)|| +|`ra2.k8s.013`|Kubernetes APIs|Backward compatibility of all supported GA APIs of Kubernetes **must** be supported. |[req.int.api.04](./chapter02.md#kubernetes-architecture-requirements)|| +|`ra2.k8s.014`|Security Groups|Kubernetes **must** support NetworkPolicy feature. |[infra.net.cfg.004](chapter02.md#kubernetes-architecture-requirements)|| |`ra2.k8s.015`|Publishing Services (ServiceTypes)|Kubernetes **must** support LoadBalancer [Publishing Service (ServiceTypes)](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types). |[req.inf.ntw.15](chapter02.md#kubernetes-architecture-requirements)|| |`ra2.k8s.016`|Publishing Services (ServiceTypes)|Kubernetes **must** support [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/). |[req.inf.ntw.16](chapter02.md#kubernetes-architecture-requirements)|| |`ra2.k8s.017`|Publishing Services (ServiceTypes)|Kubernetes **should** support NodePort [Publishing Service (ServiceTypes)](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types). |[req.inf.ntw.17](chapter02.md#kubernetes-architecture-requirements)|| |`ra2.k8s.018`|Publishing Services (ServiceTypes)|Kubernetes **should** support ExternalName [Publishing Service (ServiceTypes)](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types). |[req.inf.ntw.18](chapter02.md#kubernetes-architecture-requirements)|| -|`ra2.k8s.019`|Kubernetes APIs|Kubernetes Beta APIs **must** be supported only when a stable GA of the same version doesn't exist. |[req.int.api.04](./chapter02.md#23-kubernetes-architecture-requirements)|| +|`ra2.k8s.019`|Kubernetes APIs|Kubernetes Beta APIs **must** be supported only when a stable GA of the same version doesn't exist. |[req.int.api.04](./chapter02.md#kubernetes-architecture-requirements)|| -

Table 4-2: Kubernetes Specifications

+**Table 4-2:** Kubernetes Specifications -## 4.4 Container runtimes +## Container runtimes |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| -|`ra2.crt.001`|Conformance with OCI 1.0 runtime spec|The container runtime **must** be implemented as per the [OCI 1.0](https://github.com/opencontainers/runtime-spec/blob/master/spec.md) (Open Container Initiative 1.0) specification.|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.crt.002`|Kubernetes Container Runtime Interface (CRI)|The Kubernetes container runtime **must** be implemented as per the [Kubernetes Container Runtime Interface (CRI)](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| +|`ra2.crt.001`|Conformance with OCI 1.0 runtime spec|The container runtime **must** be implemented as per the [OCI 1.0](https://github.com/opencontainers/runtime-spec/blob/master/spec.md) (Open Container Initiative 1.0) specification.|[req.gen.ost.01](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.crt.002`|Kubernetes Container Runtime Interface (CRI)|The Kubernetes container runtime **must** be implemented as per the [Kubernetes Container Runtime Interface (CRI)](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/)|[req.gen.ost.01](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| -

Table 4-3: Container Runtime Specifications

+**Table 4-3:** Container Runtime Specifications -## 4.5 Networking solutions +## Networking solutions In order for the networking solution(s) to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| -|`ra2.ntw.001`|Centralised network administration|The networking solution deployed within the implementation **must** be administered through the Kubernetes API using native Kubernetes API resources and objects, or Custom Resources.|[req.inf.ntw.03](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.002`|Default Pod Network - CNI|The networking solution deployed within the implementation **must** use a CNI-conformant Network Plugin for the Default Pod Network, as the alternative (kubenet) does not support cross-node networking or Network Policies.|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)
[req.inf.ntw.08](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.003`|Multiple connection points|The networking solution deployed within the implementation **must** support the capability to connect at least FIVE connection points to each Pod, which are additional to the default connection point managed by the default Pod network CNI plugin.|[e.cap.004](chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.004`|Multiple connection points presentation|The networking solution deployed within the implementation **must** ensure that all additional non-default connection points are requested by Pods using standard Kubernetes resource scheduling mechanisms such as annotations or container resource requests and limits.|[req.inf.ntw.03](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.005`|Multiplexer/meta-plugin|The networking solution deployed within the implementation **may** use a multiplexer/meta-plugin.|[req.inf.ntw.06](chapter02.md#23-kubernetes-architecture-requirements)
[req.inf.ntw.07](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.006`|Multiplexer/meta-plugin CNI Conformance|If used, the selected multiplexer/meta-plugin **must** integrate with the Kubernetes control plane via CNI.|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.007`|Multiplexer/meta-plugin CNI Plugins|If used, the selected multiplexer/meta-plugin **must** support the use of multiple CNI-conformant Network Plugins.|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)
[req.inf.ntw.06](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.008`|SR-IOV Device Plugin for Network Intensive|When hosting workloads that match the Network Intensive profile and require SR-IOV acceleration, a Device Plugin for SR-IOV **must** be used to configure the SR-IOV devices and advertise them to the `kubelet`.|[e.cap.013](chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.009`|Multiple connection points with multiplexer/meta-plugin|When a multiplexer/meta-plugin is used, the additional non-default connection points **must** be managed by a CNI-conformant Network Plugin.|[req.gen.ost.01](chapter02.md#23-kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.010`|User plane networking|When hosting workloads matching the Network Intensive profile, CNI network plugins that support the use of DPDK, VPP, and/or SR-IOV **must** be deployed as part of the networking solution.|[infra.net.acc.cfg.001](chapter02.md#223-cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.011`|NATless connectivity|When hosting workloads that require source and destination IP addresses to be preserved in the traffic headers, a NATless CNI plugin that exposes the pod IP directly to the external networks (e.g. Calico, MACVLAN or IPVLAN CNI plugins) **must** be used.|[req.inf.ntw.14](chapter02.md#23-kubernetes-architecture-requirements)| -|`ra2.ntw.012`|Device Plugins|When hosting workloads matching the Network Intensive profile that require the use of FPGA, SR-IOV or other Acceleration Hardware, a Device Plugin for that FPGA or Acceleration Hardware **must** be used.|[e.cap.016](chapter02.md#221-cloud-infrastructure-software-profile-capabilities), [e.cap.013](chapter02.md#221-cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#431-installation-on-bare-metal-infratructure)| -|`ra2.ntw.013`|Dual stack CNI|The networking solution deployed within the implementation **must** use a CNI-conformant Network Plugin that is able to support dual-stack IPv4/IPv6 networking.|[req.inf.ntw.04](chapter02.md#23-kubernetes-architecture-requirements)| -|`ra2.ntw.014`|Security Groups|The networking solution deployed within the implementation **must** support network policies.|[infra.net.cfg.004](chapter02.md#223-cloud-infrastructure-software-profile-requirements)| -|`ra2.ntw.015`|IPAM plugin for multiplexer|When a multiplexer/meta-plugin is used, a CNI-conformant IPAM Network Plugin **must** be installed to allocate IP addresses for secondary network interfaces across all nodes of the cluster.|[req.inf.ntw.10](chapter02.md#23-kubernetes-architecture-requirements)| - -

Table 4-4: Networking Solution Specifications

- -## 4.6 Storage components +|`ra2.ntw.001`|Centralised network administration|The networking solution deployed within the implementation **must** be administered through the Kubernetes API using native Kubernetes API resources and objects, or Custom Resources.|[req.inf.ntw.03](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.002`|Default Pod Network - CNI|The networking solution deployed within the implementation **must** use a CNI-conformant Network Plugin for the Default Pod Network, as the alternative (kubenet) does not support cross-node networking or Network Policies.|[req.gen.ost.01](chapter02.md#kubernetes-architecture-requirements), [req.inf.ntw.08](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.003`|Multiple connection points|The networking solution deployed within the implementation **must** support the capability to connect at least FIVE connection points to each Pod, which are additional to the default connection point managed by the default Pod network CNI plugin.|[e.cap.004](chapter02.md#cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.004`|Multiple connection points presentation|The networking solution deployed within the implementation **must** ensure that all additional non-default connection points are requested by Pods using standard Kubernetes resource scheduling mechanisms such as annotations or container resource requests and limits.|[req.inf.ntw.03](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.005`|Multiplexer/meta-plugin|The networking solution deployed within the implementation **may** use a multiplexer/meta-plugin.|[req.inf.ntw.06](chapter02.md#kubernetes-architecture-requirements), [req.inf.ntw.07](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.006`|Multiplexer/meta-plugin CNI Conformance|If used, the selected multiplexer/meta-plugin **must** integrate with the Kubernetes control plane via CNI.|[req.gen.ost.01](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.007`|Multiplexer/meta-plugin CNI Plugins|If used, the selected multiplexer/meta-plugin **must** support the use of multiple CNI-conformant Network Plugins.|[req.gen.ost.01](chapter02.md#kubernetes-architecture-requirements), [req.inf.ntw.06](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.008`|SR-IOV Device Plugin for Network Intensive|When hosting workloads that match the Network Intensive profile and require SR-IOV acceleration, a Device Plugin for SR-IOV **must** be used to configure the SR-IOV devices and advertise them to the `kubelet`.|[e.cap.013](chapter02.md#cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.009`|Multiple connection points with multiplexer/meta-plugin|When a multiplexer/meta-plugin is used, the additional non-default connection points **must** be managed by a CNI-conformant Network Plugin.|[req.gen.ost.01](chapter02.md#kubernetes-architecture-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.010`|User plane networking|When hosting workloads matching the Network Intensive profile, CNI network plugins that support the use of DPDK, VPP, and/or SR-IOV **must** be deployed as part of the networking solution.|[infra.net.acc.cfg.001](chapter02.md#cloud-infrastructure-software-profile-requirements)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.011`|NATless connectivity|When hosting workloads that require source and destination IP addresses to be preserved in the traffic headers, a NATless CNI plugin that exposes the pod IP directly to the external networks (e.g. Calico, MACVLAN or IPVLAN CNI plugins) **must** be used.|[req.inf.ntw.14](chapter02.md#kubernetes-architecture-requirements)| +|`ra2.ntw.012`|Device Plugins|When hosting workloads matching the Network Intensive profile that require the use of FPGA, SR-IOV or other Acceleration Hardware, a Device Plugin for that FPGA or Acceleration Hardware **must** be used.|[e.cap.016](chapter02.md#cloud-infrastructure-software-profile-capabilities), [e.cap.013](chapter02.md#cloud-infrastructure-software-profile-capabilities)|[4.3.1](../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure)| +|`ra2.ntw.013`|Dual stack CNI|The networking solution deployed within the implementation **must** use a CNI-conformant Network Plugin that is able to support dual-stack IPv4/IPv6 networking.|[req.inf.ntw.04](chapter02.md#kubernetes-architecture-requirements)| +|`ra2.ntw.014`|Security Groups|The networking solution deployed within the implementation **must** support network policies.|[infra.net.cfg.004](chapter02.md#cloud-infrastructure-software-profile-requirements)| +|`ra2.ntw.015`|IPAM plugin for multiplexer|When a multiplexer/meta-plugin is used, a CNI-conformant IPAM Network Plugin **must** be installed to allocate IP addresses for secondary network interfaces across all nodes of the cluster.|[req.inf.ntw.10](chapter02.md#kubernetes-architecture-requirements)| + +**Table 4-4:** Networking Solution Specifications + +## Storage components In order for the storage solutions to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| -|`ra2.stg.001`| Ephemeral Storage | An implementation must support ephemeral storage, for the unpacked container images to be stored and executed from, as a directory in the filesystem on the worker node on which the container is running.
See the [Container runtimes](#44-container-runtimes) section above for more information on how this meets the requirement for ephemeral storage for containers. || +|`ra2.stg.001`| Ephemeral Storage | An implementation must support ephemeral storage, for the unpacked container images to be stored and executed from, as a directory in the filesystem on the worker node on which the container is running. See the [Container runtimes](#container-runtimes) section above for more information on how this meets the requirement for ephemeral storage for containers. || |`ra2.stg.002`| Kubernetes Volumes | An implementation may attach additional storage to containers using Kubernetes Volumes. || |`ra2.stg.003`| Kubernetes Volumes | An implementation may use Volume Plugins (see `ra2.stg.005` below) to allow the use of a storage protocol (e.g., iSCSI, NFS) or management API (e.g., Cinder, EBS) for the attaching and mounting of storage into a Pod. || -|`ra2.stg.004`| Persistent Volumes | An implementation may support Kubernetes Persistent Volumes (PV) to provide persistent storage for Pods.
Persistent Volumes exist independent of the lifecycle of containers and/or pods. |[req.inf.stg.01](chapter02.md#23-kubernetes-architecture-requirements)| +|`ra2.stg.004`| Persistent Volumes | An implementation may support Kubernetes Persistent Volumes (PV) to provide persistent storage for Pods. Persistent Volumes exist independent of the lifecycle of containers and/or pods. |[req.inf.stg.01](chapter02.md#kubernetes-architecture-requirements)| |`ra2.stg.005`| Storage Volume Types | An implementation must support the following Volume types: `emptyDir`, `ConfigMap`, `Secret` and `PersistentVolumeClaim`. Other Volume plugins may be supported to allow for the use of a range of backend storage systems. || -|`ra2.stg.006`| Container Storage Interface (CSI) | An implementation may support the Container Storage Interface (CSI), an Out-of-tree plugin.
In order to support CSI, the feature gates `CSIDriverRegistry` and `CSINodeInfo` must be enabled.
The implementation must use a CSI driver (a full list of CSI drivers can be found [here](https://kubernetes-csi.github.io/docs/drivers.html)).
An implementation may support ephemeral storage through a CSI-compatible volume plugin in which case the `CSIInlineVolume` feature gate must be enabled.
An implementation may support Persistent Volumes through a CSI-compatible volume plugin in which case the `CSIPersistentVolume` feature gate must be enabled. | | +|`ra2.stg.006`| Container Storage Interface (CSI) | An implementation may support the Container Storage Interface (CSI), an Out-of-tree plugin. In order to support CSI, the feature gates `CSIDriverRegistry` and `CSINodeInfo` must be enabled. The implementation must use a CSI driver (a full list of CSI drivers can be found [here](https://kubernetes-csi.github.io/docs/drivers.html)). An implementation may support ephemeral storage through a CSI-compatible volume plugin in which case the `CSIInlineVolume` feature gate must be enabled. An implementation may support Persistent Volumes through a CSI-compatible volume plugin in which case the `CSIPersistentVolume` feature gate must be enabled. | | |`ra2.stg.007`| | An implementation should use Kubernetes Storage Classes to support automation and the separation of concerns between providers of a service and consumers of the service. | | -

Table 4-6: Storage Solution Specifications

+**Table 4-6:** Storage Solution Specifications A note on object storage: -- This Reference Architecture does not include any specifications for object +* This Reference Architecture does not include any specifications for object storage, as this is neither a native Kubernetes object, nor something that is required by CSI drivers. Object storage is an application-level requirement that would ordinarily be provided by a highly scalable service offering rather @@ -159,24 +141,24 @@ than being something an individual Kubernetes cluster could offer. > Todo: specifications/commentary to support req.inf.stg.04 (SDS) and req.inf.stg.05 (high performance and horizontally scalable storage). Also req.sec.gen.06 (storage resource isolation), req.sec.gen.10 (CIS - if applicable) and req.sec.zon.03 (data encryption at rest). -## 4.7 Service meshes +## Service meshes -Application service meshes are not in scope for the architecture. The service mesh is a dedicated infrastructure layer for handling service-to-service communication, and it is recommended to secure service-to-service communications within a cluster and to reduce the attack surface. The benefits of the service mesh framework are described in [5.4.3](./chapter05.md#543-use-transport-layer-security-and-service-mesh). In addition to securing communications, the use of a service mesh extends Kubernetes capabilities regarding observability and reliability. +Application service meshes are not in scope for the architecture. The service mesh is a dedicated infrastructure layer for handling service-to-service communication, and it is recommended to secure service-to-service communications within a cluster and to reduce the attack surface. The benefits of the service mesh framework are described in [5.4.3](./chapter05.md#use-transport-layer-security-and-service-mesh). In addition to securing communications, the use of a service mesh extends Kubernetes capabilities regarding observability and reliability. -Network service mesh specifications are handled in section [4.5 Networking solutions](#45-networking-solutions). +Network service mesh specifications are handled in section [4.5 Networking solutions](#networking-solutions). -## 4.8 Kubernetes Application package manager +## Kubernetes Application package manager In order for the application package managers to be conformant with the Reference Architecture they must be implemented as per the following specifications: |Ref|Specification|Details|Requirement Trace|Reference Implementation Trace| |---|---|---|---|---| -|`ra2.pkg.001`|API-based package management| A package manager must use the Kubernetes APIs to manage application artifacts. Cluster-side components such as Tiller are not supported. | [req.int.api.02](./chapter02.md#23-kubernetes-architecture-requirements) || +|`ra2.pkg.001`|API-based package management| A package manager must use the Kubernetes APIs to manage application artifacts. Cluster-side components such as Tiller are not supported. | [req.int.api.02](./chapter02.md#kubernetes-architecture-requirements) || -

Table 4-7: Kubernetes Application Package Manager Specifications

+**Table 4-7:** Kubernetes Application Package Manager Specifications -## 4.9 Kubernetes workloads +## Kubernetes workloads In order for the Kubernetes workloads to be conformant with the Reference Architecture they must be implemented as per the following specifications: @@ -188,14 +170,14 @@ Architecture they must be implemented as per the following specifications: |`ra2.app.003`| [Process](https://github.com/opencontainers/runtime-spec/blob/master/config.md#process) Parameter Group (OCI Spec) | Specifies the container process. | TBD | N/A | |`ra2.app.004`| [Hostname](https://github.com/opencontainers/runtime-spec/blob/master/config.md#hostname) Parameter Group (OCI Spec) | Specifies the container's hostname as seen by processes running inside the container. | TBD | N/A | |`ra2.app.005`| [User](https://github.com/opencontainers/runtime-spec/blob/master/config.md#user) Parameter Group (OCI Spec) | User for the process is a platform-specific structure that allows specific control over which user the process runs as. | TBD | N/A | -|`ra2.app.006`| Consumption of additional, non-default connection points | The workload must request additional non-default connection points through the use of workload annotations or resource requests and limits within the container spec passed to the Kubernetes API Server. | [req.int.api.01](chapter02.md#23-kubernetes-architecture-requirements) | N/A | -|`ra2.app.007`| Host Volumes | Workloads should not use `hostPath` volumes, as [Pods with identical configuration](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) (such as those created from a PodTemplate) may behave differently on different nodes due to different files on the nodes. | [req.kcm.gen.02](chapter02.md#23-kubernetes-architecture-requirements). | N/A | +|`ra2.app.006`| Consumption of additional, non-default connection points | The workload must request additional non-default connection points through the use of workload annotations or resource requests and limits within the container spec passed to the Kubernetes API Server. | [req.int.api.01](chapter02.md#kubernetes-architecture-requirements) | N/A | +|`ra2.app.007`| Host Volumes | Workloads should not use `hostPath` volumes, as [Pods with identical configuration](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) (such as those created from a PodTemplate) may behave differently on different nodes due to different files on the nodes. | [req.kcm.gen.02](chapter02.md#kubernetes-architecture-requirements). | N/A | |`ra2.app.008`| Infrastructure dependency | Workloads must not rely on the availability of the master nodes for the successful execution of their functionality (i.e. loss of the master nodes may affect non-functional behaviours such as healing and scaling, but components that are already running will continue to do so without issue). | TBD | N/A | |`ra2.app.009`| Device plugins | Workload descriptors must use the resources advertised by the device plugins to indicate their need for an FPGA, SR-IOV or other acceleration device. | TBD | N/A | |`ra2.app.010`| Node Feature Discovery (NFD)|Workload descriptors must use the labels advertised by [Node Feature Discovery](https://kubernetes-sigs.github.io/node-feature-discovery/stable/get-started/index.html) to indicate which node software of hardware features they need. | TBD | N/A | -

Table 4-8: Kubernetes Workload Specifications

+**Table 4-8:** Kubernetes Workload Specifications -## 4.10 Additional required components +## Additional required components > This chapter should list any additional components needed to provide the services defined in Chapter 3.2 (e.g., Prometheus) diff --git a/doc/ref_arch/kubernetes/chapters/chapter04.rst b/doc/ref_arch/kubernetes/chapters/chapter04.rst new file mode 100644 index 0000000000..08c0c45878 --- /dev/null +++ b/doc/ref_arch/kubernetes/chapters/chapter04.rst @@ -0,0 +1,212 @@ +Component Level Architecture +============================ + +Introduction +------------ + +This chapter describes in detail the Kubernetes Reference Architecture in terms +of the functional capabilities and how they relate to the Reference Model +requirements, i.e. how the infrastructure profiles are determined, documented +and delivered. + +The specifications defined in this chapter will be detailed with unique +identifiers, which will follow the pattern: ``ra2.
.``, e.g. +``ra2.ch.001`` for the first requirement in the Kubernetes Node section. These +specifications will then be used as requirements input for the Kubernetes +Reference Implementation and any vendor or community implementations. + +Figure 4-1 below shows the architectural components that are described in the +subsequent sections of this chapter. + +.. image:: ../figures/ch04_k8s_architecture.png + :alt: "Figure 4-1: Kubernetes Reference Architecture" + + +**Figure 4-1:** Kubernetes Reference Architecture + +Kubernetes Node +--------------- + +This section describes the configuration that will be applied to the physical or +virtual machine and an installed Operating System. In order for a Kubernetes Node +to be conformant with the Reference Architecture it must be implemented as per +the following specifications: + +============== ============================================== ===================================================================================================================================================================================================================================================================================================================== ========================================================================================================================================================================================== ===================================================================================================== +Ref Specification Details Requirement Trace Reference Implementation Trace +============== ============================================== ===================================================================================================================================================================================================================================================================================================================== ========================================================================================================================================================================================== ===================================================================================================== +``ra2.ch.001`` Huge pages When hosting workloads matching the Network Intensive profile, it **must** be possible to enable Huge pages (2048KiB and 1048576KiB) within the Kubernetes Node OS, exposing schedulable resources ``hugepages-2Mi`` and ``hugepages-1Gi``. `infra.com.cfg.004 <./chapter02.md#cloud-infrastructure-software-profile-requirements>`__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ch.002`` SR-IOV capable NICs When hosting workloads matching the Network Intensive profile, the physical machines on which the Kubernetes Nodes run **must** be equipped with NICs that are SR-IOV capable. `e.cap.013 <./chapter02.md#cloud-infrastructure-software-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.003`` SR-IOV Virtual Functions When hosting workloads matching the Network Intensive profile, SR-IOV virtual functions (VFs) **must** be configured within the Kubernetes Node OS, as the SR-IOV Device Plugin does not manage the creation of these VFs. `e.cap.013 <./chapter02.md#cloud-infrastructure-software-profile-requirements>`__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ch.004`` CPU Simultaneous Multi-Threading (SMT) SMT **must** be enabled in the BIOS on the physical machine on which the Kubernetes Node runs. `infra.hw.cpu.cfg.004 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.005`` CPU Allocation Ratio - VMs For Kubernetes nodes running as Virtual Machines, the CPU allocation ratio between vCPU and physical CPU core **must** be 1:1. `infra.com.cfg.001 <./chapter02.md#cloud-infrastructure-software-profile-requirements>`__ +``ra2.ch.006`` CPU Allocation Ratio - Pods To ensure the CPU allocation ratio between vCPU and physical CPU core is 1:1, the sum of CPU requests and limits by containers in Pod specifications **must** remain less than the allocatable quantity of CPU resources (i.e. ``requests.cpu < allocatable.cpu`` and ``limits.cpu < allocatable.cpu``). `infra.com.cfg.001 <./chapter02.md#cloud-infrastructure-software-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.007`` IPv6DualStack To support IPv4/IPv6 dual stack networking, the Kubernetes Node OS **must** support and be allocated routable IPv4 and IPv6 addresses. `req.inf.ntw.04 <./chapter02.md#kubernetes-architecture-requirements>`__ +``ra2.ch.008`` Physical CPU Quantity The physical machines on which the Kubernetes Nodes run **must** be equipped with at least 2 physical sockets, each with at least 20 CPU cores. `infra.hw.cpu.cfg.001 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__, `infra.hw.cpu.cfg.002 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.009`` Physical Storage The physical machines on which the Kubernetes Nodes run **should** be equipped with Sold State Drives (SSDs). `infra.hw.stg.ssd.cfg.002 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.010`` Local Filesystem Storage Quantity The Kubernetes Nodes **must** be equipped with local filesystem capacity of at least 320GB for unpacking and executing containers. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc. `e.cap.003 <./chapter02.md#cloud-infrastructure-software-profile-capabilities>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.011`` Virtual Node CPU Quantity If using VMs, the Kubernetes Nodes **must** be equipped with at least 16 vCPUs. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc. `e.cap.001 <./chapter02.md#cloud-infrastructure-software-profile-capabilities>`__ +``ra2.ch.012`` Kubernetes Node RAM Quantity The Kubernetes Nodes **must** be equipped with at least 32GB of RAM. Note, extra should be provisioned to cater for any overhead required by the Operating System and any required OS processes such as the container runtime, Kubernetes agents, etc. `e.cap.002 <./chapter02.md#cloud-infrastructure-software-profile-capabilities>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.013`` Physical NIC Quantity The physical machines on which the Kubernetes Nodes run **must** be equipped with at least four (4) Network Interface Card (NIC) ports. `infra.hw.nic.cfg.001 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.014`` Physical NIC Speed - Basic Profile The speed of NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Basic Profile **must** be at least 10Gbps. `infra.hw.nic.cfg.002 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.015`` Physical NIC Speed - Network Intensive Profile The speed of NIC ports housed in the physical machines on which the Kubernetes Nodes run for workloads matching the Network Intensive profile **must** be at least 25Gbps. `infra.hw.nic.cfg.002 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__ `3.3 <../../../ref_impl/cntt-ri2/chapters/chapter03.md#infrastructure-requirements>`__ +``ra2.ch.016`` Physical PCIe slots The physical machines on which the Kubernetes Nodes run **must** be equipped with at least eight (8) Gen3.0 PCIe slots, each with at least eight (8) lanes. +``ra2.ch.017`` Immutable infrastructure Whether physical or virtual machines are used, the Kubernetes Node **must not** be changed after it is instantiated. New changes to the Kubernetes Node must be implemented as new Node instances. This covers any changes from BIOS through Operating System to running processes and all associated configurations. `req.gen.cnt.02 <./chapter02.md#kubernetes-architecture-requirements>`__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ch.018`` NFD `Node Feature Discovery `__ **must** be used to advertise the detailed software and hardware capabilities of each node in the Kubernetes Cluster. TBD `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +============== ============================================== ===================================================================================================================================================================================================================================================================================================================== ========================================================================================================================================================================================== ===================================================================================================== + +**Table 4-1:** Node Specifications + +Kubernetes +---------- + +In order for the Kubernetes components to be conformant with the Reference Architecture they must be implemented as per the following specifications: + +=============== ================================== ==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== ====================================================================================================================================================================================================================================================================================================== ===================================================================================================== +Ref Specification Details Requirement Trace Reference Implementation Trace +=============== ================================== ==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== ====================================================================================================================================================================================================================================================================================================== ===================================================================================================== +``ra2.k8s.001`` Kubernetes Conformance The Kubernetes distribution, product, or installer used in the implementation **must** be listed in the `Kubernetes Distributions and Platforms document `__ and marked (X) as conformant for the Kubernetes version defined in `README <../README.md#required-versions-of-most-important-components>`__. `req.gen.cnt.03 <./chapter02.md#kubernetes-architecture-requirements>`__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.k8s.002`` Highly available etcd An implementation **must** consist of either three, five or seven nodes running the etcd service (can be colocated on the master nodes, or can run on separate nodes, but not on worker nodes). `req.gen.rsl.02 req.gen.avl.01 <./chapter02.md#kubernetes-architecture-requirements>`__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.k8s.003`` Highly available control plane An implementation **must** consist of at least one master node per availability zone or fault domain to ensure the high availability and resilience of the Kubernetes control plane services. `req.gen.rsl.02 <./chapter02.md#kubernetes-architecture-requirements>`__, `req.gen.avl.01 <./chapter02.md#kubernetes-architecture-requirements>`__ +``ra2.k8s.012`` Control plane services A master node **must** run at least the following Kubernetes control plane services: ``kube-apiserver``, ``kube-scheduler`` and ``kube-controller-manager``. `req.gen.rsl.02 <./chapter02.md#kubernetes-architecture-requirements>`__, `req.gen.avl.01 <./chapter02.md#kubernetes-architecture-requirements>`__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.k8s.004`` Highly available worker nodes An implementation **must** consist of at least one worker node per availability zone or fault domain to ensure the high availability and resilience of workloads managed by Kubernetes `req.gen.rsl.01 <./chapter02.md#kubernetes-architecture-requirements>`__, `req.gen.avl.01 <./chapter02.md#kubernetes-architecture-requirements>`__, `req.kcm.gen.02 <./chapter02.md#kubernetes-architecture-requirements>`__, `req.inf.com.01 <./chapter02.md#kubernetes-architecture-requirements>`__ +``ra2.k8s.005`` Kubernetes API Version In alignment with the `Kubernetes version support policy `__, an implementation **must** use a Kubernetes version as per the subcomponent versions table in `README <../README.md#required-versions-of-most-important-components>`__. TBC +``ra2.k8s.006`` NUMA Support When hosting workloads matching the Network Intensive profile, the ``TopologyManager`` and ``CPUManager`` feature gates **must** be enabled and configured on the kubelet (note, TopologyManager is enabled by default in Kubernetes v1.18 and later, with CPUManager enabled by default in Kubernetes v1.10 and later). ``--feature-gates="...,TopologyManager=true,CPUManager=true" --topology-manager-policy=single-numa-node --cpu-manager-policy=static`` `e.cap.007 `__ `infra.com.cfg.002 <./chapter02.md#cloud-infrastructure-software-profile-requirements>`__ `infra.hw.cpu.cfg.003 <./chapter02.md#cloud-infrastructure-hardware-profile-requirements>`__ +``ra2.k8s.007`` DevicePlugins Feature Gate When hosting workloads matching the Network Intensive profile, the DevicePlugins feature gate **must** be enabled (note, this is enabled by default in Kubernetes v1.10 or later). ``--feature-gates="...,DevicePlugins=true,..."`` Various, e.g. `e.cap.013 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.k8s.008`` System Resource Reservations To avoid resource starvation issues on nodes, the implementation of the architecture **must** reserve compute resources for system daemons and Kubernetes system daemons such as kubelet, container runtime, etc. Use the following kubelet flags: ``--reserved-cpus=[a-z]``, using two of ``a-z`` to reserve 2 SMT threads. `i.cap.014 `__ +``ra2.k8s.009`` CPU Pinning When hosting workloads matching the Network Intensive profile, in order to support CPU Pinning, the kubelet **must** be started with the ``--cpu-manager-policy=static`` option. (Note, only containers in ``Guaranteed`` pods - where CPU resource ``requests`` and ``limits`` are identical - and configured with positive-integer CPU ``requests`` will take advantage of this. All other Pods will run on CPUs in the remaining shared pool.) `infra.com.cfg.003 <./chapter02.md#cloud-infrastructure-software-profile-requirements>`__ +``ra2.k8s.010`` IPv6DualStack To support IPv6 and IPv4, the ``IPv6DualStack`` feature gate **must** be enabled on various components (requires Kubernetes v1.16 or later). kube-apiserver: ``--feature-gates="IPv6DualStack=true"``. kube-controller-manager: ``--feature-gates="IPv6DualStack=true" --cluster-cidr=, --service-cluster-ip-range=, --node-cidr-mask-size-ipv4 ¦ --node-cidr-mask-size-ipv6`` defaults to /24 for IPv4 and /64 for IPv6. kubelet: ``--feature-gates="IPv6DualStack=true"``. kube-proxy: ``--cluster-cidr=, --feature-gates="IPv6DualStack=true"`` `req.inf.ntw.04 <./chapter02.md#kubernetes-architecture-requirements>`__ +``ra2.k8s.011`` Anuket profile labels To clearly identify which worker nodes are compliant with the different profiles defined by Anuket the worker nodes **must** be labelled according to the following pattern: an ``anuket.io/profile/basic`` label must be set to ``true`` on the worker node if it can fulfil the requirements of the basic profile and an ``anuket.io/profile/network-intensive`` label must be set to ``true`` on the worker node if it can fulfil the requirements of the network intensive profile. The requirements for both profiles can be found in `chapter 2 <./chapter02.md#reference-model-requirements>`__ +``ra2.k8s.012`` Kubernetes APIs Kubernetes `Alpha API `__ are recommended only for testing, therefore all Alpha APIs **must** be disabled. `req.int.api.03 <./chapter02.md#reference-model-requirements>`__ +``ra2.k8s.013`` Kubernetes APIs Backward compatibility of all supported GA APIs of Kubernetes **must** be supported. `req.int.api.04 <./chapter02.md#kubernetes-architecture-requirements>`__ +``ra2.k8s.014`` Security Groups Kubernetes **must** support NetworkPolicy feature. `infra.net.cfg.004 `__ +``ra2.k8s.015`` Publishing Services (ServiceTypes) Kubernetes **must** support LoadBalancer `Publishing Service (ServiceTypes) `__. `req.inf.ntw.15 `__ +``ra2.k8s.016`` Publishing Services (ServiceTypes) Kubernetes **must** support `Ingress `__. `req.inf.ntw.16 `__ +``ra2.k8s.017`` Publishing Services (ServiceTypes) Kubernetes **should** support NodePort `Publishing Service (ServiceTypes) `__. `req.inf.ntw.17 `__ +``ra2.k8s.018`` Publishing Services (ServiceTypes) Kubernetes **should** support ExternalName `Publishing Service (ServiceTypes) `__. `req.inf.ntw.18 `__ +``ra2.k8s.019`` Kubernetes APIs Kubernetes Beta APIs **must** be supported only when a stable GA of the same version doesn't exist. `req.int.api.04 <./chapter02.md#kubernetes-architecture-requirements>`__ +=============== ================================== ==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== ====================================================================================================================================================================================================================================================================================================== ===================================================================================================== + +**Table 4-2:** Kubernetes Specifications + +Container runtimes +------------------ + +=============== ============================================ ======================================================================================================================================================================================================== ====================================================================== ===================================================================================================== +Ref Specification Details Requirement Trace Reference Implementation Trace +=============== ============================================ ======================================================================================================================================================================================================== ====================================================================== ===================================================================================================== +``ra2.crt.001`` Conformance with OCI 1.0 runtime spec The container runtime **must** be implemented as per the `OCI 1.0 `__ (Open Container Initiative 1.0) specification. `req.gen.ost.01 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.crt.002`` Kubernetes Container Runtime Interface (CRI) The Kubernetes container runtime **must** be implemented as per the `Kubernetes Container Runtime Interface (CRI) `__ `req.gen.ost.01 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +=============== ============================================ ======================================================================================================================================================================================================== ====================================================================== ===================================================================================================== + +**Table 4-3:** Container Runtime Specifications + +Networking solutions +-------------------- + +In order for the networking solution(s) to be conformant with the Reference +Architecture they must be implemented as per the following specifications: + +=============== ======================================================= ========================================================================================================================================================================================================================================================================== ================================================================================================================================================================ ===================================================================================================== +Ref Specification Details Requirement Trace Reference Implementation Trace +=============== ======================================================= ========================================================================================================================================================================================================================================================================== ================================================================================================================================================================ ===================================================================================================== +``ra2.ntw.001`` Centralised network administration The networking solution deployed within the implementation **must** be administered through the Kubernetes API using native Kubernetes API resources and objects, or Custom Resources. `req.inf.ntw.03 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.002`` Default Pod Network - CNI The networking solution deployed within the implementation **must** use a CNI-conformant Network Plugin for the Default Pod Network, as the alternative (kubenet) does not support cross-node networking or Network Policies. `req.gen.ost.01 `__, `req.inf.ntw.08 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.003`` Multiple connection points The networking solution deployed within the implementation **must** support the capability to connect at least FIVE connection points to each Pod, which are additional to the default connection point managed by the default Pod network CNI plugin. `e.cap.004 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.004`` Multiple connection points presentation The networking solution deployed within the implementation **must** ensure that all additional non-default connection points are requested by Pods using standard Kubernetes resource scheduling mechanisms such as annotations or container resource requests and limits. `req.inf.ntw.03 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.005`` Multiplexer/meta-plugin The networking solution deployed within the implementation **may** use a multiplexer/meta-plugin. `req.inf.ntw.06 `__, `req.inf.ntw.07 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.006`` Multiplexer/meta-plugin CNI Conformance If used, the selected multiplexer/meta-plugin **must** integrate with the Kubernetes control plane via CNI. `req.gen.ost.01 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.007`` Multiplexer/meta-plugin CNI Plugins If used, the selected multiplexer/meta-plugin **must** support the use of multiple CNI-conformant Network Plugins. `req.gen.ost.01 `__, `req.inf.ntw.06 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.008`` SR-IOV Device Plugin for Network Intensive When hosting workloads that match the Network Intensive profile and require SR-IOV acceleration, a Device Plugin for SR-IOV **must** be used to configure the SR-IOV devices and advertise them to the ``kubelet``. `e.cap.013 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.009`` Multiple connection points with multiplexer/meta-plugin When a multiplexer/meta-plugin is used, the additional non-default connection points **must** be managed by a CNI-conformant Network Plugin. `req.gen.ost.01 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.010`` User plane networking When hosting workloads matching the Network Intensive profile, CNI network plugins that support the use of DPDK, VPP, and/or SR-IOV **must** be deployed as part of the networking solution. `infra.net.acc.cfg.001 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.011`` NATless connectivity When hosting workloads that require source and destination IP addresses to be preserved in the traffic headers, a NATless CNI plugin that exposes the pod IP directly to the external networks (e.g. Calico, MACVLAN or IPVLAN CNI plugins) **must** be used. `req.inf.ntw.14 `__ +``ra2.ntw.012`` Device Plugins When hosting workloads matching the Network Intensive profile that require the use of FPGA, SR-IOV or other Acceleration Hardware, a Device Plugin for that FPGA or Acceleration Hardware **must** be used. `e.cap.016 `__, `e.cap.013 `__ `4.3.1 <../../../ref_impl/cntt-ri2/chapters/chapter04.md#installation-on-bare-metal-infratructure>`__ +``ra2.ntw.013`` Dual stack CNI The networking solution deployed within the implementation **must** use a CNI-conformant Network Plugin that is able to support dual-stack IPv4/IPv6 networking. `req.inf.ntw.04 `__ +``ra2.ntw.014`` Security Groups The networking solution deployed within the implementation **must** support network policies. `infra.net.cfg.004 `__ +``ra2.ntw.015`` IPAM plugin for multiplexer When a multiplexer/meta-plugin is used, a CNI-conformant IPAM Network Plugin **must** be installed to allocate IP addresses for secondary network interfaces across all nodes of the cluster. `req.inf.ntw.10 `__ +=============== ======================================================= ========================================================================================================================================================================================================================================================================== ================================================================================================================================================================ ===================================================================================================== + +**Table 4-4:** Networking Solution Specifications + +Storage components +------------------ + +In order for the storage solutions to be conformant with the Reference +Architecture they must be implemented as per the following specifications: + +=============== ================================= ============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== ====================================================================== ============================== +Ref Specification Details Requirement Trace Reference Implementation Trace +=============== ================================= ============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== ====================================================================== ============================== +``ra2.stg.001`` Ephemeral Storage An implementation must support ephemeral storage, for the unpacked container images to be stored and executed from, as a directory in the filesystem on the worker node on which the container is running. See the `Container runtimes <#container-runtimes>`__ section above for more information on how this meets the requirement for ephemeral storage for containers. +``ra2.stg.002`` Kubernetes Volumes An implementation may attach additional storage to containers using Kubernetes Volumes. +``ra2.stg.003`` Kubernetes Volumes An implementation may use Volume Plugins (see ``ra2.stg.005`` below) to allow the use of a storage protocol (e.g., iSCSI, NFS) or management API (e.g., Cinder, EBS) for the attaching and mounting of storage into a Pod. +``ra2.stg.004`` Persistent Volumes An implementation may support Kubernetes Persistent Volumes (PV) to provide persistent storage for Pods. Persistent Volumes exist independent of the lifecycle of containers and/or pods. `req.inf.stg.01 `__ +``ra2.stg.005`` Storage Volume Types An implementation must support the following Volume types: ``emptyDir``, ``ConfigMap``, ``Secret`` and ``PersistentVolumeClaim``. Other Volume plugins may be supported to allow for the use of a range of backend storage systems. +``ra2.stg.006`` Container Storage Interface (CSI) An implementation may support the Container Storage Interface (CSI), an Out-of-tree plugin. In order to support CSI, the feature gates ``CSIDriverRegistry`` and ``CSINodeInfo`` must be enabled. The implementation must use a CSI driver (a full list of CSI drivers can be found `here `__). An implementation may support ephemeral storage through a CSI-compatible volume plugin in which case the ``CSIInlineVolume`` feature gate must be enabled. An implementation may support Persistent Volumes through a CSI-compatible volume plugin in which case the ``CSIPersistentVolume`` feature gate must be enabled. +``ra2.stg.007`` An implementation should use Kubernetes Storage Classes to support automation and the separation of concerns between providers of a service and consumers of the service. +=============== ================================= ============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================== ====================================================================== ============================== + +**Table 4-6:** Storage Solution Specifications + +A note on object storage: + +- This Reference Architecture does not include any specifications for object + storage, as this is neither a native Kubernetes object, nor something that is + required by CSI drivers. Object storage is an application-level requirement + that would ordinarily be provided by a highly scalable service offering rather + than being something an individual Kubernetes cluster could offer. + +.. + + Todo: specifications/commentary to support req.inf.stg.04 (SDS) and req.inf.stg.05 (high performance and horizontally scalable storage). Also req.sec.gen.06 (storage resource isolation), req.sec.gen.10 (CIS - if applicable) and req.sec.zon.03 (data encryption at rest). + +Service meshes +-------------- + +Application service meshes are not in scope for the architecture. The service mesh is a dedicated infrastructure layer for handling service-to-service communication, and it is recommended to secure service-to-service communications within a cluster and to reduce the attack surface. The benefits of the service mesh framework are described in `5.4.3 <./chapter05.md#use-transport-layer-security-and-service-mesh>`__. In addition to securing communications, the use of a service mesh extends Kubernetes capabilities regarding observability and reliability. + +Network service mesh specifications are handled in section `4.5 Networking solutions <#networking-solutions>`__. + +Kubernetes Application package manager +-------------------------------------- + +In order for the application package managers to be conformant with the Reference +Architecture they must be implemented as per the following specifications: + +=============== ============================ ========================================================================================================================================= ======================================================================== ============================== +Ref Specification Details Requirement Trace Reference Implementation Trace +=============== ============================ ========================================================================================================================================= ======================================================================== ============================== +``ra2.pkg.001`` API-based package management A package manager must use the Kubernetes APIs to manage application artifacts. Cluster-side components such as Tiller are not supported. `req.int.api.02 <./chapter02.md#kubernetes-architecture-requirements>`__ +=============== ============================ ========================================================================================================================================= ======================================================================== ============================== + +**Table 4-7:** Kubernetes Application Package Manager Specifications + +Kubernetes workloads +-------------------- + +In order for the Kubernetes workloads to be conformant with the Reference +Architecture they must be implemented as per the following specifications: + +=============== ======================================================================================================================= =================================================================================================================================================================================================================================================================================================== ======================================================================= ============================== +Ref Specification Details Requirement Trace Reference Implementation Trace +=============== ======================================================================================================================= =================================================================================================================================================================================================================================================================================================== ======================================================================= ============================== +``ra2.app.001`` `Root `__ Parameter Group (OCI Spec) Specifies the container's root filesystem. TBD N/A +``ra2.app.002`` `Mounts `__ Parameter Group (OCI Spec) Specifies additional mounts beyond root. TBD N/A +``ra2.app.003`` `Process `__ Parameter Group (OCI Spec) Specifies the container process. TBD N/A +``ra2.app.004`` `Hostname `__ Parameter Group (OCI Spec) Specifies the container's hostname as seen by processes running inside the container. TBD N/A +``ra2.app.005`` `User `__ Parameter Group (OCI Spec) User for the process is a platform-specific structure that allows specific control over which user the process runs as. TBD N/A +``ra2.app.006`` Consumption of additional, non-default connection points The workload must request additional non-default connection points through the use of workload annotations or resource requests and limits within the container spec passed to the Kubernetes API Server. `req.int.api.01 `__ N/A +``ra2.app.007`` Host Volumes Workloads should not use ``hostPath`` volumes, as `Pods with identical configuration `__ (such as those created from a PodTemplate) may behave differently on different nodes due to different files on the nodes. `req.kcm.gen.02 `__. N/A +``ra2.app.008`` Infrastructure dependency Workloads must not rely on the availability of the master nodes for the successful execution of their functionality (i.e. loss of the master nodes may affect non-functional behaviours such as healing and scaling, but components that are already running will continue to do so without issue). TBD N/A +``ra2.app.009`` Device plugins Workload descriptors must use the resources advertised by the device plugins to indicate their need for an FPGA, SR-IOV or other acceleration device. TBD N/A +``ra2.app.010`` Node Feature Discovery (NFD) Workload descriptors must use the labels advertised by `Node Feature Discovery `__ to indicate which node software of hardware features they need. TBD N/A +=============== ======================================================================================================================= =================================================================================================================================================================================================================================================================================================== ======================================================================= ============================== + +**Table 4-8:** Kubernetes Workload Specifications + +Additional required components +------------------------------ + + This chapter should list any additional components needed to provide the services defined in Chapter 3.2 (e.g., Prometheus) + diff --git a/doc/ref_arch/kubernetes/chapters/chapter05.md b/doc/ref_arch/kubernetes/chapters/chapter05.md index e9d15d07e1..2e73664aa2 100644 --- a/doc/ref_arch/kubernetes/chapters/chapter05.md +++ b/doc/ref_arch/kubernetes/chapters/chapter05.md @@ -1,130 +1,95 @@ -[<< Back](../../kubernetes) - -# 5. Security Guidance -

scope

- -## Table of Contents - - [5.1 Introduction](#51-introduction) - - [5.1.1 Security Perimeters](#511--security-perimeters) - - [5.2 Principles](#52-principles) - - [5.3 Node Hardening](#53-node-hardening) - - [5.3.1 Node hardening: Securing Kubernetes - hosts](#531-node-hardening-securing-kubernetes-hosts) - - [5.3.2 Restrict direct access to - nodes](#532-restrict-direct-access-to-nodes) - - [5.3.3 Vulnerability assessment](#533-vulnerability-assessment) - - [5.3.4 Patch management](#534-patch-management) - - [5.4 Securing Kubernetes orchestrator](#54-securing-kubernetes-orchestrator) - - [5.4.1 Control network access to sensitive - ports](#541-control-network-access-to-sensitive-ports) - - [5.4.2 Controlling access to the Kubernetes - API](#542-controlling-access-to-the-kubernetes-api) - - [5.4.3 Use Transport Layer Security and Service Mesh](#543-use-transport-layer-security-and-service-mesh) - - [5.4.4 API Authentication, API - Authorisation](#544-api-authentication-api-authorisation) - - [5.4.5 Restrict access to etcd and encrypt contents within - etcd](#545-restrict-access-to-etcd-and-encrypt-contents-within-etcd) - - [5.4.6 Controlling access to the - Kubelet](#546-controlling-access-to-the-kubelet) - - [5.4.7 Securing Kubernetes Dashboard](#547-securing-kubernetes-dashboard) - - [5.5 Use Namespaces to Establish Security - Boundaries](#55-use-namespaces-to-establish-security-boundaries) - - [5.6 Separate Sensitive Workload](#56-separate-sensitive-workload) - - [5.7 Create and Define Network - Policies](#57-create-and-define-network-policies) - - [5.8 Run latest Version](#58-run-latest-version) - - [5.9 Secure Platform Metadata](#59-secure-platform-metadata) - - [5.10 Enable Logging and Monitoring](#510--enable-logging-and-monitoring) - - [5.11 Run-Time Security](#511--run-time-security) - - [5.12 Secrets Management](#512--secrets-management) - - [5.13 Trusted Registry](#513--trusted-registry) - - [5.14 Isolation](#514--isolation) - - [5.14.1 VM vs. Container Isolation](#5141-vm-vs-container-isolation) - - [5.14.2 Container Isolation in Kubernetes - Cluster](#5142-container-isolation-in-kubernetes-cluster) - - [5.14.2.1 Namespaces](#51421-namespaces) - -## 5.1 Introduction + +# Security Guidance + +## Introduction Securing Kubernetes requires several layers of security features to provide end to end security for cloud native applications. It is recommended that: -- Security testing is fully integrated into the CI/CD pipelines of all parties +* Security testing is fully integrated into the CI/CD pipelines of all parties (e.g., vendors and operators). -- Automated security policies are used to flag builds with issues. -- Image registries are monitored to automatically block or replace images with +* Automated security policies are used to flag builds with issues. +* Image registries are monitored to automatically block or replace images with known vulnerabilities, while also ensuring policies are used to gate what can be deployed and who can deploy from the registry. -- Adopt a layered packaging model which supports separation of concerns during +* Adopt a layered packaging model which supports separation of concerns during image build. The following functionalities are recommended for securing Kubernetes platforms: -- Image Certification (Scan for vulnerabilities) and Signing -- Role-base Access Control -- Secret Management -- How to overcome the lack of hard Kubernetes Cluster Multi-tenancy - - Tenants without hard multi-tenancy requirements (multiple development teams +* Image Certification (Scan for vulnerabilities) and Signing +* Role-base Access Control +* Secret Management +* How to overcome the lack of hard Kubernetes Cluster Multi-tenancy + + * Tenants without hard multi-tenancy requirements (multiple development teams in the same organisation) separated from each other by Namespaces - - For strict multi tenancy, a dedicated Kubernetes Cluster per tenant should + * For strict multi tenancy, a dedicated Kubernetes Cluster per tenant should be used -- Integration with other security ecosystem like monitoring and alerting tools -### 5.1.1 Security Perimeters +* Integration with other security ecosystem like monitoring and alerting tools + +### Security Perimeters + When applications or workloads run on Kubernetes, there are several layers which come into picture that govern the security. Each of these layers needs to be secured within their perimeters. The various layers that come into picture are: -- **Container Registry**: A container registry is a repository to manage +* **Container Registry**: A container registry is a repository to manage container **images. The access to container registry needs to be secured in order to **prevent unauthorised access or image tampering. -- **Container Images**: Stored instance of a container that holds a set of +* **Container Images**: Stored instance of a container that holds a set of software needed to run an application. Before loading them to container registry, they need to be secured by performing various checks like vulnerability analysis, scans etc. These images should also be signed from trusted sources. -- **Containers**: A lightweight and portable executable image that contains +* **Containers**: A lightweight and portable executable image that contains software and all of its dependencies. The containers need to be prevented from accessing the underlying OS like loading of kernel modules, mounting of directories of underlying OS etc and ensuring that they don't run in privileged mode. -- **Pods**: A Pod represents a set of running containers on your Cluster. +* **Pods**: A Pod represents a set of running containers on your Cluster. Kubernetes inherently offers pod security policies that define a set of conditions that a pod needs to run with in order to be accepted into the system. These policies help in ensuring the necessary checks for running the pods. -- **Kubernetes Node**: A Kubernetes node in an unsecured boundary can lead to a +* **Kubernetes Node**: A Kubernetes node in an unsecured boundary can lead to a potential threat to the running workloads. Such a node should be hardened by disabling unused ports, prohibiting root access etc. -- **Kubernetes Master**: A master node in an unsecured boundary can lead to a +* **Kubernetes Master**: A master node in an unsecured boundary can lead to a potential threat to the running workloads. A master may be hardened in terms of security by disabling unused ports, prohibiting root access etc. -- **Kubernetes Control Plane**: The container orchestration layer that exposes +* **Kubernetes Control Plane**: The container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers. The communication over these APIs needs to be secured via different mechanisms like TLS encryption, API authentication via LDAP etc. -## 5.2 Principles +## Principles + The following are core principles to consider when securing cloud native applications: -- Deploy only secure applications and trusted codes -- Only deploy applications from validated and verified images -- Only deploy applications from trusted registries -- Containers orchestration (Kubernetes) secured with administrative boundaries +* Deploy only secure applications and trusted codes +* Only deploy applications from validated and verified images +* Only deploy applications from trusted registries +* Containers orchestration (Kubernetes) secured with administrative boundaries between tenants - - Use Namespaces to establish security boundaries between tenants - - Create and define Cluster network policies - - Run a Cluster-wide pod security policy - - Turn on Audit Logging - - Separate sensitive workloads using Namespaces - - Secure tenant metadata Access -- Segregate container networks using security zoning and network standards -- Harden the Host OS running the containers -- Use container-aware runtime defence tools -- Enable Role-Based Access Control (RBAC) - -## 5.3 Node Hardening -### 5.3.1 Node hardening: Securing Kubernetes hosts + + * Use Namespaces to establish security boundaries between tenants + * Create and define Cluster network policies + * Run a Cluster-wide pod security policy + * Turn on Audit Logging + * Separate sensitive workloads using Namespaces + * Secure tenant metadata Access + +* Segregate container networks using security zoning and network standards +* Harden the Host OS running the containers +* Use container-aware runtime defence tools +* Enable Role-Based Access Control (RBAC) + +## Node Hardening + +### Node hardening: Securing Kubernetes hosts + When an operating system or application is installed, it comes with default settings. Usually, all ports are open, and all application services are turned on. In other words, freshly installed assets are highly insecure. @@ -134,25 +99,28 @@ well known security frameworks. Security benchmarks, for example, CIS benchmarks are a set of configuration standards and best practices designed to help ‘harden’ the security of their digital assets. -### 5.3.2 Restrict direct access to nodes +### Restrict direct access to nodes + Restrict root/administrative access to Kubernetes nodes while avoiding direct access to nodes for operational activities including debugging, troubleshooting, and other tasks. -### 5.3.3 Vulnerability assessment +### Vulnerability assessment + Vulnerability assessments are a crucial part of IT risk management lifecycles. The mitigation of the vulnerabilities helps in protecting systems and data from unauthorised access and breaches. Implement necessary vulnerability scanner tools (e.g., OpenVAS and many other opensource and commercial tools) to identify threats and flaws within the infrastructure that represents potential risks. -### 5.3.4 Patch management +### Patch management + Patch management is another key aspect of IT risk management lifecycle for security, compliance, uptime, feature enhancements, etc. Implement necessary patch management to ensure your environment is not susceptible to exploitation. -## 5.4 Securing Kubernetes orchestrator +## Securing Kubernetes orchestrator The communication over the Kubernetes control plane APIs needs to be secured via different mechanisms like TLS encryption, API authentication via @@ -163,34 +131,37 @@ etc. They following are security recommendations for orchestration manager: -- Cluster management Network isolation can help protect the master node and +* Cluster management Network isolation can help protect the master node and control where administrative commands can run. Use network isolation techniques, configure RBAC on the Cluster manager and configure node service accounts following the principle of least privilege. -- Ensure that access control is applied to registries requiring unique +* Ensure that access control is applied to registries requiring unique credentials, to limit who can control the build or add images. -- Network access runs over TLS connections. -- User roles and access levels are configured to provide segregation of duties. - - Do not mix container and non-containers services on the same node - - Do not run containers as root -- Multi-factor authentication is used for all administrative access. -- Harden the configuration by using CIS (Center for Internet Security) +* Network access runs over TLS connections. +* User roles and access levels are configured to provide segregation of duties. + + * Do not mix container and non-containers services on the same node + * Do not run containers as root + +* Multi-factor authentication is used for all administrative access. +* Harden the configuration by using CIS (Center for Internet Security) benchmarks, which are available for container runtime and Kubernetes -- Deploy security products that provide whitelisting, behaviour monitoring and +* Deploy security products that provide whitelisting, behaviour monitoring and anomaly detection for preventing malicious activity -- Avoid privileged container application through policy management to reduce the +* Avoid privileged container application through policy management to reduce the effects of potential attacks. -- Enable integration with other security ecosystem (SIEM) -- Isolate environments (Dev /test /Production) from other environments within +* Enable integration with other security ecosystem (SIEM) +* Isolate environments (Dev /test /Production) from other environments within the Cluster. -- Create administrative boundaries between resources using Namespace and avoid +* Create administrative boundaries between resources using Namespace and avoid using default Namespaces. -- Enable Seccomp to ensure that the workloads have restricted actions available +* Enable Seccomp to ensure that the workloads have restricted actions available within the container application. -- Limit discovery by restricting services and users that can access Cluster +* Limit discovery by restricting services and users that can access Cluster management metadata on configuration, containers and nodes -### 5.4.1 Control network access to sensitive ports +### Control network access to sensitive ports + Kubernetes clusters usually listen on a range of well-defined and distinctive ports which makes it easy to identify the clusters and attack them. Hence, it is highly recommended to configure authentication and authorisation on the cluster @@ -217,11 +188,13 @@ API server except from trusted networks. | TCP | 10250 | Kubelet API | | TCP | 30000-32767 | NodePort Services | -### 5.4.2 Controlling access to the Kubernetes API +### Controlling access to the Kubernetes API + The Kubernetes platform is controlled using APIs, which are the first items to be secured in order to defend against attackers. Controlling who has access and what actions they are allowed to perform is the primary concern. -### 5.4.3 Use Transport Layer Security and Service Mesh +### Use Transport Layer Security and Service Mesh + Communication in the cluster between services should be handled using TLS, encrypting all traffic by default. Kubernetes expects that all API communication in the cluster is encrypted by default with TLS, and the majority of installation methods @@ -241,9 +214,11 @@ The two documents, [NIST SP 800-204A](https://nvlpubs.nist.gov/nistpubs/SpecialP [NIST SP 800-204B]( https://csrc.nist.gov/publications/detail/sp/800-204b/final) (Attribute-based Access Control for Microservices-based Applications Using a Service Mesh) provide guidance to deploy service mesh. -### 5.4.4 API Authentication, API Authorisation +### API Authentication, API Authorisation + Secure all connections to a Kubernetes Cluster. Adopt the following security authentication mechanisms: + * Configure user roles and access levels to provide segregation of duties (RBAC) * Use multi-factor authentication for all administrative access * Use token-based or certificate-based service and session authentication @@ -251,7 +226,8 @@ authentication mechanisms: * Integrated with existing identity management platforms e.g., SAML, AD, etc. for access control -### 5.4.5 Restrict access to etcd and encrypt contents within etcd +### Restrict access to etcd and encrypt contents within etcd + etcd is a critical Kubernetes component which stores information on state and secrets, and it should be protected differently from the rest of your cluster. Write access to the API server's etcd is equivalent to gaining root on the @@ -269,12 +245,14 @@ their etcd server, such as mutual auth via TLS client certificates, and it is often recommended to isolate the etcd servers behind a firewall that only the API servers may access. -### 5.4.6 Controlling access to the Kubelet +### Controlling access to the Kubelet + Kubelets expose HTTPS endpoints which grant powerful control over the node and containers. By default Kubelets allow unauthenticated access to this API. Production clusters should enable Kubelet authentication and authorization -### 5.4.7 Securing Kubernetes Dashboard +### Securing Kubernetes Dashboard + The Kubernetes dashboard is a webapp for managing your cluster. It is not a part of the Kubernetes cluster itself, it has to be installed by the owners of the cluster; a number of tutorials show how to do this. @@ -304,19 +282,22 @@ To prevent attacks via the dashboard, you should follow some best practices: user's credentials instead of using a privileged ServiceAccount. This method can be used on both on-prem and managed cloud clusters. -## 5.5 Use Namespaces to Establish Security Boundaries +## Use Namespaces to Establish Security Boundaries + Namespaces in Kubernetes is the first level of isolation between components. It is easier to apply security controls (Network Policies, Pod policies, etc.) to different types of workloads when deployed in separate Namespaces. -## 5.6 Separate Sensitive Workload +## Separate Sensitive Workload + To limit the potential impact of a compromise, it is recommended to run sensitive workloads on a dedicated set of nodes. This approach reduces the risk of a sensitive application being accessed through a less-secure application that shares a container runtime or host. - The separation can be achieved by using node pools and Kubernetes Namespaces. -## 5.7 Create and Define Network Policies +## Create and Define Network Policies + Network Policies allow Kubernetes managers to control network access into and out of the cloud native applications. It is recommended to have a well defined ingress and egress policy for cloud native applications. It is also important to @@ -324,42 +305,48 @@ modify the default network policies, such as blocking or allowing traffic from other Namespaces or Clusters while ensuring the Namespaces/Clusters are running with policy support enabled. -## 5.8 Run latest Version +## Run latest Version + As new security features and patches are added in every quarterly update, it is important to take advantage of these fixes and patches. -- It is recommended to run the latest release with its most recent patches. -## 5.9 Secure Platform Metadata +* It is recommended to run the latest release with its most recent patches. + +## Secure Platform Metadata + Kubernetes metadata contain sensitive information including kubelet admin credentials. It is recommended to secure them using encryption to avoid this being stolen and use to for escalated privileges in the the Cluster. -- Limit discovery by restricting services and users that can access Cluster +* Limit discovery by restricting services and users that can access Cluster management metadata on configuration, container application, and nodes -- Ensure all metadata is encrypted and network access runs over TLS connections +* Ensure all metadata is encrypted and network access runs over TLS connections + +## Enable Logging and Monitoring -## 5.10 Enable Logging and Monitoring Logging, monitoring, alerting and log aggregation are essential for Kubernetes. Enable and monitor audit logs for anomalous or unwanted API calls, especially any authorisation failure. -## 5.11 Run-Time Security +## Run-Time Security + The following are recommended best practices for container run-time: -- Integrate run-time processes to Security Information and Event Monitoring +* Integrate run-time processes to Security Information and Event Monitoring (SIEM) -- Use container-aware run-time defence tools -- Ensure all running cloud native applications are from secure and verified +* Use container-aware run-time defence tools +* Ensure all running cloud native applications are from secure and verified images -- Cloud native applications are not run with root privileges -- Ensure sensitive workloads are properly segmented by Namespaces or Cluster to +* Cloud native applications are not run with root privileges +* Ensure sensitive workloads are properly segmented by Namespaces or Cluster to mitigate the scope of compromise. -## 5.12 Secrets Management +## Secrets Management + It is recommended that the principle of least privilege is applied to secret management in Kubernetes: -- Ensure that the cloud native applications can only read the secrets that these +* Ensure that the cloud native applications can only read the secrets that these applications need -- Have different set of secrets for different environments(like production, +* Have different set of secrets for different environments(like production, development, and testing) Secret values protect sensitive data, it is recommended to protect them from @@ -374,43 +361,48 @@ key manager or vault. Only pull credentials on demand and over secure channels to ensure sensitive data is not written to disk unprotected. The key manager or vault encryption keys should be backed by a FIPS 140-2 Hardware Security Module. It is also important to implement the following: -- Check there are no hard-coded passwords, keys, and other sensitive items in + +* Check there are no hard-coded passwords, keys, and other sensitive items in the container application. -- Where possible use security tools to automate scanning for hard-coded +* Where possible use security tools to automate scanning for hard-coded passwords, keys, and other sensitive items in the container application -## 5.13 Trusted Registry +## Trusted Registry + Ensure that the container registry only accepts container images from trusted sources that have tested and validated the images. Where images are provided by third parties, define and follow a formal process to validate compliance with security requirements. Also ensure that access control is applied to registries requiring unique credentials, to limit who can control the build or add images. -- It is strongly recommended that network access to the registry is secured +* It is strongly recommended that network access to the registry is secured using TLS, SSL or VPN connections to ensure trust. -- Ensure container applications are validated to assess their use and +* Ensure container applications are validated to assess their use and applicability as well as scanned for viruses and vulnerabilities. Only deploy container application from images that are signed with a trusted key -- Ensure the latest certified container application is always selected by +* Ensure the latest certified container application is always selected by versioning images -- Trusted repository and registry services should reject containers that are not +* Trusted repository and registry services should reject containers that are not properly signed -- Use approved registries for images loaded into production -- Where possible, use third-party products to validate container content both +* Use approved registries for images loaded into production +* Where possible, use third-party products to validate container content both before and after deployment Ensure stale images are removed from the registry. Remove unsafe, vulnerable images (e.g. containers should no longer be used based on time triggers and labels associated with images). -## 5.14 Isolation -### 5.14.1 VM vs. Container Isolation +## Isolation + +### VM vs. Container Isolation + Sometimes container isolation is compared directly with VM based isolation, with the conclusion "*there are issues with container isolation, it is not as good as VM isolation*". Such 1:1 comparison is not reasonable because VM and container based isolation are fundamentally different: -- VMs: hard isolation, in the layers underlying the application SW -- Containers: isolation by SW based mechanisms available in OS, Docker and + +* VMs: hard isolation, in the layers underlying the application SW +* Containers: isolation by SW based mechanisms available in OS, Docker and Kubernetes. A container workload is just a set of Linux processes. It is _possible_ to configure SW based _additional isolation_ for container workloads, for example by kernel namespaces. @@ -421,8 +413,10 @@ should not be deployed together in the same Kubernetes Cluster - unless these applications have been planned and verified to co-exist. Thus, the default is to allocate one Namespace per Cloud Native Network Function (CNF). -### 5.14.2 Container Isolation in Kubernetes Cluster -#### 5.14.2.1 Namespaces +### Container Isolation in Kubernetes Cluster + +#### Namespaces + Kubernetes Namespaces should be used to provide resource isolation within a Kubernetes Cluster. They should not be used to isolate different steps in the deployment process like Development, Production, or Testing. The most reliable diff --git a/doc/ref_arch/kubernetes/chapters/chapter05.rst b/doc/ref_arch/kubernetes/chapters/chapter05.rst new file mode 100644 index 0000000000..a5ca28a970 --- /dev/null +++ b/doc/ref_arch/kubernetes/chapters/chapter05.rst @@ -0,0 +1,481 @@ +Security Guidance +================= + +Introduction +------------ + +Securing Kubernetes requires several layers of security features to provide end +to end security for cloud native applications. It is recommended that: + +- Security testing is fully integrated into the CI/CD pipelines of all parties + (e.g., vendors and operators). +- Automated security policies are used to flag builds with issues. +- Image registries are monitored to automatically block or replace images with + known vulnerabilities, while also ensuring policies are used to gate what can + be deployed and who can deploy from the registry. +- Adopt a layered packaging model which supports separation of concerns during + image build. + +The following functionalities are recommended for securing Kubernetes platforms: + +- Image Certification (Scan for vulnerabilities) and Signing + +- Role-base Access Control + +- Secret Management + +- How to overcome the lack of hard Kubernetes Cluster Multi-tenancy + + - Tenants without hard multi-tenancy requirements (multiple development teams + in the same organisation) separated from each other by Namespaces + - For strict multi tenancy, a dedicated Kubernetes Cluster per tenant should + be used + +- Integration with other security ecosystem like monitoring and alerting tools + +Security Perimeters +~~~~~~~~~~~~~~~~~~~ + +When applications or workloads run on Kubernetes, there are several layers which +come into picture that govern the security. Each of these layers needs to be +secured within their perimeters. The various layers that come into picture are: + +- **Container Registry**: A container registry is a repository to manage + container \**images. The access to container registry needs to be secured in + order to \**prevent unauthorised access or image tampering. +- **Container Images**: Stored instance of a container that holds a set of + software needed to run an application. Before loading them to container + registry, they need to be secured by performing various checks like + vulnerability analysis, scans etc. These images should also be signed from + trusted sources. +- **Containers**: A lightweight and portable executable image that contains + software and all of its dependencies. The containers need to be prevented from + accessing the underlying OS like loading of kernel modules, mounting of + directories of underlying OS etc and ensuring that they don't run in + privileged mode. +- **Pods**: A Pod represents a set of running containers on your Cluster. + Kubernetes inherently offers pod security policies that define a set of + conditions that a pod needs to run with in order to be accepted into the + system. These policies help in ensuring the necessary checks for running the + pods. +- **Kubernetes Node**: A Kubernetes node in an unsecured boundary can lead to a + potential threat to the running workloads. Such a node should be hardened by + disabling unused ports, prohibiting root access etc. +- **Kubernetes Master**: A master node in an unsecured boundary can lead to a + potential threat to the running workloads. A master may be hardened in terms + of security by disabling unused ports, prohibiting root access etc. +- **Kubernetes Control Plane**: The container orchestration layer that exposes + the API and interfaces to define, deploy, and manage the lifecycle of + containers. The communication over these APIs needs to be secured via + different mechanisms like TLS encryption, API authentication via LDAP etc. + +Principles +---------- + +The following are core principles to consider when securing cloud native +applications: + +- Deploy only secure applications and trusted codes + +- Only deploy applications from validated and verified images + +- Only deploy applications from trusted registries + +- Containers orchestration (Kubernetes) secured with administrative boundaries + between tenants + + - Use Namespaces to establish security boundaries between tenants + - Create and define Cluster network policies + - Run a Cluster-wide pod security policy + - Turn on Audit Logging + - Separate sensitive workloads using Namespaces + - Secure tenant metadata Access + +- Segregate container networks using security zoning and network standards + +- Harden the Host OS running the containers + +- Use container-aware runtime defence tools + +- Enable Role-Based Access Control (RBAC) + +Node Hardening +-------------- + +Node hardening: Securing Kubernetes hosts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When an operating system or application is installed, it comes with default +settings. Usually, all ports are open, and all application services are turned +on. In other words, freshly installed assets are highly insecure. + +Ensure Kubernetes nodes are secure, hardened and configured correctly following +well known security frameworks. Security benchmarks, for example, CIS benchmarks +are a set of configuration standards and best practices designed to help ‘harden’ +the security of their digital assets. + +Restrict direct access to nodes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Restrict root/administrative access to Kubernetes nodes while avoiding direct +access to nodes for operational activities including debugging, troubleshooting, +and other tasks. + +Vulnerability assessment +~~~~~~~~~~~~~~~~~~~~~~~~ + +Vulnerability assessments are a crucial part of IT risk management lifecycles. +The mitigation of the vulnerabilities helps in protecting systems and data from unauthorised access and breaches. +Implement necessary vulnerability scanner tools (e.g., OpenVAS and many other +opensource and commercial tools) to identify threats and flaws within the +infrastructure that represents potential risks. + +Patch management +~~~~~~~~~~~~~~~~ + +Patch management is another key aspect of IT risk management lifecycle for +security, compliance, uptime, feature enhancements, etc. Implement +necessary patch management to ensure your environment is not susceptible to +exploitation. + +Securing Kubernetes orchestrator +-------------------------------- + +The communication over the Kubernetes control plane APIs needs to be +secured via different mechanisms like TLS encryption, API authentication via +LDAP etc. A control plane node in an unsecured boundary can lead to a potential +threat to the running workloads. It is recommended that a control plane node is +hardened in terms of security by disabling unused ports, prohibiting root access +etc. + +They following are security recommendations for orchestration manager: + +- Cluster management Network isolation can help protect the master node and + control where administrative commands can run. Use network isolation + techniques, configure RBAC on the Cluster manager and configure node service + accounts following the principle of least privilege. + +- Ensure that access control is applied to registries requiring unique + credentials, to limit who can control the build or add images. + +- Network access runs over TLS connections. + +- User roles and access levels are configured to provide segregation of duties. + + - Do not mix container and non-containers services on the same node + - Do not run containers as root + +- Multi-factor authentication is used for all administrative access. + +- Harden the configuration by using CIS (Center for Internet Security) + benchmarks, which are available for container runtime and Kubernetes + +- Deploy security products that provide whitelisting, behaviour monitoring and + anomaly detection for preventing malicious activity + +- Avoid privileged container application through policy management to reduce the + effects of potential attacks. + +- Enable integration with other security ecosystem (SIEM) + +- Isolate environments (Dev /test /Production) from other environments within + the Cluster. + +- Create administrative boundaries between resources using Namespace and avoid + using default Namespaces. + +- Enable Seccomp to ensure that the workloads have restricted actions available + within the container application. + +- Limit discovery by restricting services and users that can access Cluster + management metadata on configuration, containers and nodes + +Control network access to sensitive ports +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Kubernetes clusters usually listen on a range of well-defined and distinctive +ports which makes it easy to identify the clusters and attack them. Hence, it is +highly recommended to configure authentication and authorisation on the cluster +and cluster nodes. + +The `Kubernetes documentation `__ specifies the default ports used in Kubernetes. Make sure that your +network blocks access to unnecessary ports and consider limiting access to the Kubernetes +API server except from trusted networks. + +**Master node(s):** + +======== ========== ======================= +Protocol Port Range Purpose +======== ========== ======================= +TCP 6443 Kubernetes API Server +TCP 2379-2380 etcd server client API +TCP 10250 Kubelet API +TCP 10259 kube-scheduler +TCP 10257 kube-controller-manager +======== ========== ======================= + +**Worker nodes:** + +======== =========== ================= +Protocol Port Range Purpose +======== =========== ================= +TCP 10250 Kubelet API +TCP 30000-32767 NodePort Services +======== =========== ================= + +Controlling access to the Kubernetes API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Kubernetes platform is controlled using APIs, which are the first items to be secured in order to defend against attackers. +Controlling who has access and what actions they are allowed to perform is the primary concern. + +Use Transport Layer Security and Service Mesh +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Communication in the cluster between services should be handled using TLS, +encrypting all traffic by default. Kubernetes expects that all API communication +in the cluster is encrypted by default with TLS, and the majority of installation methods +will allow the necessary certificates to be created and distributed to the cluster components. +Note that some components and installation methods may enable local ports over +HTTP and administrators should familiarize themselves with the settings of each +component to identify potentially unsecured traffic. + +Advances in network technology, such as the service mesh, have led to the +creation of products like LinkerD and Istio which can enable TLS by default +while providing extra telemetry information on transactions between services. +The service mesh is a mesh of layer 7 proxies handling service-to-service communications. +The service mesh architecture consists of data plane components made up of network proxies paired with each micro-service, +and control plane components providing proxies configuration, managing TLS certificates and policies. +The two documents, `NIST SP 800-204A `__ +(Building Secure Microservices-based Applications Using Service-Mesh Architecture) and +`NIST SP 800-204B `__ +(Attribute-based Access Control for Microservices-based Applications Using a Service Mesh) provide guidance to deploy service mesh. + +API Authentication, API Authorisation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Secure all connections to a Kubernetes Cluster. Adopt the following security +authentication mechanisms: + +- Configure user roles and access levels to provide segregation of duties (RBAC) +- Use multi-factor authentication for all administrative access +- Use token-based or certificate-based service and session authentication + mechanisms +- Integrated with existing identity management platforms e.g., SAML, AD, etc. for + access control + +Restrict access to etcd and encrypt contents within etcd +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +etcd is a critical Kubernetes component which stores information on state and +secrets, and it should be protected differently from the rest of your cluster. +Write access to the API server's etcd is equivalent to gaining root on the +entire cluster, and even read access can be used to escalate privileges fairly +easily. + +The Kubernetes scheduler will search etcd for pod definitions that do not have a +node. It then sends the pods it finds to an available kubelet for scheduling. +Validation for submitted pods is performed by the API server before it writes +them to etcd, so malicious users writing directly to etcd can bypass many +security mechanisms e.g., PodSecurityPolicies. + +Administrators should always use strong credentials from the API servers to +their etcd server, such as mutual auth via TLS client certificates, and it is +often recommended to isolate the etcd servers behind a firewall that only the +API servers may access. + +Controlling access to the Kubelet +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Kubelets expose HTTPS endpoints which grant powerful control over the node and +containers. By default Kubelets allow unauthenticated access to this API. +Production clusters should enable Kubelet authentication and authorization + +Securing Kubernetes Dashboard +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Kubernetes dashboard is a webapp for managing your cluster. It is not a +part of the Kubernetes cluster itself, it has to be installed by the owners of +the cluster; a number of tutorials show how to do this. +Unfortunately, most of them create a service account with very high privileges. +This caused Tesla and some others to be hacked via such a poorly configured Kubernetes +dashboard (Reference: `Tesla cloud resources are hacked to run +cryptocurrency-mining malware `__). + +To prevent attacks via the dashboard, you should follow some best practices: + +- Do not expose the dashboard without additional authentication to the public. + There is no need to access such a powerful tool from outside your LAN +- Turn on RBAC, so you can limit the service account the dashboard uses +- Review the privileges granted to the service account of the dashboard privileges, + and remove disable any additional privileges assigned. +- Grant permissions per user, so each user can only access what they are supposed to + access +- If using network policies, block requests to the dashboard + even from internal pods (this will not affect the proxy tunnel via kubectl + proxy) +- Before version 1.8, the dashboard had a service account with full privileges, + so check that there is no role binding for cluster-admin left. +- Deploy the dashboard with an authenticating reverse proxy, with multi-factor + authentication enabled. This can be done with either embeded OpenID Connect (OIDC) id_tokens or + using Kubernetes Impersonation. This allows the use of the dashboard with the + user's credentials instead of using a privileged ServiceAccount. This method + can be used on both on-prem and managed cloud clusters. + +Use Namespaces to Establish Security Boundaries +----------------------------------------------- + +Namespaces in Kubernetes is the first level of isolation between components. It +is easier to apply security controls (Network Policies, Pod policies, etc.) to +different types of workloads when deployed in separate Namespaces. + +Separate Sensitive Workload +--------------------------- + +To limit the potential impact of a compromise, it is recommended to run +sensitive workloads on a dedicated set of nodes. This approach reduces the +risk of a sensitive application being accessed through a less-secure application +that shares a container runtime or host. + +- The separation can be achieved by using node pools and Kubernetes Namespaces. + +Create and Define Network Policies +---------------------------------- + +Network Policies allow Kubernetes managers to control network access into and +out of the cloud native applications. It is recommended to have a well defined +ingress and egress policy for cloud native applications. It is also important to +modify the default network policies, such as blocking or allowing traffic from +other Namespaces or Clusters while ensuring the Namespaces/Clusters are running +with policy support enabled. + +Run latest Version +------------------ + +As new security features and patches are added in every quarterly update, it is +important to take advantage of these fixes and patches. + +- It is recommended to run the latest release with its most recent patches. + +Secure Platform Metadata +------------------------ + +Kubernetes metadata contain sensitive information including kubelet admin +credentials. It is recommended to secure them using encryption to avoid this +being stolen and use to for escalated privileges in the the Cluster. + +- Limit discovery by restricting services and users that can access Cluster + management metadata on configuration, container application, and nodes +- Ensure all metadata is encrypted and network access runs over TLS connections + +Enable Logging and Monitoring +----------------------------- + +Logging, monitoring, alerting and log aggregation are essential for Kubernetes. +Enable and monitor audit logs for anomalous or unwanted API calls, especially +any authorisation failure. + +Run-Time Security +----------------- + +The following are recommended best practices for container run-time: + +- Integrate run-time processes to Security Information and Event Monitoring + (SIEM) +- Use container-aware run-time defence tools +- Ensure all running cloud native applications are from secure and verified + images +- Cloud native applications are not run with root privileges +- Ensure sensitive workloads are properly segmented by Namespaces or Cluster to + mitigate the scope of compromise. + +Secrets Management +------------------ + +It is recommended that the principle of least privilege is applied to secret +management in Kubernetes: + +- Ensure that the cloud native applications can only read the secrets that these + applications need +- Have different set of secrets for different environments(like production, + development, and testing) + +Secret values protect sensitive data, it is recommended to protect them from +unauthorised access. Ideally, by being protected at rest and in transit. +Encryption in transit is achieved by encrypting the traffic between the +Kubernetes control-plane components and worker nodes using TLS. + +It is recommended that Secrets are not be stored in scripts or code but provided +dynamically at runtime as needed. Keep any sensitive data, including SSH keys, +API access keys, and database credentials, in a secure data repository such as a +key manager or vault. Only pull credentials on demand and over secure channels +to ensure sensitive data is not written to disk unprotected. The key manager or +vault encryption keys should be backed by a FIPS 140-2 Hardware Security Module. +It is also important to implement the following: + +- Check there are no hard-coded passwords, keys, and other sensitive items in + the container application. +- Where possible use security tools to automate scanning for hard-coded + passwords, keys, and other sensitive items in the container application + +Trusted Registry +---------------- + +Ensure that the container registry only accepts container images from trusted +sources that have tested and validated the images. Where images are provided by +third parties, define and follow a formal process to validate compliance with +security requirements. Also ensure that access control is applied to registries +requiring unique credentials, to limit who can control the build or add images. + +- It is strongly recommended that network access to the registry is secured + using TLS, SSL or VPN connections to ensure trust. +- Ensure container applications are validated to assess their use and + applicability as well as scanned for viruses and vulnerabilities. Only deploy + container application from images that are signed with a trusted key +- Ensure the latest certified container application is always selected by + versioning images +- Trusted repository and registry services should reject containers that are not + properly signed +- Use approved registries for images loaded into production +- Where possible, use third-party products to validate container content both + before and after deployment + +Ensure stale images are removed from the registry. Remove unsafe, vulnerable +images (e.g. containers should no longer be used based on time triggers and +labels associated with images). + +Isolation +--------- + +.. _vm-vs-container-isolation: + +VM vs. Container Isolation +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sometimes container isolation is compared directly with VM based isolation, with +the conclusion "*there are issues with container isolation, it is not as good as +VM isolation*". Such 1:1 comparison is not reasonable because VM and container +based isolation are fundamentally different: + +- VMs: hard isolation, in the layers underlying the application SW +- Containers: isolation by SW based mechanisms available in OS, Docker and + Kubernetes. A container workload is just a set of Linux processes. It is + *possible* to configure SW based *additional isolation* for container + workloads, for example by kernel namespaces. + +The primary isolation mechanism in Kubernetes environment should be VM or +physical machine based. This implies that multiple cloud native applications +should not be deployed together in the same Kubernetes Cluster - unless these +applications have been planned and verified to co-exist. Thus, the default is to +allocate one Namespace per Cloud Native Network Function (CNF). + +Container Isolation in Kubernetes Cluster +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Namespaces +^^^^^^^^^^ + +Kubernetes Namespaces should be used to provide resource isolation within a +Kubernetes Cluster. They should not be used to isolate different steps in the +deployment process like Development, Production, or Testing. The most reliable +separation is achieved by deploying sensitive workloads into dedicated Clusters. diff --git a/doc/ref_arch/kubernetes/chapters/chapter06.md b/doc/ref_arch/kubernetes/chapters/chapter06.md index 968b6ffc64..b9d1d5caa2 100644 --- a/doc/ref_arch/kubernetes/chapters/chapter06.md +++ b/doc/ref_arch/kubernetes/chapters/chapter06.md @@ -1,41 +1,22 @@ -[<< Back](../../kubernetes) -# 6. API and Feature Testing requirements +# API and Feature Testing requirements -

scope

- -## Table of Contents - -- [6. API and Feature Testing requirements](#6-api-and-feature-testing-requirements) - - [6.1 Introduction](#61-introduction) - - [6.1.1 Kubernetes feature gate policy](#611-kubernetes-feature-gate-policy) - - [6.1.2 Kubernetes API policy](#612-kubernetes-api-policy) - - [6.2 API Machinery Special Interest Group](#62-api-machinery-special-interest-group) - - [6.3 Apps Special Interest Group](#63-apps-special-interest-group) - - [6.4 Auth Special Interest Group](#64-auth-special-interest-group) - - [6.5 Cluster Lifecycle Special Interest Group](#65-cluster-lifecycle-special-interest-group) - - [6.6 Instrumentation Special Interest Group](#66-instrumentation-special-interest-group) - - [6.7 Network Special Interest Group](#67-network-special-interest-group) - - [6.8 Node Special Interest Group](#68-node-special-interest-group) - - [6.9 Scheduling Special Interest Group](#69-scheduling-special-interest-group) - - [6.10 Storage Special Interest Group](#610-storage-special-interest-group) - -## 6.1 Introduction +## Introduction The CNCF has defined a [Testing Special Interest Group](https://github.com/kubernetes/community/blob/master/sig-testing/charter.md) to make it easier for the community to write and run tests, and to contribute, analyse and act upon test results. This chapter maps the requirements written in the previous chapters as mandatory Special Interest Group Features. It enforces the overall requirements traceability to testing, especially those offered for [End-to-End Testing](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md). The Anuket Reference Conformance (RC2) testing then matches the following Features tabs defined here. -### 6.1.1 Kubernetes feature gate policy +### Kubernetes feature gate policy [Feature gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) are a set of key-value pairs that describe Kubernetes features. The components of the control plane of Kubernetes Clusters can be run with different Feature Gate settings. A feature can be in Alpha, Beta or GA stage: -- Alpha features are disabled by default, may be buggy, and support may be dropped -- Beta features are enabled by default, are well tested, and support will not be dropped (although breaking API changes may happen) -- GA features are stable, always enabled and cannot be disabled. +* Alpha features are disabled by default, may be buggy, and support may be dropped +* Beta features are enabled by default, are well tested, and support will not be dropped (although breaking API changes may happen) +* GA features are stable, always enabled and cannot be disabled. The policy for RA2 to include Kubernetes features as mandatory is: @@ -43,7 +24,7 @@ The policy for RA2 to include Kubernetes features as mandatory is: A list of feature gates is available [here](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates). -### 6.1.2 Kubernetes API policy +### Kubernetes API policy The [Kubernetes API](https://kubernetes.io/docs/reference/using-api/) supports all operations and communications between components, and external user commands. Everything in the Kubernetes platform is treated as an API object. @@ -80,7 +61,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | scheduling.k8s.io | v1 | | storage.k8s.io | v1 | -## 6.2 [API Machinery Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-api-machinery) +## [API Machinery Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-api-machinery) | **Labels** | **Mandatory** | **Description** | |----------------------------------------|:-------------:|:----------------| @@ -92,7 +73,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | Feature:ScopeSelectors | X | Verify ResourceQuota with terminating scopes through scope selectors | | Feature:[StorageVersionAPI](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#storageversion-v1alpha1-internal-apiserver-k8s-io)| | -## 6.3 [Apps Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-apps) +## [Apps Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-apps) | **Labels** | **Mandatory** | **Description** | |------------------------------|:-------------:|:----------------| @@ -106,7 +87,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | Feature:[TaintEviction](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions)| | All pods on the unreachable node should be marked as NotReady upon the node turn NotReady AND all pods should be evicted after eviction timeout passes | | Feature:[TTLAfterFinished](https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/)| X | Job should be deleted once it finishes after TTL seconds | -## 6.4 [Auth Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-auth) +## [Auth Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-auth) | **Labels** | **Mandatory** | **Description** | |----------------------------------------|:-------------:|:----------------| @@ -118,7 +99,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | Feature:PodSecurityPolicy | | Should enforce the restricted policy.PodSecurityPolicy | | NodeFeature:FSGroup | X | ServiceAccounts should set ownership and permission when RunAsUser or FsGroup is present | -## 6.5 [Cluster Lifecycle Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle) +## [Cluster Lifecycle Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle) | **Labels** | **Mandatory** | **Description** | |-------------------------|:-------------:|:----------------| @@ -126,7 +107,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | None | X | Kubernetes mainstream features | | Feature:BootstrapTokens | X | Should delete the token secret when the secret expired | -## 6.6 [Instrumentation Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-instrumentation) +## [Instrumentation Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-instrumentation) | **Labels** | **Mandatory** | **Description** | |------------------------------------------|:-------------:|:----------------| @@ -139,7 +120,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | Feature:StackdriverMetadataAgent | | Stackdriver Monitoring should run Stackdriver Metadata Agent | | Feature:StackdriverMonitoring | | | -## 6.7 [Network Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-network) +## [Network Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-network) | **Labels** | **Mandatory** | **Description** | |-------------------------------------|:-------------:|:----------------| @@ -161,7 +142,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | Feature:SCTP | | should allow creating a basic SCTP service with pod and endpoints | | Feature:SCTPConnectivity | | Pods should function for intra-pod communication: sctp | -## 6.8 [Node Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-node) +## [Node Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-node) | **Labels** | **Mandatory** | **Description** | |-------------------------------------------|:-------------:|:----------------| @@ -178,7 +159,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | NodeFeature:RuntimeHandler | | RuntimeClass should run a Pod requesting a RuntimeClass with a configured handler | | NodeFeature:[Sysctls](https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/)| X | Should not launch unsafe, but not explicitly enabled sysctls on the node | -## 6.9 [Scheduling Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-scheduling) +## [Scheduling Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-scheduling) | **Labels** | **Mandatory** | **Description** | |---------------------------------------|:-------------:|:----------------| @@ -188,7 +169,7 @@ The list of [API groups](https://kubernetes.io/docs/reference/generated/kubernet | Feature:[LocalStorageCapacityIsolation](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) | X | Validates local ephemeral storage resource limits of pods that are allowed to run | | Feature:Recreate | | Run Nvidia GPU Device Plugin tests with a recreation | -## 6.10 [Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) +## [Storage Special Interest Group](https://github.com/kubernetes/community/tree/master/sig-storage) | **Labels** | **Mandatory** | **Description** | |---------------------------------------|:-------------:|:-------------------------------| diff --git a/doc/ref_arch/kubernetes/chapters/chapter06.rst b/doc/ref_arch/kubernetes/chapters/chapter06.rst new file mode 100644 index 0000000000..a072167494 --- /dev/null +++ b/doc/ref_arch/kubernetes/chapters/chapter06.rst @@ -0,0 +1,218 @@ +API and Feature Testing requirements +==================================== + +Introduction +------------ + +The CNCF has defined a +`Testing Special Interest Group `__ to make it easier for the community to write and run tests, and to contribute, analyse and act upon test results. +This chapter maps the requirements written in the previous chapters as mandatory Special Interest Group Features. It enforces the overall requirements traceability to testing, especially those offered for `End-to-End Testing `__. +The Anuket Reference Conformance (RC2) testing then matches the following Features tabs defined here. + +Kubernetes feature gate policy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`Feature gates `__ are a set of key-value pairs that describe Kubernetes features. The components of the control plane of Kubernetes Clusters can be run with different Feature Gate settings. + +A feature can be in Alpha, Beta or GA stage: + +- Alpha features are disabled by default, may be buggy, and support may be dropped +- Beta features are enabled by default, are well tested, and support will not be dropped (although breaking API changes may happen) +- GA features are stable, always enabled and cannot be disabled. + +The policy for RA2 to include Kubernetes features as mandatory is: + + Only features that are either in Beta or GA stage can be made mandatory, subject to RA2 requirements. + +A list of feature gates is available `here `__. + +Kubernetes API policy +~~~~~~~~~~~~~~~~~~~~~ + +The `Kubernetes API `__ supports all operations and communications between components, and external user commands. +Everything in the Kubernetes platform is treated as an API object. +Different API versions indicate different levels of stability and support. An API can have Alpha, Beta or Stable versions. The version of APIs that are backed by a feature will match the stage of the feature itself (i.e. Alpha, Beta or GA/Stable). + +The policy for RA2 to include Kubernetes APIs as mandatory is: + + Only APIs that are either in Beta or Stable stage can be made mandatory, subject to RA2 requirements. + +The Kubernetes API reference is available `here `__. + +The list of `API groups `__ that are mandatory is: + +============================ ==================== +Group Version +============================ ==================== +admissionregistration.k8s.io v1 +apiextensions.k8s.io v1 +apiregistration.k8s.io v1 +apps v1 +authentication.k8s.io v1 +authorization.k8s.io v1 +autoscaling v1, v2beta2, v2beta1 +batch v1 +certificates.k8s.io v1 +coordination.k8s.io v1 +core v1 +discovery.k8s.io v1 +events.k8s.io v1 +flowcontrol.apiserver.k8s.io v1beta1 +networking.k8s.io v1 +node.k8s.io v1 +policy v1 +rbac.authorization.k8s.io v1 +scheduling.k8s.io v1 +storage.k8s.io v1 +============================ ==================== + +`API Machinery Special Interest Group `__ +---------------------------------------------------------------------------------------------------------------- + +====================================================================================================================================================== ============= ========================================================================================= +**Labels** **Mandatory** **Description** +====================================================================================================================================================== ============= ========================================================================================= +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:ComprehensiveNamespaceDraining X Namespaces should always delete fast (ALL of 100 namespaces in 150 seconds) +Feature:`CrossNamespacePodAffinity `__ Should verify ResourceQuota with cross namespace pod affinity scope using scope-selectors +Feature:`PodPriority `__ X Verify ResourceQuota's priority class scope against a pod with different priority class +Feature:ScopeSelectors X Verify ResourceQuota with terminating scopes through scope selectors +Feature:`StorageVersionAPI `__ +====================================================================================================================================================== ============= ========================================================================================= + +`Apps Special Interest Group `__ +---------------------------------------------------------------------------------------------- + +====================================================================================================================================== ============= ====================================================================================================================================================== +**Labels** **Mandatory** **Description** +====================================================================================================================================== ============= ====================================================================================================================================================== +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:`DaemonSetUpdateSurge `__ Daemon set should surge pods onto nodes when spec was updated and update strategy is RollingUpdate +Feature:`IndexedJob `__ Should create pods for an Indexed job with completion indexes +Feature:`StatefulSet `__ Should creating a working zookeeper cluster +Feature:StatefulUpgrade Stateful upgrade should maintain a functioning cluster +Feature:`SuspendJob `__ Should not create pods when created in suspend state +Feature:`TaintEviction `__ All pods on the unreachable node should be marked as NotReady upon the node turn NotReady AND all pods should be evicted after eviction timeout passes +Feature:`TTLAfterFinished `__ X Job should be deleted once it finishes after TTL seconds +====================================================================================================================================== ============= ====================================================================================================================================================== + +`Auth Special Interest Group `__ +---------------------------------------------------------------------------------------------- + +============================================================================================================================================================= ============= ======================================================================================================= +**Labels** **Mandatory** **Description** +============================================================================================================================================================= ============= ======================================================================================================= +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:`BoundServiceAccountTokenVolume `__ ServiceAccount admission controller migration master upgrade should maintain a functioning cluster +Feature:NodeAuthenticator X The kubelet's main port 10250 should reject requests with no credentials +Feature:NodeAuthorizer X Setting existing and non-existent attributes should exit with the Forbidden error, not a NotFound error +Feature:PodSecurityPolicy Should enforce the restricted policy.PodSecurityPolicy +NodeFeature:FSGroup X ServiceAccounts should set ownership and permission when RunAsUser or FsGroup is present +============================================================================================================================================================= ============= ======================================================================================================= + +`Cluster Lifecycle Special Interest Group `__ +------------------------------------------------------------------------------------------------------------------------ + +======================= ============= ====================================================== +**Labels** **Mandatory** **Description** +======================= ============= ====================================================== +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:BootstrapTokens X Should delete the token secret when the secret expired +======================= ============= ====================================================== + +`Instrumentation Special Interest Group `__ +-------------------------------------------------------------------------------------------------------------------- + +======================================== ============= ============================================================================================= +**Labels** **Mandatory** **Description** +======================================== ============= ============================================================================================= +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:Elasticsearch Should check that the Kibana logging instance is alive +Feature:StackdriverAcceleratorMonitoring Stackdriver Monitoring should have accelerator metrics +Feature:StackdriverCustomMetrics Stackdriver Monitoring should run Custom Metrics - Stackdriver Adapter for new resource model +Feature:StackdriverExternalMetrics Stackdriver Monitoring should run Custom Metrics - Stackdriver Adapter for external metrics +Feature:StackdriverMetadataAgent Stackdriver Monitoring should run Stackdriver Metadata Agent +Feature:StackdriverMonitoring +======================================== ============= ============================================================================================= + +`Network Special Interest Group `__ +---------------------------------------------------------------------------------------------------- + +=============================================================================================== ============= ====================================================================================================================================================================================================================================================================================== +**Labels** **Mandatory** **Description** +=============================================================================================== ============= ====================================================================================================================================================================================================================================================================================== +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:Example Should create pod that uses DNS +Feature:Ingress Should prevent Ingress creation if more than 1 IngressClass marked as default +Feature:`IPv6DualStack `__ IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to Pods and Services. IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in 1.21, allowing the simultaneous assignment of both IPv4 and IPv6 addresses. +Feature:kubemci Should create ingress with pre-shared certificate +Feature:KubeProxyDaemonSetMigration Upgrade kube-proxy from static pods to a DaemonSet should maintain a functioning cluster +Feature:KubeProxyDaemonSetUpgrade Upgrade kube-proxy from static pods to a DaemonSet should maintain a functioning cluster +Feature:NEG Should sync endpoints to NEG +Feature:NoSNAT X Should be able to send traffic between Pods without SNAT +Feature:Networking-IPv4 X Networking should provide Internet connection for containers +Feature:Networking-IPv6 Networking should provide Internet connection for containers +Feature:Networking-Performance X run iperf2 +Feature:NetworkPolicy NetworkPolicy between server and client should enforce policy to allow traffic only from a different namespace, based on NamespaceSelector +Feature:PerformanceDNS Should answer DNS query for maximum number of services per cluster +Feature:SCTP should allow creating a basic SCTP service with pod and endpoints +Feature:SCTPConnectivity Pods should function for intra-pod communication: sctp +=============================================================================================== ============= ====================================================================================================================================================================================================================================================================================== + +`Node Special Interest Group `__ +---------------------------------------------------------------------------------------------- + +========================================================================================================================================================================================= ============= ==================================================================================================================================== +**Labels** **Mandatory** **Description** +========================================================================================================================================================================================= ============= ==================================================================================================================================== +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:Example X Liveness pods should be automatically restarted +Feature:ExperimentalResourceUsageTracking Resource tracking for 100 pods per node +Feature:GPUUpgrade Master upgrade should NOT disrupt GPU Pod +Feature:PodGarbageCollector Should handle the creation of 1000 pods +Feature:RegularResourceUsageTracking Resource tracking for 0 pods per node +Feature:`ProbeTerminationGracePeriod `__ X Probing container should override timeoutGracePeriodSeconds when LivenessProbe field is set +NodeFeature:`DownwardAPIHugePages `__ Downward API tests for huge pages should provide container's limits.hugepages-pagesize; and requests.hugepages-pagesize& as env vars +NodeFeature:`PodReadinessGate `__  X Pods should support pod readiness gates +NodeFeature:RuntimeHandler RuntimeClass should run a Pod requesting a RuntimeClass with a configured handler +NodeFeature:`Sysctls `__ X Should not launch unsafe, but not explicitly enabled sysctls on the node +========================================================================================================================================================================================= ============= ==================================================================================================================================== + +`Scheduling Special Interest Group `__ +---------------------------------------------------------------------------------------------------------- + +========================================================================================================================== ============= ================================================================================= +**Labels** **Mandatory** **Description** +========================================================================================================================== ============= ================================================================================= +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:GPUDevicePlugin Run Nvidia GPU Device Plugin tests +Feature:`LocalStorageCapacityIsolation `__ X Validates local ephemeral storage resource limits of pods that are allowed to run +Feature:Recreate Run Nvidia GPU Device Plugin tests with a recreation +========================================================================================================================== ============= ================================================================================= + +`Storage Special Interest Group `__ +---------------------------------------------------------------------------------------------------- + +==================================== ============= ============================== +**Labels** **Mandatory** **Description** +==================================== ============= ============================== +Conformance X Kubernetes conformance test +None X Kubernetes mainstream features +Feature:ExpandInUsePersistentVolumes +Feature:Flexvolumes +Feature:GKELocalSSD +Feature:VolumeSnapshotDataSource +Feature:Volumes X +Feature:vsphere +Feature:Windows +NodeFeature:EphemeralStorage X +NodeFeature:FSGroup X +==================================== ============= ============================== diff --git a/doc/ref_arch/kubernetes/chapters/chapter07.md b/doc/ref_arch/kubernetes/chapters/chapter07.md index 9802498b53..96b2f12d64 100644 --- a/doc/ref_arch/kubernetes/chapters/chapter07.md +++ b/doc/ref_arch/kubernetes/chapters/chapter07.md @@ -1,39 +1,17 @@ -[<< Back](../../kubernetes) -# 7. Gaps, Innovation, and Development +# Gaps, Innovation, and Development -

scope

- -## Table of Contents - -- [7. Gaps, Innovation, and Development](#7-gaps-innovation-and-development) - - [7.1 Introduction](#71-introduction) - - [7.2 Gap analysis](#72-gap-analysis) - - [7.2.0 Gap template](#720-gap-template) - - [7.2.1 Container run-time Interfaces towards NFVI resources](#721-container-run-time-interfaces-towards-nfvi-resources) - - [7.2.2 Multi-tenancy and workload isolation with Kubernetes](#722-multi-tenancy-and-workload-isolation-with-kubernetes) - - [7.2.3 Kubernetes as a VM-based VNF Orchestrator](#723-kubernetes-as-a-vm-based-vnf-orchestrator) - - [7.2.4 Multiple network interfaces on Pods](#724-multiple-network-interfaces-on-pods) - - [7.2.5 Dynamic network management](#725-dynamic-network-management) - - [7.2.6 Control Plane Efficiency](#726-control-plane-efficiency) - - [7.2.7 Interoperability with VNF-based networking](#727-interoperability-with-vnf-based-networking) - - [7.2.8 HW topology aware hugepages](#728-hw-topology-aware-hugepages) - - [7.2.9 User namespaces in Kubernetes](#729-user-namespaces-in-kubernetes) - - [7.3 Proposals & Resolution](#73-proposals--resolution) - - [7.3.1 Multi-tenancy and workload isolation with Kubernetes](#731-multi-tenancy-and-workload-isolation-with-kubernetes) - - [7.4 Development Efforts](#74-development-efforts) - -## 7.1 Introduction +## Introduction While this Reference Architecture is being developed, Gaps will be identified that require addressing. This chapter will highlight these gaps in detail and may provide proposed solutions. As a result, various “upstream” community projects will be identified and will be targeted for development efforts. -## 7.2 Gap analysis +## Gap analysis * Container Run-Time Interfaces towards NFVI resources. * Multi-Tenancy * K8s as VM based VNF Orchestrator -### 7.2.0 Gap template +### Gap template > **Related requirements:** List the requirement references `req.abc.xyz.00` from RA2 or RM which this gap tries to address. @@ -41,13 +19,13 @@ While this Reference Architecture is being developed, Gaps will be identified th > **Gap description:** Describe which functionality described in the related requirements is currently missing in the implementations you're aware of. Include references to work ongoing in the target project, which may adress the gap. -### 7.2.1 Container run-time Interfaces towards NFVI resources +### Container run-time Interfaces towards NFVI resources > This is the southbound infrastructure resources from the container platform as presented by the IaaS provider. > e.g. network interface type that is presented to a running container. -### 7.2.2 Multi-tenancy and workload isolation with Kubernetes +### Multi-tenancy and workload isolation with Kubernetes **Related requirements:** `e.man.004` `sec.ci.008` `sec.wl.005``sec.wl.006` @@ -55,7 +33,7 @@ While this Reference Architecture is being developed, Gaps will be identified th **Gap description:** Today, Kubernetes lacks hard multi-tenancy capabilities that give the ability to allow untrusted tenants to share infrastructure resources. This presents a security problem when operators seek to separate workloads by categorization or even just production vs non-production. Further, tenant networks need to be both segregated, but still centrally administered and maintained. Beyond just security, this also presents an operational problem. Trying to deploy too many CNFs into the same cluster could result in version conflicts, configuration conflicts, and problems with software life cycle management. Finally, without proper isolation there is an increased risk of cascading failures. -### 7.2.3 Kubernetes as a VM-based VNF Orchestrator +### Kubernetes as a VM-based VNF Orchestrator > **Related requirements:** None. @@ -63,23 +41,23 @@ While this Reference Architecture is being developed, Gaps will be identified th > **Gap description:** Kubernetes and at least one CRI compliant runtime should support the running of VNFs without requiring changes to the VNF's architecture and deployment artifacts. -### 7.2.4 Multiple network interfaces on Pods +### Multiple network interfaces on Pods -> **Related requirements:** [RM Chapter 4.2.2](../../../ref_model/chapters/chapter04.md#422-virtual-network-interface-specifications) +> **Related requirements:** [RM Chapter 4.2.2](../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications) > **Baseline project:** _Kubernetes_ > **Gap description:** Kubernetes does not have native support for multiple Pod interfaces, therefore a CNI multiplexer, like [DANM](https://github.com/nokia/danm) or [Multus](https://github.com/intel/multus-cni) is needed to provision multiple interfaces. Implementation of different network services for the interfaces, like Network Policies, Ingress, Egress or Load Balancers depends on the feature set of the CNI multiplexer and the CNI plugins it uses, therefore it is inconsistent. -### 7.2.5 Dynamic network management +### Dynamic network management -> **Related requirements:** [req.inf.ntw.03](chapter02.md#23-kubernetes-architecture-requirements) +> **Related requirements:** [req.inf.ntw.03](chapter02.md#kubernetes-architecture-requirements) > **Baseline project:** _Kubernetes_ > **Gap description:** Kubernetes does not have an API for network management, therefore a different CNI plugin, like [DANM](https://github.com/nokia/danm) needs to be used to expose Kubernetes network services on an API. Alternatively this is done today with Netconf etc. integration with SDN controllers, for example connecting individual VPNs - e.g. L3VPN - onto the CNF, on demand. -### 7.2.6 Control Plane Efficiency +### Control Plane Efficiency > **Related requirements:** None @@ -88,7 +66,7 @@ While this Reference Architecture is being developed, Gaps will be identified th > **Gap description:** For example, in situations where multiple sites / availability zones exist, an operator may choose to run multiple Kubernetes clusters, not only for security/multitenancy reasons but also fault, resilience, latency, etc. This produces an overhead of Kubernetes Masters - is there a way of making this more efficient whilst still able to meet the non-functional requirements of the operator (fault, resilience, latency, etc.) -### 7.2.7 Interoperability with VNF-based networking +### Interoperability with VNF-based networking > **Related requirements:** None @@ -98,15 +76,15 @@ This produces an overhead of Kubernetes Masters - is there a way of making this Note: with an underlying IaaS this is possible, but then it introduces (undesirable) dependency between workload orchestration in K8s and infrastructure orchestration in IaaS. -### 7.2.8 HW topology aware huge pages +### HW topology aware huge pages **Related requirements:** `nfvi.com.cfg.004` and `nfvi.com.cfg.002` **Baseline project:** _Kubernetes_ -**Gap description:** Memory Manager was added in v1.21 as alpha feature. More in [3.2.1.3 Memory and Huge Pages Resources Management](chapter03.md#3213-memory-and-huge-pages-resources-management). +**Gap description:** Memory Manager was added in v1.21 as alpha feature. More in [3.2.1.3 Memory and Huge Pages Resources Management](chapter03.md#memory-and-huge-pages-resources-management). -### 7.2.9 User namespaces in Kubernetes +### User namespaces in Kubernetes **Related requirements:** @@ -119,9 +97,9 @@ Note: with an underlying IaaS this is possible, but then it introduces (undesira **Gap description:** Kubernetes does not support namespace scoped user IDs (UIDs). Therefore, when a container-based application requires system privileges the container either needs to run in privileged mode or the infrastructure needs to provide random system UIDs. Randomised UIDs result in errors when the application needs to set kernel capabilities (e.g.: in case of VLAN trunking) or when a Pod shares data with other Pods via persistent storage. The "privileged mode" solution is not secure while "random UID" solution is error prone, and therefore these techniques should not be used. Support for proper user namespaces in Kubernetes is [under discussion](https://github.com/kubernetes/enhancements/pull/2101). -## 7.3 Proposals & Resolution +## Proposals & Resolution -### 7.3.1 Multi-tenancy and workload isolation with Kubernetes +### Multi-tenancy and workload isolation with Kubernetes Kubernetes is not a single cluster solution. This has been demonstrated across the industry from case studies at prominent companies like [Twitter](https://www.alibabacloud.com/blog/what-can-we-learn-from-twitters-move-to-kubernetes_595156), [USA Today](https://medium.com/usa-today-network/there-and-back-again-scaling-multi-tenant-kubernetes-cluster-s-67afb437716c), [Zalando](https://www.youtube.com/watch?v=LpFApeaGv7A), and [Alibaba](https://www.cncf.io/blog/2019/12/12/demystifying-kubernetes-as-a-service-how-does-alibaba-cloud-manage-10000s-of-kubernetes-clusters/) to the bi-annual CNCF survey that finds that the number of clusters being deployed within an organization is growing. While there are many reasons behind the multi cluster paradigm, examining the gap above we find that a multi cluster solution can address many of these problems like security and software life cycle management. @@ -129,4 +107,4 @@ Without multi tenancy within a clusters, separate clusters must be used to provi If running multiple clusters is the only solution to meeting these workload and infrastructure requirements, the operational burden of this model must also be considered. Running a multitude of clusters at scale could be a massive operational challenge if done manually. Any operator considering running Kubernetes at scale should carefully evaluate their multi cluster management strategy including the management of the applications within those clusters. -## 7.4 Development Efforts +## Development Efforts diff --git a/doc/ref_arch/kubernetes/chapters/chapter07.rst b/doc/ref_arch/kubernetes/chapters/chapter07.rst new file mode 100644 index 0000000000..e44b5a1836 --- /dev/null +++ b/doc/ref_arch/kubernetes/chapters/chapter07.rst @@ -0,0 +1,145 @@ +Gaps, Innovation, and Development +================================= + +Introduction +------------ + +While this Reference Architecture is being developed, Gaps will be identified that require addressing. This chapter will highlight these gaps in detail and may provide proposed solutions. As a result, various “upstream” community projects will be identified and will be targeted for development efforts. + +Gap analysis +------------ + +- Container Run-Time Interfaces towards NFVI resources. +- Multi-Tenancy +- K8s as VM based VNF Orchestrator + +Gap template +~~~~~~~~~~~~ + + **Related requirements:** List the requirement references ``req.abc.xyz.00`` from RA2 or RM which this gap tries to address. + +.. + + **Baseline project:** Describe against which upstream project the gap exists e.g. *Kubernetes*. If the gap is not against any specific upstream project, state "none". + + **Gap description:** Describe which functionality described in the related requirements is currently missing in the implementations you're aware of. Include references to work ongoing in the target project, which may adress the gap. + +Container run-time Interfaces towards NFVI resources +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + This is the southbound infrastructure resources from the container platform as presented by the IaaS provider. + +.. + + e.g. network interface type that is presented to a running container. + +Multi-tenancy and workload isolation with Kubernetes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Related requirements:** ``e.man.004`` ``sec.ci.008`` :literal:`sec.wl.005``sec.wl.006` + +**Baseline project:** *Kubernetes* + +**Gap description:** Today, Kubernetes lacks hard multi-tenancy capabilities that give the ability to allow untrusted tenants to share infrastructure resources. This presents a security problem when operators seek to separate workloads by categorization or even just production vs non-production. Further, tenant networks need to be both segregated, but still centrally administered and maintained. Beyond just security, this also presents an operational problem. Trying to deploy too many CNFs into the same cluster could result in version conflicts, configuration conflicts, and problems with software life cycle management. Finally, without proper isolation there is an increased risk of cascading failures. + +Kubernetes as a VM-based VNF Orchestrator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + **Related requirements:** None. + +.. + + **Baseline project:** *Kubernetes*, *Kubevirt* + + **Gap description:** Kubernetes and at least one CRI compliant runtime should support the running of VNFs without requiring changes to the VNF's architecture and deployment artifacts. + +Multiple network interfaces on Pods +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + **Related requirements:** `RM Chapter 4.2.2 <../../../ref_model/chapters/chapter04.md#virtual-network-interface-specifications>`__ + +.. + + **Baseline project:** *Kubernetes* + + **Gap description:** Kubernetes does not have native support for multiple Pod interfaces, therefore a CNI multiplexer, like `DANM `__ or `Multus `__ is needed to provision multiple interfaces. Implementation of different network services for the interfaces, like Network Policies, Ingress, Egress or Load Balancers depends on the feature set of the CNI multiplexer and the CNI plugins it uses, therefore it is inconsistent. + +Dynamic network management +~~~~~~~~~~~~~~~~~~~~~~~~~~ + + **Related requirements:** `req.inf.ntw.03 `__ + +.. + + **Baseline project:** *Kubernetes* + + **Gap description:** Kubernetes does not have an API for network management, therefore a different CNI plugin, like `DANM `__ needs to be used to expose Kubernetes network services on an API. Alternatively this is done today with Netconf etc. integration with SDN controllers, for example connecting individual VPNs - e.g. L3VPN - onto the CNF, on demand. + +Control Plane Efficiency +~~~~~~~~~~~~~~~~~~~~~~~~ + + **Related requirements:** None + +.. + + **Baseline project:** *Kubernetes* + + **Gap description:** For example, in situations where multiple sites / availability zones exist, an operator may choose to run multiple Kubernetes clusters, not only for security/multitenancy reasons but also fault, resilience, latency, etc. + This produces an overhead of Kubernetes Masters - is there a way of making this more efficient whilst still able to meet the non-functional requirements of the operator (fault, resilience, latency, etc.) + +Interoperability with VNF-based networking +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + **Related requirements:** None + +.. + + **Baseline project:** *Kubernetes* + + **Gap description:** For example, today in existing networks L3 VPNs are commonly used for traffic separation (e.g. separate L3 VPN for signalling, charging, LI, O&M etc.). CNFs will have to interwork with existing network elements and therefore a K8s POD will somehow need to be connected to a L3 VPN. Today this is only possible via Multus (or DANM), however typically there is a network orchestration responsibility to connect the network interface to a gateway router (where the L3 VPN is terminated). This network orchestration is not taken care of by K8s, nor there is a production grade solution in the open source space to take care of this. + +Note: with an underlying IaaS this is possible, but then it introduces (undesirable) dependency between workload orchestration in K8s and infrastructure orchestration in IaaS. + +HW topology aware huge pages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Related requirements:** ``nfvi.com.cfg.004`` and ``nfvi.com.cfg.002`` + +**Baseline project:** *Kubernetes* + +**Gap description:** Memory Manager was added in v1.21 as alpha feature. More in `3.2.1.3 Memory and Huge Pages Resources Management `__. + +User namespaces in Kubernetes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**Related requirements:** + +=========== =============================================================================================================================================================== +Reference Requirement +=========== =============================================================================================================================================================== +e.man.004 Capability to isolate resources between tenants +sec.sys.007 The Platform must implement controls enforcing separation of duties and privileges, least privilege use and least common mechanism (Role-Based Access Control). +=========== =============================================================================================================================================================== + +**Baseline project:** *Kubernetes* + +**Gap description:** Kubernetes does not support namespace scoped user IDs (UIDs). Therefore, when a container-based application requires system privileges the container either needs to run in privileged mode or the infrastructure needs to provide random system UIDs. Randomised UIDs result in errors when the application needs to set kernel capabilities (e.g.: in case of VLAN trunking) or when a Pod shares data with other Pods via persistent storage. The "privileged mode" solution is not secure while "random UID" solution is error prone, and therefore these techniques should not be used. Support for proper user namespaces in Kubernetes is `under discussion `__. + +.. _proposals--resolution: + +Proposals & Resolution +---------------------- + +.. _multi-tenancy-and-workload-isolation-with-kubernetes-1: + +Multi-tenancy and workload isolation with Kubernetes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Kubernetes is not a single cluster solution. This has been demonstrated across the industry from case studies at prominent companies like `Twitter `__, `USA Today `__, `Zalando `__, and `Alibaba `__ to the bi-annual CNCF survey that finds that the number of clusters being deployed within an organization is growing. While there are many reasons behind the multi cluster paradigm, examining the gap above we find that a multi cluster solution can address many of these problems like security and software life cycle management. + +Without multi tenancy within a clusters, separate clusters must be used to provide adequate separation for CNFs that require strong isolation. Putting CNFs may need to be separated for various reasons including different types of workloads based on their vendors, type like production vs. non production, per categorization, or supporting independent lifecycles. Having multiple clusters to deploy CNFs into allows operators to chose similar CNFs together while segregating those with different lifecycles from each other. CNFs deployed into the same cluster can be upgraded together to reduce the operational load while CNFs that require different versions, configurations, and dependencies can run in separate clusters and be upgraded independently if needed. + +If running multiple clusters is the only solution to meeting these workload and infrastructure requirements, the operational burden of this model must also be considered. Running a multitude of clusters at scale could be a massive operational challenge if done manually. Any operator considering running Kubernetes at scale should carefully evaluate their multi cluster management strategy including the management of the applications within those clusters. + +Development Efforts +-------------------