sync (#935)

* RA2 Ch4 - Host OS, Kubernetes, Container Runtime and Storage content (#874)

* RA2 Ch04 Initial Content

* Update chapter04.md

* RA2 Ch4 - HA notes

* RA2 Ch4 move content to Ch3

Host OS description

* RA2 Ch4 - updates to 4.2, 4.3, 4.4

Updates to Host OS, Kubernetes, Runtimes sections

* RA2 Ch4 - add more detail

* RA2 Ch4 storage updates

Storage updates, added OCI and changed status.

* RA2 Ch3 - typo correction

4-1 --> 3-1

* RA2 Ch4 - typos and updates

Typos and added note about etcd not running on worker nodes.

* RA2 Ch4 - added note about kubeadm

kubeadm driving Host OS versions

* RA2 Ch4 - remove rkt

Archived by CNCF

* RA2 Ch4 - remove note about Kata/gVisor

* RA2 Ch4 - update kubernetes version statement

added content to row60

* Update to 4.2 and Table 4-1 - remove specific operating systems and focus on kernel versions.

* RA2 Ch3 - typo correction

row54

* RA2 Ch4 - remove linux 4+ as redundant

* RA2 Ch4 - update to k8s version spec

row54

* RA2 Ch4 - 4.2 changes

row34/35

* RA2 Ch4 - merge conflict

* [RA2 Ch04]: Networking part of Ch4 in RA2 (#821)

* Initial commmit of the networking part of Ch4 in RA2

* Adding references to requirements

This change adds references to the different requirements from Ch01.

* Comparision between DANM and Multus added

Due to requests in comments this change adds a comparision table of the
relevant requirements and additional features of DANM and Multus.

* Adding MACVLAN and User Space CNI backend options

Addressing incoming comments this change adds the option to use MACVLAN CNI,
lists the backend options for the User Space CNi and adds the SR-IOV
Device Plugin to the SR-IOV related note.

* Add an eitors note about the list of CNI plugins

As we were not able to agree on the set of CNI plugins this
change adds a note that the list can be changed in the future.

* Must -> may for CNI-s

This change switches the "must" statements regarding to CNI-s to
"may" statements and adds a cladification text to the related
editors note how this will be fixed in the next releases.

* [RA-1 Ch02] update to handle multiple issues (#558)

* [RA-1 Ch02] update to handle multiple issues 

PR covers Issues #266, #314, #339 and #435

* [RA-1 Ch02] Updated

[RA-1 Ch02] Updated to resolve valuable feedback from multiple folks and @collivier suggested changes to Block storage and networking.

Resolves Issues #266,  #314, #339  and #435

* [RA-1 Ch02] Resolve Issues

[RA-1 Ch02] Resolve Issues
Changes to 'req.inf.com.08' and add 'req.inf.com.09'

* [RA-1 Ch02] Resolve "remote Block Storage"

Changed:
| `req.inf.stg.01` | Storage | The Architecture **must** provide remote (not directly attached to the host) Block storage for VM Instances. |
|--------|-------|------------------------|

* [RA-1 Ch02} Resolve 2.3 Architecture introduction

[RA-1 Ch02} Resolve 2.3 Architecture introduction as per @CsatariGergely

* [RA-1 Ch02] Commented out req.sec.ntw.01/02 

[RA-1 Ch02] Commented out req.sec.ntw.01 and req.sec.ntw.02

Co-authored-by: Rabi Abdel <45387599+rabiabdel@users.noreply.github.com>

* [RA1 Ch08] Add gaps in openstack release ocata, pike and stein (#843)

* [RI1 Ch08] Add gaps in openstack release ocata, pike and stein

* Add API versions
* Add features of pike and stein
* Add upgrade check for Stein

Fixes #842

* Add nova features

* Fix JWT short form and remaining services

* Update table of content

* Move into RA-8 from RI-8

* Delete Deprecated Annex (#933)

* [RI Ch07] Sridhar ch7 sec5and6 (#931)

* Chapter-7: Section 5 and 6.

Adding contents from Sections 5 and 6 of Chapter-7

Signed-off-by: Sridhar K. N. Rao <sridhar.rao@spirent.com>

* Fixed Typos and removed wrong special chars.

Signed-off-by: Sridhar K. N. Rao <sridhar.rao@spirent.com>

Co-authored-by: Rabi Abdel <51988225+rabi-abdel@users.noreply.github.com>

* [RC1] Fill all gaps already tracked as issues in CNTT (#918)

* Fill all gaps already tracked as issues in CNTT

Port VTP test cases to Xtesting [1] is key because the TC doesn't conform
to the principles and requirements defined.

#917

Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>

* Precise the gaps to avoid confusion

Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>

* Add tab for the new TC

Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>

* [RA-1 Ch04] Update Figure 4-1 (#927)

* [RA-1 Ch04] Update Figure 4-1

[RA-1 Ch04] @iangardner22 Updated Figure 4-1 and added network descriptions
(I am making the changes on behalf of @iangardner22)

* [RA-1 Ch04] Upload latest version of Figure 4-1

[RA-1 Ch04] Upload latest version of Figure 4-1

* [RA-1 Ch04] Re upload of new Figure 4-1

[RA-1 Ch04] Re upload of new Figure 4-1

* [RA-1 Ch04] Delete Figure_4_1_OpenStack_Network_Layout_20200110

[RA-1 Ch04] Delete Figure_4_1_OpenStack_Network_Layout_20200110 -- will be updated with modified version

* [RA-1 Ch04] Upload corrected version of Figure 4-1

* [WIP] [RI Ch07] Update Deployment Cookbook (#817)

* Topology Diagram

Description of lab topology, aligning with OPNFV Pharos.

Signed-off-by: beierl <mbeierl@vmware.com>

#406

* [RI1 Ch06]i Topology Diagram

Description of lab topology, aligning with OPNFV Pharos.

Signed-off-by: beierl <mbeierl@vmware.com>

* [WIP] [RI Ch07] Update Deployment Cookbook 

#797

* Create path for Airship vs. future installers

* Adds requirements

Added section on hardware used and information about how
to get VPN access and what routes are available.

* Adds requirements

Added section on hardware used and information about how
to get VPN access and what routes are available.

* Revert "Adds requirements"

This reverts commit b25e762.

* Changes to Ch 07 only

Cleanup of chapter 6 in this PR so that it does not
interfere with any other PR.

* Changes to Ch 07 only

Cleanup of chapter 6 in this PR so that it does not
interfere with any other PR.

Co-authored-by: Rabi Abdel <45387599+rabiabdel@users.noreply.github.com>

* [RC] One missing word (nit) (#934)

Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>

Co-authored-by: Tom Kivlin <52716470+tomkivlin@users.noreply.github.com>
Co-authored-by: Gergely Csatari <gergely.csatari@nokia.com>
Co-authored-by: Pankaj Goyal <52107136+pgoyal01@users.noreply.github.com>
Co-authored-by: Deepanshu Bhatia <deep23bhatia@gmail.com>
Co-authored-by: Mark Shostak (AT&T) <49962775+markshostak@users.noreply.github.com>
Co-authored-by: Sridhar K N Rao <ngignir@gmail.com>
Co-authored-by: Rabi Abdel <51988225+rabi-abdel@users.noreply.github.com>
Co-authored-by: collivier <ollivier.cedric@gmail.com>
Co-authored-by: Mark Beierl <mbeierl@vmware.com>

doc/ref_arch/kubernetes/chapters/chapter03.md

@@ -51,6 +51,19 @@ These features and capabilities that are described in the software and hardware
 
 > This chapter should describe considerations about container compute services.
 
+The primary interface between the Physical / Virtual Infrastructure and any container-relevant components is the Host Operating System.  This is the OS within which the container runtime exists, and within which the containers run (and therefore, the OS whose kernel is shared by said containers).  This is shown in Figure 3-1 below.
+
+<p align="center"><img src="../figures/ch03_hostOS.png" alt="Kubernetes Host Operating System" Title="Kubernetes Host Operating System" width="65%"/></p>
+<p align="center"><b>Figure 3-1:</b> Kubernetes Host Operating System</p>
+
+The Host OS (as with any OS) consists of two main layers:
+- Kernel space
+- User space
+
+The Kernel is the tightly controlled space that provides an API via system calls to applications running in the user space (which usually have their own southbound interface in an interpreter or libraries).  Key containerisation capabilities such as Control Groups (cgroups) and namespaces are kernel features, and are used and managed by the container runtime in order to provide isolation between the user space processes (of which the container itself is one, and the processes running within the container are also).  The Host OS is key therefore to the overall security posture and must be appropriately secured to ensure processes running in one container cannot escalate their privileges, for example (covered further in [chapter 6](./chapter06.md)).
+
+A key thing to note is that the container runtime itself is also a set of processes that run in user space, and interact with the kernel via system calls as well.  Many diagrams will show containers as running on top of the runtime, or inside the runtime.  More accurately, the containers themselves are simply some processes running within an OS, and the container runtime is another set of processes that are used to manage those containers (pull, run, delete, etc.) and the kernel features required to provide the isolation (cgroups, namespaces, filesystems, etc.).
+
 <a name="3.2.1.1."></a>
 #### 3.2.1.1 Memory management
 

doc/ref_arch/kubernetes/chapters/chapter04.md

@@ -1,7 +1,7 @@
 [<< Back](../../kubernetes)
 
 # 4. Component Level Architecture
-<p align="right"><img src="../figures/bogo_ifo.png" alt="scope" title="Scope" width="35%"/></p>
+<p align="right"><img src="../figures/bogo_sdc.png" alt="scope" title="Scope" width="35%"/></p>
 
 ## Table of Contents
 * [4.1 Introduction](#4.1)
@@ -17,10 +17,32 @@
 <a name="4.1"></a>
 ## 4.1 Introduction
 
+This chapter will describe in detail the Kubernetes Reference Architecture in terms of the functional capabilities and how they relate to the Reference Model requirements, i.e. how the infrastructure profiles are delivered and described.
+
+Figure 4-1 below shows the architectural components that are described in the subsequent sections of this chapter.
+
+<p align="center"><img src="../figures/ch04_k8s_architecture.png" alt="Kubernetes Reference Architecture" Title="Kubernetes Reference Architecture" width="65%"/></p>
+<p align="center"><b>Figure 4-1:</b> Kubernetes Reference Architecture</p>
+
 <a name="4.2"></a>
 ## 4.2 Host OS
 
-> This chapter should describe the requirements for a host os to run the software stack of Reference Architecture 2.
+In order for a Host OS to be compliant with this Reference Architecture it must meet the following requirements:
+- A deb/rpm compatible distribution of Linux (this must be used for the master nodes, and can be used for worker nodes).
+- A version of the Linux kernel that is [compatible with kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#kubeadm-init-workflow-internal-design) - this has been chosen as the baseline because kubeadm is focussed on installing and managing the lifecycle of Kubernetes and nothing else, hence it is easily integrated into higher-level and more complete tooling for the full lifecycle management of the infrastructure, cluster add-ons, etc.
+- Windows Server 2019 (this can be used for worker nodes, but be aware of the [limitations](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#limitations)).
+- In order to support `req.gen.cnt.03` (immutable infrastructure), the Host OS must be disposable, meaning the configuration of the Host OS (and associated infrastructure such as VM or bare metal server) must be consistent - e.g. the system software and configuration of that software must be identical apart from those areas of configuration that must be different such as IP addresses and hostnames.
+- This approach to configuration management supports `req.lcm.gen.01` (automated deployments)
+
+Table 4-1 lists the Linux kernel versions that comply with this Reference Architecture specification.
+
+|OS Family|Version(s)|Notes|
+|---|---|---|
+|Linux|3.10+||
+|Windows|1809 (10.0.17763)|For worker nodes only|
+
+<p align="center"><b>Table 4-1:</b> Compliant OS Kernels</p>
+
 
 <a name="4.3"></a>
 ## 4.3 Kubernetes
@@ -30,25 +52,125 @@
 > * Which optional features are used and which optional API-s are available
 > * Which [alfa or beta features](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) are used
 
+In alignment with the [Kubernetes version support policy](https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions), a Reference Implementation must use one of three latest minor versions (`n-2`) - e.g. if the latest version is 1.17 then the RI must use either 1.17, 1.16 or 1.15. The Kubernetes distribution or product that is used in the RI must be listed in the [Kubernetes Distributions and Platforms document](https://docs.google.com/spreadsheets/d/1LxSqBzjOxfGx3cmtZ4EbB_BGCxT_wlxW_xgHVVa23es/edit#gid=0) and marked (`X`) as conformant for the Kubernetes version that is being used.
+
+This Reference Architecture also specifies:
+
+- Master nodes must run the following Kubernetes control plane services:
+    - kube-apiserver
+    - kube-scheduler
+    - kube-controller-manager
+- Master nodes can also run the etcd service and host the etcd database, however etcd can also be hosted on separate nodes
+- In order to support `req.gen.rsl.01`, `req.gen.rsl.02` and `req.gen.avl.01` a Reference Implementation must:
+    - Consist of either three, five or seven nodes running the etcd service (can be colocated on the master nodes, or can run on separate nodes, but not on worker nodes)
+    - At least one master node per availability zone or fault domain to ensure the high availability and resilience of Kubernetes control plane services
+    - At least one worker node per availability zone or fault domain to ensure the high availability and resilience of workloads managed by Kubernetes
+- Master node services, including etcd, and worker node services (e.g. consumer workloads) must be kept separate - i.e. there must be at least one master node, and at least one worker node
+- Workloads must ***not*** rely on the availability of the master nodes for the successful execution of their functionality (i.e. loss of the master nodes may affect non-functional behaviours such as healing and scaling, but components that are already running will continue to do so without issue)
+- The following kubelet features must be enabled
+    - CPU Manager
+    - Device Plugin
+    - Topology Manager
+
+All kubelet features can be enabled/disabled by using the `feature-gates:` section in the kubelet config file.  e.g.
+```
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+feature-gates:
+  CPUManager: true|false (BETA - default=true)
+  DevicePlugins: true|false (BETA - default=true)
+  TopologyManager: true|false (ALPHA - default=false)
+```
+
 <a name="4.4"></a>
 ## 4.4 Container runtimes
 
-> This chapter should describe which container runtimes are part of the Reference Architecture.
+In order to support `req.inf.com.03`, the chosen runtime must be compliant with the [Kubernetes Container Runtime Interface (CRI)](https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/) and the [Open Container Initiative (OCI) runtime spec](https://github.com/opencontainers/runtime-spec). Examples of container runtimes that are compliant with these specification are (note this is not a complete list and in no particular order):
+- container-d (with CRI plugin enabled, which it is by default)
+- Docker CE (via the dockershim, which is currently built in to the kubelet)
+- CRI-O
+- Frakti
+
+These specifications cover the [full lifecycle of a container](https://github.com/opencontainers/runtime-spec/blob/master/runtime.md#lifecycle) `creating > created > running > stopped` which includes the use of storage that is required during this lifecycle - this is management of the Host OS filesystem by the container runtime. This lifecycle management by the container runtime (when compliant with the above specifications) supports the requirement `req.inf.stg.06` for ephemeral storage for Pods.
+
+> Todo: details and RA2 specifications relating to runtimes in order to meet RM features and requirements from RM chapters 4 and 5.
 
 <a name="4.5"></a>
 ## 4.5 CNI plugins
 
-> This chapter should describe which CNI plugins are part of the Rerefence Architecture.
+> Editors note: The following chapter lists a set of CNI plugins compliant with the Reference Architecture. In future releases the list of CNI plugins should be refined in a way that there is only component selected for each functionality. 
+
+The used CNI multiplexer/metapulgin may be [DANM](https://github.com/nokia/danm)
+as it provides the possibility to use several other CNI plugins (`req.inf.ntw.16`) and provides an API based solution to administer the networks (`req.inf.ntw.10`) from a central point (`req.inf.ntw.11`).<br>
+
+The following table contains a comparision of relevant features and requirements in Multus and DANM.
+
+| Requirement | Support in Multus | Support in DANM |
+|-------------|-------------------|-----------------|
+| `req.inf.ntw.01` | Supported | Supported |
+| `req.inf.ntw.02` | Supported via an other CNI plugin | Supported via an other CNI plugin |
+| `req.inf.ntw.03` | Supported via an other CNI plugin | Supported |
+| `req.inf.ntw.04` | Supported via an other CNI plugin | Supported via an other CNI plugin |
+| `req.inf.ntw.06` | Supported | Supported |
+| `req.inf.ntw.07` | Supported | Supported |
+| `req.inf.ntw.08` | Supported | Supported |
+| `req.inf.ntw.09` | Supported via LCM tools |  Supported via LCM tools |
+| `req.inf.ntw.10` | Not supported | Suported |
+| `req.inf.ntw.11` | Not supported | Partially supported |
+| `req.inf.ntw.14` | Supported via an other CNI plugin | Supported via an other CNI plugin |
+| `req.inf.ntw.15` | Not relevant | Not relevant |
+| `req.inf.ntw.16` | Supported | Supported |
+| Cluster wide IP address management | Not suported | Supported |
+| Service based dicovery of all provisioned interfaces | Not supported | Supported |
+
+ [Calico](https://github.com/projectcalico/cni-plugin) may be used as the CNI what complies with the basic networking assumptions of Kubernetes based on the requirement `req.inf.ntw.02` due to it's capability to handle `NetworkPolicies`, what is missing from [Flannel](https://github.com/coreos/flannel-cni).
+For the network of signalling connections the built in IPVLAN CNI of DANM or the [MACVLAN CNI](https://github.com/containernetworking/plugins/tree/master/plugins/main/macvlan) may be used as these provide NAT-less connectivity (`req.inf.ntw.03`). For the user plane network(s) fullfilling requirement `req.inf.ntw.04` the [User Space CNI](https://github.com/intel/userspace-cni-network-plugin) may be used. The User Space CNI may use VPP or OVS-DPDK as a backend.
+
+> Editors note: The usage SR-IOV in container environments, therefore the inclusion of an SR-IOV CNI plugin and the [SR-IOV Device Plugin](https://github.com/intel/sriov-network-device-plugin) to the architecture are under debate.
 
 <a name="4.6"></a>
 ## 4.5 Storage components
 
-> This chapter should describe the components used to provide storage services by the reference architecture.
+As described in [chapter 3](./chapter03.md), storage in Kubernetes consists of three types of storage:
+1. Ephemeral storage that is used to execute the containers
+    - **Ephemeral storage follows the lifecycle of a container**
+    - See the [Container runtimes](#4.4) section above for more information how this meets the requirement `req.inf.stg.06` for ephemeral storage for Pods
+1. Kubernetes Volumes, which are used to present additional storage to containers
+    - **A Volume follow the lifecycle of a Pod**
+    - This is a native Kubernetes capability and therefore `req.inf.stg.01` is supported by default
+    - This capability also delivers support for `req.inf.stg.06` although depending on the Volume Plugin used there may be additional steps required in order to remove data from disk (not all plugins manage the full lifecycle of the storage mounted using Volumes)
+1. Kubernetes Persistent Volumes, which are a subset of the above whose lifecycle persists beyond the lifetime of a Pod to allow for data persistence
+    - **Persistent Volumes have a lifecycle that is independent of Containers and/or Pods**
+    - This supports the requirement `req.inf.stg.07` for persistent storage for Pods
+
+Volume plugins are used in Kubernetes to allow for the use of a range of backend storage systems. There are two types of Volume plugin:
+1. In-tree
+    - These plugins are built, linked, compiled and shipped with the core Kubernetes binaries
+    - Therefore if a new backend storage system needs adding this is a change to the core Kubernetes code
+1. Out-of-tree
+    - These plugins allow new storage plugins to be created without any changes to the core Kubernetes code
+    - The Container Storage Interface (CSI) is such an out-of-tree plugin and many in-tree drivers are being migrated to use the CSI plugin instead (e.g. the [Cinder CSI plugin](https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-cinder-csi-plugin.md))
+    - In order to support the requirement `req.inf.stg.03` (CSI support), the following feature gates must be enabled:
+      - `CSIDriverRegistry`
+      - `CSINodeInfo`
+    - In addition to these feature gates, a CSI driver must be used (as opposed to an in-tree volume plugin) - a full list of CSI drivers can be found [here](https://kubernetes-csi.github.io/docs/drivers.html)
+    - In order to support ephemeral storage use through a CSI-compatible volume plugin, the `CSIInlineVolume` feature gate must be enabled
+    - In order to support Persistent Volumes through a CSI-compatible volume plugin, the `CSIPersistentVolume` feature gate must be enabled
+
+> Should the following paragraph be moved to the Security chapter?
+
+> In order to support `req.sec.gen.09` and more generally to support automation and the separation of concerns between providers of a service and consumers of the service, Kubernetes Storage Classes should be used. Storage Classes allow a consumer of the Kubernetes platform to request Persistent Storage using a Persistent Volume Claim and for a Persistent Volume to be dynamically created based on the "class" that has been requested. This avoids having to grant `create`/`update`/`delete` permissions in RBAC to PersistentVolume resources, which are cluster-scoped rather than namespace-scoped (meaning an identity can manage all PVs or none).
+
+A note on object storage:
+- This Reference Architecture does not include any specifications for object storage, as this is neither a native Kubernetes object, nor something that is required by CSI drivers.  Object storage is an application-level requirement that would ordinarily be provided by a highly scalable service offering rather than being something an individual Kubernetes cluster could offer.
+
+> Todo: specifications/commentary to support req.inf.stg.04 (SDS) and req.inf.stg.05 (high performance and horizontally scalable storage). Also req.sec.gen.06 (storage resource isolation), req.sec.gen.10 (CIS - if applicable) and req.sec.zon.03 (data encryption at rest).
+
 
 <a name="4.7"></a>
 ## 4.7 Service meshes
 
-> This chapter should describe which service meshes are part of the Reference Architecture. For the shake of simplcity this chapter should discuss both the "normal" service meshes and Network Service Mesh.
+No service meshes are part of the architecture.
 
 <a name="4.8"></a>
 ## 4.8 Kubernetes Application package manager