Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/publish-pdf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ jobs:
- name: Create PDF release
run: |
echo $GITHUB_RUN_NUMBER > version.txt
mv main.pdf cnf-best-practices-guide.pdf
gh release create v2.0.$GITHUB_RUN_NUMBER cnf-best-practices-guide.pdf
mv main.pdf k8s-best-practices-guide.pdf
gh release create v2.0.$GITHUB_RUN_NUMBER k8s-best-practices-guide.pdf

env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
8 changes: 4 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
index.html
cnf-best-practices.sublime-project
cnf-best-practices.sublime-workspace
k8s-best-practices.sublime-project
k8s-best-practices.sublime-workspace
main.pdf
cnf-best-practices.pdf
verizon-cnf-best-practices.pdf
k8s-best-practices.pdf
verizon-k8s-best-practices.pdf
pdf-assets
assets
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ This document provides guidance and information for developing OpenShift applica

New releases, Github-pages deployments (HTML versions), PDF versions are triggered automatically upon commit to the main branch.

It is therefore recommended to wait for the workflow actions (https://github.com/test-network-function/cnf-best-practices-guide/actions) to finish and auto generate a new release with the latest commits, then edit the release notes and version tag manually if so desired (instead of generating a manual release, which may cause issues preventing auto generation of the PDF/HTML files).
It is therefore recommended to wait for the workflow actions (https://github.com/test-network-function/k8s-best-practices-guide/actions) to finish and auto generate a new release with the latest commits, then edit the release notes and version tag manually if so desired (instead of generating a manual release, which may cause issues preventing auto generation of the PDF/HTML files).

The latest release will be auto published (via Github-pages bot) to https://test-network-function.github.io/cnf-best-practices-guide
The latest release will be auto published (via Github-pages bot) to https://test-network-function.github.io/k8s-best-practices-guide

The latest (and previous) PDF versions can be downloaded from: https://github.com/test-network-function/cnf-best-practices-guide/releases
The latest (and previous) PDF versions can be downloaded from: https://github.com/test-network-function/k8s-best-practices-guide/releases


Contribution/PR Review Process: WIP
Expand Down
4 changes: 2 additions & 2 deletions main.adoc
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
[id="cnf-best-practices"]
[id="k8s-best-practices"]
= Red Hat Best Practices for Kubernetes
Workload best practices
:revnumber: v2
:revdate: 2023
:toc: left
:toclevels: 5
:repo-url: https://github.com/test-network-function/cnf-best-practices
:repo-url: https://github.com/test-network-function/k8s-best-practices-guide
:icons: image
:icontype: svg
:iconsdir: pdf/icons
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-affinity-anti-affinity"]
[id="k8s-best-practices-affinity-anti-affinity"]
= Affinity and anti-affinity

In OpenShift Container Platform pod affinity and pod anti-affinity allow you to constrain which nodes your pod are eligible to be scheduled based on the key/value labels on other pods. There are two types of affinity rules, required and preferred. Required rules must be met, whereas preferred rules are best effort.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-analyzing-your-application"]
[id="k8s-best-practices-analyzing-your-application"]
= Analyzing your application

To find out which capabilities the application needs, Red Hat has developed a SystemTap script (`container_check.stp`). With this tool, the workload developer can find out what capabilities an application requires in order to run in a container. It also shows the syscalls which were invoked. Find more info at link:https://linuxera.org/capabilities-seccomp-kubernetes/[]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-attaching-the-vf-to-a-pod"]
[id="k8s-best-practices-attaching-the-vf-to-a-pod"]
= Attaching the VF to a pod

Once the right network attachment definition is found, applying the `k8s.v1.cni.cncf.io/networks` annotation with the name of the network attachment definition to the pod will add the additional network interfaces in the pod namespace, as per the following example:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-automount-services-for-pods"]
[id="k8s-best-practices-automount-services-for-pods"]
= Automount services for pods

Pods which do not require API access should set the value of `automountServiceAccountToken` to false within the pod spec, for example:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-avoid-accessing-resource-on-host"]
[id="k8s-best-practices-avoid-accessing-resource-on-host"]
= Avoid accessing resource on host

It is not recommended for an application to access following resources on the host.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-avoid-mounting-host-directories-as-volumes"]
[id="k8s-best-practices-avoid-mounting-host-directories-as-volumes"]
= Avoid mounting host directories as volumes

It is not necessary to mount host `/sys/` or host `/dev/` directories as a volume in a pod in order to use a network device such as SR-IOV VF. The moving of a network interface into the pod network namespace is done automatically by CNI. Mounting the whole `/sys/` or `/dev/` directory in the container will overwrite the network device descriptor inside the container which causes `device not found` or `no such file or directory` error.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-avoid-the-host-network-namespace"]
[id="k8s-best-practices-avoid-the-host-network-namespace"]
= Avoid the host network namespace

Application pods must avoid using `hostNetwork`. Applications may not use the host network, including `nodePort` for network communication. Any networking needs beyond the functions provided by the pod network and ingress/egress proxy must be serviced via a MULTUS connected interface.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-block-storage"]
[id="k8s-best-practices-block-storage"]
= Block storage

OpenShift Container Platform can provision raw block volumes. These volumes do not have a file system, and can provide performance benefits for applications that either write to the disk directly or implement their own storage service.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-ci-cd"]
[id="k8s-best-practices-ci-cd"]
= CI/CD framework

Applications should target a CI/CD approach for deployment and validation.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cloud-native-design-best-practices"]
[id="k8s-best-practices-cloud-native-design-best-practices"]
= Cloud-native design best practices

The following best practices highlight some key principles of cloud-native application design.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cnf-operator-requirements"]
[id="k8s-best-practices-cnf-operator-requirements"]
= Workload Operator requirements

.Workload requirement
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cnf-securing-cnf-networks"]
[id="k8s-best-practices-cnf-securing-cnf-networks"]
= Securing workload networks

Workloads must have the least permissions possible and must implement Network Policies that drop all traffic by default and permit only the relevant ports and protocols to the narrowest ranges of addresses possible.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cnf-security"]
[id="k8s-best-practices-cnf-security"]
= Workload security

In OCP, it is possible to run privileged containers that have all of the root capabilities on a host machine, allowing the ability to access resources which are not accessible in ordinary containers. This, however, increases the security risk to the whole cluster. Containers should only request those privileges they need to run their legitimate functions. No containers will be allowed to run with full privileges without an exception.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cni-ovn"]
[id="k8s-best-practices-cni-ovn"]
= CNI-OVN

OVN is the default pod network CNI plugin for OpenShift and is supported by Red Hat. OVN is Red Hat's CNI for pods. It is a Geneve based overlay that requires L3 reachability between the host nodes. This L3 reachability can be over L2 or a pre-existing overlay network. OpenShift's OVN forwarding is based on flow rules and implemented with `nftables` on the host operating system CNI pod.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-container-runtime"]
[id="k8s-best-practices-container-runtime"]
= Container runtime

OpenShift uses CRI-O as a CRI interface for Kubernetes. CRI-O manages `runC` for container image execution. CRI-O is an open-source container engine that provides a stable, performant platform for running OCI compatible runtimes. CRI-O is developed, tested and released in tandem with Kubernetes major and minor releases.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
:!sectnums:
[id="cnf-best-practices-copyright"]
[id="k8s-best-practices-copyright"]
= Copyright

© Copyright 2023 Red Hat Inc.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cpu-allocation"]
[id="k8s-best-practices-cpu-allocation"]
= CPU allocation

It is important to note that when the OpenShift scheduler is placing pods, it first reviews the Pod CPU request and schedules it if there is a node that meets the requirements. It will then impose the CPU "Limits" to ensure the Pod doesn't consume more than the intended allocation. The limit can never be lower than the request.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cpu-isolation"]
[id="k8-best-practices-cpu-isolation"]
= CPU isolation

The Node Tuning Operator manages host CPUs by dividing them into reserved CPUs for cluster and operating system housekeeping duties, and isolated CPUs for workloads. CPUs that are used for low latency workloads are set as isolated.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cpu-manager-pinning"]
[id="k8s-best-practices-cpu-manager-pinning"]
= CPU manager/pinning

The OpenShift platform can use the Kubernetes CPU Manager to support CPU pinning for applications.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-csi"]
[id="k8s-best-practices-csi"]
= Container storage (CSI)

Pod volumes are supported via local storage and the CSI for persistent volumes. Local storage is truly ephemeral storage, it is local only to the physical node that a pod is running on and is lost when a pod is killed and recreated. If a pod requires persistent storage, the CSI can be used via Kubernetes native primitives `persistentVolume` (`PV`) and `persistentVolumeClaim` (`PVC`) to get persistent storage, such as an NFS share via the CSI backed by NetApp Trident.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-custom-role-to-access-application-crds"]
[id="k8s-best-practices-custom-role-to-access-application-crds"]
= Custom role to access application CRDs

If an application requires installing/deploying CRDs (Custom Resource Definitions), the application must provide a role that allows necessary permissions to create CRs within the CRDs. The custom role to access CRDs must not create any permissions to access any other API resources than the CRDs.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-developer-guide"]
[id="k8s-best-practices-developer-guide"]
= Workload developer guide

This section discusses recommendations and requirements for workload application builders.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-discovering-sr-iov-devices-properties-from-the-application"]
[id="k8s-best-practices-discovering-sr-iov-devices-properties-from-the-application"]
= Discovering SR-IOV devices properties from the application

All the properties of the interfaces are added to the pod's `k8s.v1.cni.cncf.io/network-status` annotation. The annotation is json-formatted and for each network object contains information such as IPs (where available), MAC address, PCI address. For example:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-disruption-budgets"]
[id="k8s-best-practices-disruption-budgets"]
= Disruption budgets

When managing the platform there are at least two types of disruptions that can occur. They are voluntary and involuntary. When dealing with voluntary disruptions a pod disruption budget can be set that determines how many replicas of the application must remain running at any given time. For example, consider the case where an administrator is shutting down a node for
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-ephemeral-storage"]
[id="k8s-best-practices-ephemeral-storage"]
= Ephemeral storage

Pods and containers can require ephemeral or transient local storage for their operation. The lifetime of this ephemeral storage does not extend beyond the life of the individual pod, and this ephemeral storage cannot be shared across pods.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-cloud-native-function-expectations-and-permissions"]
[id="k8s-best-practices-cloud-native-function-expectations-and-permissions"]
= Cloud-native function expectations and permissions

Cloud-native applications are developed as loosely-coupled well-behaved manageable microservices running in containers managed by a container orchestration engine such as kubernetes.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-finding-the-capabilities-that-an-application-needs"]
[id="k8s-best-practices-finding-the-capabilities-that-an-application-needs"]
= Finding the capabilities that an application needs

Here is an example of how to find out the capabilities that an application needs. `testpmd` is a DPDK based layer-2 forwarding application. It needs the `CAP_IPC_LOCK` to allocate the hugepage memory.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-foreword"]
[id="k8s-best-practices-foreword"]
= Foreword

This _Red Hat Best Practices for Kubernetes_ document was originally created in 2020 by Verizon with assistance from Red Hat.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-graceful-termination"]
[id="k8s-best-practices-graceful-termination"]
= Graceful termination

There are different reasons that a pod may need to shutdown on an OpenShift cluster. It might be that the node the pod is running on needs to be shut down for maintenance, or the administrator is doing a rolling update of an application to a new version which requires that the old versions are shutdown properly.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-helm"]
[id="k8s-best-practices-helm"]
= Helm v3

Helm v3 is a serverless mechanism for defining templates that describe a complete Kubernetes application. This allows you to build generic templates for applications that you can use with site or deployment specific values to be provided as inputs to the template.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-high-level-cnf-expectations"]
[id="k8s-best-practices-high-level-cnf-expectations"]
= High-level workload expectations

* Workloads shall be built to be cloud-native
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-host-os"]
[id="k8s-best-practices-host-os"]
= Container host operating system

Red Hat Enterprise Linux CoreOS (RHCOS) is the next generation container operating system. RCHOS is part of OpenShift Container Platform and is used as the operating system for the control plane. it is the default operating system for worker nodes. RHCOS is based on RHEL, has some immutability, leverages the CRI-O runtime, contains container tools, and is updated through the Machine Config Operator (MCO).
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-huge-pages"]
[id="k8s-best-practices-huge-pages"]
= Huge pages

In OpenShift Container Platform, nodes/hosts must pre-allocate huge pages.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-image-tagging"]
[id="k8s-best-practices-image-tagging"]
= Image tagging

An image tag is a label applied to a container image in a repository that distinguishes a specific image from other images. Image tags may be used to categorize images (for example: latest, stable, development) and by versions within the categories. This allows the administrator to be specific when declaring which image to test, or which image to run in production.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-init-containers"]
[id="k8s-best-practices-init-containers"]
= init containers

Init containers can be used for running tools / commands / or any other action that needs to be done before the actual pod is started. For example, loading a database schema, or constructing a config file from a definition passed in via configMap or secret.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-ipv4-&-ipv6"]
[id="k8s-best-practices-ipv4-&-ipv6"]
= IPv4 & IPv6

Applications should discover services via DNS by doing an AAAA and A query. If an application gets a AAAA response the application should prefer using the IPv6 address in the AAAA response for application sockets.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-k8s-api-versions"]
[id="k8s-best-practices-k8s-api-versions"]
= Kubernetes API versions

Review the Kubernetes and OpenShift API documentation:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
[id="cnf-best-practices-k8s"]
[id="k8s-best-practices-k8s"]
= Kubernetes

Kubernetes is an open source container orchestration suite of software that is API driven with a datastore that manages the state of the deployments residing on the cluster.
Expand Down
Loading