Kubernetes CSI

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod.rst

@@ -4,6 +4,118 @@
 Kubernetes特权Pod
 ====================
 
+Kubernetes特权Pod (privileged Pod) 是一种特殊运行Pod:
+
+- 运行在 ``privileged`` 模式下的容器，能够完全访问物理节点内核(full access to the node's kernel)
+- 可以仔细地通过屏蔽掉特定能力来授权以限制容器的特权
+- 通过定义一些安全相关特性，例如 ``runAsUser`` / ``RunAsNonRoot`` 等
+
+准备工作
+=========
+
+要运行 privileged pod，只需要在容器配置的 ``securityContext`` 部分设置 ``privileged: true`` :
+
+创建 privileged pod
+---------------------
+
+.. literalinclude:: k8s_privileged_pod/privileged-pod-1.yaml
+   :language: yaml
+   :caption: 一个特权pod案例
+
+- 创建测试pod:
+
+.. literalinclude:: k8s_privileged_pod/create_privileged-pod
+   :caption: 创建测试 privilege pod
+
+- 当上述测试pod运行正常后，进入该pod:
+
+.. literalinclude:: k8s_privileged_pod/exec_privileged-pod
+   :caption: 进入privileged pod
+
+- 然后在这个 ``privileged`` pod中检查容器能力:
+
+.. literalinclude:: k8s_privileged_pod/capsh_privileged-pod
+   :caption: 在容器内部检查该容器能力
+
+输出可以看到:
+
+.. literalinclude:: k8s_privileged_pod/capsh_privileged-pod_output
+   :caption: 在容器内部检查该容器能力，输出信息
+
+创建 non-privileged pod
+--------------------------
+
+.. note::
+
+   虽然配置 ``privileged: false`` 和 ``allowPrivilegedEscalation: false`` ，但是实际上 pod 还是会有一些privileges的
+
+- 创建 ``non-privileged`` pod:
+
+.. literalinclude:: k8s_privileged_pod/privileged-pod-2.yaml
+   :language: yaml
+   :caption: 一个 ``non-privileged`` pod案例
+
+- 运行:
+
+.. literalinclude:: k8s_privileged_pod/create_non-privileged-pod
+   :caption: 创建测试 privilege pod
+
+- 然后在这个 ``non-privileged`` pod中检查容器能力:
+
+.. literalinclude:: k8s_privileged_pod/capsh_privileged-pod
+   :caption: 在容器内部检查该容器能力( ``non-privileged``  )
+
+则输出内容明显降低了能力:
+
+.. literalinclude:: k8s_privileged_pod/capsh_non-privileged-pod_output
+   :caption: 在 ``non-privileged`` pod容器内部检查该容器能力，输出信息
+
+创建完全drop privileged的pod
+------------------------------
+
+最严格的是 ``drop: ALL`` 的 ``non-privileged`` pod:
+
+- 创建 ``drop ALL`` 的 ``non-privileged`` pod
+
+.. literalinclude:: k8s_privileged_pod/privileged-pod-3.yaml
+   :language: yaml
+   :caption: 一个 drop ALL 的 ``non-privileged`` pod案例
+
+- 运行:
+
+.. literalinclude:: k8s_privileged_pod/create_drop_all_non-privileged-pod
+   :caption: 创建测试 drop ALL的 ``non-privilege`` pod
+
+在这个 drop ALL的 non-privileged pod中，可以看到没有任何能力
+
+.. literalinclude:: k8s_privileged_pod/capsh_drop_all_non-privileged-pod_output
+   :caption: 在 drop ALL 的  ``non-privileged`` pod容器内部检查该容器能力，输出信息
+
+此时容器中不能安装rpm包，不能删除文件
+
+创建以特定非Root用户运行 non-privileged pod
+----------------------------------------------
+
+- 以 1000 uid 运行的容器配置:
+
+.. literalinclude:: k8s_privileged_pod/privileged-pod-4.yaml
+   :language: yaml
+   :caption: 一个特定用户的 drop ALL ``non-privileged`` pod案例
+
+创建特定能力的特定非Root用户运行 non-privileged pod
+--------------------------------------------------------
+
+进一步，可以给容器一些特定的权限，例如允许调整进程的nice值:
+
+.. literalinclude:: k8s_privileged_pod/privileged-pod-5.yaml
+   :language: yaml
+   :caption: 一个特定能力的用户 drop ALL ``non-privileged`` pod案例
+
+.. note::
+
+   总之，调整比较细节，可以进一步参考原文 `Kubernetes Privileged Pod Practical Examples <https://www.golinuxcloud.com/kubernetes-privileged-pod-examples/>`_
+
+
 参考
 ======
 

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/capsh_drop_all_non-privileged-pod_output

@@ -0,0 +1,11 @@
+Current: =
+Bounding set =
+Ambient set =
+Securebits: 00/0x0/1'b0
+ secure-noroot: no (unlocked)
+ secure-no-suid-fixup: no (unlocked)
+ secure-keep-caps: no (unlocked)
+ secure-no-ambient-raise: no (unlocked)
+uid=0(root)
+gid=0(root)
+groups=0(root)

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/capsh_non-privileged-pod_output

@@ -0,0 +1,11 @@
+Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+ep
+Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
+Ambient set =
+Securebits: 00/0x0/1'b0
+ secure-noroot: no (unlocked)
+ secure-no-suid-fixup: no (unlocked)
+ secure-keep-caps: no (unlocked)
+ secure-no-ambient-raise: no (unlocked)
+uid=0(root)
+gid=0(root)
+groups=0(root)

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/capsh_privileged-pod

@@ -0,0 +1 @@
+capsh --print

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/capsh_privileged-pod_output

@@ -0,0 +1,11 @@
+Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,38,39,40+ep
+Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,38,39,40
+Ambient set =
+Securebits: 00/0x0/1'b0
+ secure-noroot: no (unlocked)
+ secure-no-suid-fixup: no (unlocked)
+ secure-keep-caps: no (unlocked)
+ secure-no-ambient-raise: no (unlocked)
+uid=0(root)
+gid=0(root)
+groups=0(root)

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/create_drop_all_non-privileged-pod

@@ -0,0 +1 @@
+kubectl create -f privileged-pod-3.yaml

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/create_non-privileged-pod

@@ -0,0 +1 @@
+ kubectl create -f privileged-pod-2.yaml

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/create_privileged-pod

@@ -0,0 +1 @@
+kubectl create -f privileged-pod-1.yaml

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/exec_privileged-pod

@@ -0,0 +1 @@
+kubectl exec -it test-pod-1 -- bash

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/privileged-pod-1.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-pod-1
+  namespace: default
+spec:
+  containers:
+  - name: centos
+    image: centos
+    command: ['sh', '-c', 'sleep 999']
+    securityContext:
+       privileged: true

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/privileged-pod-2.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-pod-2
+  namespace: default
+spec:
+  containers:
+  - name: centos
+    image: centos
+    command: ['sh', '-c', 'sleep 999']
+    securityContext:
+       privileged: false
+       allowPrivilegeEscalation: false

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/privileged-pod-3.yaml

@@ -0,0 +1,17 @@
+
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-pod-3
+  namespace: default
+spec:
+  containers:
+  - name: centos
+    image: centos
+    command: ['sh', '-c', 'sleep 999']
+    securityContext:
+       privileged: false
+       allowPrivilegeEscalation: false
+       capabilities:
+          drop:
+            - ALL

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/privileged-pod-4.yaml

@@ -0,0 +1,17 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-pod-4
+  namespace: default
+spec:
+  containers:
+  - name: centos
+    image: centos
+    command: ['sh', '-c', 'sleep 999']
+    securityContext:
+       privileged: false
+       allowPrivilegeEscalation: false
+       runAsUser: 1000
+       capabilities:
+          drop:
+            - ALL

source/kubernetes/deploy/deploy_pod/k8s_privileged_pod/privileged-pod-5.yaml

@@ -0,0 +1,19 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: test-pod-5
+  namespace: default
+spec:
+  containers:
+  - name: centos
+    image: centos
+    command: ['sh', '-c', 'sleep 999']
+    securityContext:
+       privileged: false
+       allowPrivilegeEscalation: false
+       runAsUser: 1000
+       capabilities:
+          drop:
+            - ALL
+          add:
+             - SYS_NICE

source/kubernetes/storage/k8s_csi/alicloud-csi/alicloud_csi_arch.rst

@@ -0,0 +1,53 @@
+.. _alicloud_csi_arch:
+
+======================
+阿里云CSI架构
+======================
+
+阿里云ACK(定制Kubernetes)容器存储，是通过CSI(容器存储接口)访问:
+
+- 块存储: Local Disk(数据库,Hadoop) / Cloud Disk(有状态应用,数据库)
+- 文件存储: NAS(日志,共享数据) / CPFS(HPC, :ref:`deep_learning` )
+- 对象存储: OSS(多媒体,基因数据)
+- 本地存储: LVM(数据库,文件存储) / QuotaPath (数据库) / AEP(数据库)
+
+.. figure:: ../../../../_static/kubernetes/storage/k8s_csi/alicloud-csi/ack_csi.png
+
+   阿里云ACK容器存储概览
+
+阿里云的ACK集群分为托管集群(管控节点由阿里云维护) 和 自建集群(独立部署管控节点)，在部署CSI存储方式有所区别:
+
+- 托管集群 CSI插件 ``CSI-Provisioner`` 和 ``CSI-Plugin`` 都运行在用户的Worker节点
+
+.. figure:: ../../../../_static/kubernetes/storage/k8s_csi/alicloud-csi/ack_mandate_csi.png
+
+   阿里云托管ACK集群 CSI插件
+
+- 自建集群 ``CSI-Provisioner`` 运行在Master节点，而 ``CSI-Plugin`` 则作为 :ref:`daemonset` 同时运行在 Master 和 Worker 节点
+
+.. figure:: ../../../../_static/kubernetes/storage/k8s_csi/alicloud-csi/ack_deploy_csi.png
+
+   阿里云自建ACK集群 CSI插件
+
+CSI使用授权
+================
+
+CSI插件在数据卷挂载、卸载、创建和删除等操作上需要使用AccessKey授权，也支持RAM角色授权(默认)
+
+部署安装CSI-Plugin和CSI-Provisioner
+======================================
+
+阿里云在ACK的托管或自建集群，控制台默认安装 ``CSI-Plugin`` 和 ``CSI-Provisioner`` 组件:
+
+.. literalinclude:: alicloud_csi_arch/ack_csi
+   :caption: 检查ACK集群是否安装好 ``csi-plugin`` 和 ``csi-provisioner``
+
+CSI插件安装
+==============
+
+``kubelet`` 运行参数 ``--enable-controller-attach-detach=true`` 表示集群使用了CSI存储插件，如果这个参数值是 ``false`` 则表明存储插件是 ``Flexvolume``
+
+参考
+=====
+
+- `阿里云容器服务Kubernetes版ACK > 存储CSI概述 <https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/csi-overview-1>`_

source/kubernetes/storage/k8s_csi/alicloud-csi/alicloud_csi_arch/ack_csi

@@ -0,0 +1,2 @@
+kubectl get pod -n kube-system | grep csi-plugin
+kubectl get pod -n kube-system | grep csi-provisioner

source/kubernetes/storage/k8s_csi/alicloud-csi/index.rst

@@ -0,0 +1,17 @@
+.. _alicloud-csi:
+
+============================================================
+阿里云 CSI
+============================================================
+
+.. toctree::
+   :maxdepth: 1
+
+   alicloud_csi_arch.rst
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`

source/kubernetes/storage/k8s_csi/ceph-csi/ceph_csi_arch.rst

@@ -0,0 +1,28 @@
+.. _ceph_csi_arch:
+
+=====================
+Ceph CSI架构
+=====================
+
+`CSI driver for Ceph <https://github.com/ceph/ceph-csi>`_ 提供了 :ref:`ceph_rbd` , :ref:`cephfs` 的驱动，以及Kubernetes :ref:`sidecar` 部署YAML来支持CSI功能:
+
+- provisioner
+- attacher
+- resizer
+- driver-register
+- snapshotter
+
+概览
+=======
+
+Ceph CSI plugins 实现了一个 CSI-enabled 容器编排(Container Orchestrator, CO) 和 Ceph 集群之间的接口，可以动态提供Ceph 卷并将卷添加到 :ref:`k8s_workloads` :
+
+- 支持 :ref:`ceph_rbd` 和 :ref:`cephfs` 后端卷:
+
+  - :ref:`ceph_csi_rbd`
+  - :ref:`ceph_csi_cephfs`
+
+参考
+==========
+
+- `CSI driver for Ceph <https://github.com/ceph/ceph-csi>`_

source/kubernetes/storage/k8s_csi/ceph-csi/ceph_csi_cephfs.rst

@@ -0,0 +1,12 @@
+.. _ceph_csi_cephfs:
+
+========================
+Ceph CSI CephFS plugin
+========================
+
+Ceph CSI CephFS plugin 可以提供新的 CephFS 卷，将CephFS卷附加和挂载到 :ref:`k8s_workloads`
+
+参考
+======
+
+- `ceph-csi/docs/deploy-cephfs.md <https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-cephfs.md>`_

source/kubernetes/storage/k8s_csi/ceph-csi/ceph_csi_rbd.rst

@@ -0,0 +1,26 @@
+.. _ceph_csi_rbd:
+
+====================
+Ceph CSI RBD Plugin
+====================
+
+Ceph CSI RBD plugin 可以提供一个 :ref:`ceph_rbd` 镜像，并将其附加和挂载到 :ref:`k8s_workloads` :
+
+在Kubernetes中部署CSI RBD
+============================
+
+`CSI driver for Ceph <https://github.com/ceph/ceph-csi>`_ (GitHub仓库)提供部署的模版文件(源代码 ``deploy/rbd/kubernetes`` 目录下)，可以帮助我们在Kubernetes中部署:
+
+- csi-config-map.yaml
+- csidriver.yaml
+- csi-nodeplugin-rbac.yaml
+- csi-provisioner-rbac.yaml
+- csi-rbdplugin-provisioner.yaml
+- csi-rbdplugin.yaml
+
+Kubernetes集群需要允许运行 :ref:`k8s_privileged_pod`
+
+参考
+======
+
+- `ceph-csi/docs/deploy-rbd.md <https://github.com/ceph/ceph-csi/blob/devel/docs/deploy-rbd.md>`_