k8s storage

huataihuang · Sep 5, 2023 · d5313aa · d5313aa
1 parent 9401104
commit d5313aa
Show file tree

Hide file tree

Showing 28 changed files with 275 additions and 17 deletions.
diff --git a/source/devops/docs/sphinx_doc/sphinx_embed_video.rst b/source/devops/docs/sphinx_doc/sphinx_embed_video.rst
@@ -23,6 +23,11 @@ Sphinx文档嵌入视频
 
 就可以看到我使用 :ref:`yt-dlp` 从YoutTube下载的 **《杀死那个石家庄人》--万能青年旅店 影视混剪MV** (我选择了一个小规格mp4作为演示， **墙裂推荐** 观看YouTube原高清视频 `《杀死那个石家庄人》-- 万能青年旅店 影视混剪MV <https://www.youtube.com/watch?v=npHbCnf-Lpk&list=PLnqzKl0S_xnl8xgGJxWKDPuFrgafqSEfo&index=3>`_ )
 
+.. _kill_that_shijiazhuang_man:
+
+「杀死那个石家庄人」
+======================
+
 .. video:: ../../../_static/devops/docs/sphinx_doc/ssngsjzr.mp4
 
 .. note::

diff --git a/source/kubernetes/concepts/storage/index.rst b/source/kubernetes/concepts/storage/index.rst
@@ -9,6 +9,7 @@ Kubernetes存储概念
 
    k8s_volumes.rst
    k8s_persistent_volumes.rst
+   k8s_dynamic_volume_provisioning.rst
    k8s_pvc_pv_bind.rst
 
 .. only::  subproject and html

diff --git a/source/kubernetes/concepts/storage/k8s_dynamic_volume_provisioning.rst b/source/kubernetes/concepts/storage/k8s_dynamic_volume_provisioning.rst
@@ -0,0 +1,15 @@
+.. _k8s_dynamic_volume_provisioning:
+
+===================================================
+Kubernetes动态卷制备(Dynamic Volume Provisioning)
+===================================================
+
+开源动态卷制备: :ref:`openebs`
+==================================
+
+在 :ref:`k8s_csi` 有多种实现规范(SPEC)，例如 :ref:`ceph-csi` 
+
+参考
+=======
+
+- `Kubernetes Documentation>Concepts>Storage>Dynamic Volume Provisioning <https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/>`_ 对应中文版文档: `Kubernetes 文档>概念>存储>动态卷制备 <https://kubernetes.io/zh-cn/docs/concepts/storage/dynamic-provisioning/>`_
diff --git a/source/kubernetes/debug/debug_k8s_node_notready.rst b/source/kubernetes/debug/debug_k8s_node_notready.rst
@@ -0,0 +1,15 @@
+.. _debug_k8s_node_notready:
+
+=============================
+排查Kubernetes节点NotReady
+=============================
+
+参考
+======
+
+- `How to Fix Kubernetes ‘Node Not Ready’ Error <https://komodor.com/learn/how-to-fix-kubernetes-node-not-ready-error/>`_
+- `Debugging your Kubernetes nodes in the ‘not ready’ state <https://www.airplane.dev/blog/debugging-kubernetes-nodes-in-not-ready-state>`_
+- `How to debug when Kubernetes nodes are in 'Not Ready' state <https://stackoverflow.com/questions/47107117/how-to-debug-when-kubernetes-nodes-are-in-not-ready-state>`_
+- `How to Fix the Kubernetes Node NotReady Error <https://linuxhint.com/fix-kubernetes-node-not-ready-error/>`_
+- `K8s Troubleshooting — Node NotReady <https://medium.com/geekculture/k8s-troubleshooting-node-notready-8b39a026696e>`_
+- `Kubernetes: Node ‘NotReady’ [SOLVED] <https://www.shellhacks.com/kubernetes-node-notready/>`_
diff --git a/...ce/kubernetes/debug/k8s_notready_node.rst → ...debug/debug_k8s_notready_node_kubelet.rst b/...ce/kubernetes/debug/k8s_notready_node.rst → ...debug/debug_k8s_notready_node_kubelet.rst
@@ -1,7 +1,7 @@
-.. _k8s_notready_node:
+.. _debug_k8s_notready_node_kubelet:
 
 ===================================
-Kubernetes节点NotReady排查
+Kubernetes节点NotReady排查(kubelet)
 ===================================
 
 在Kubernetes集群运维时，工作节点 ``NotReady`` 状态是非常常见的故障。通常我们有一些排查思路需要依次执行以获取必要信息。这里我做一些案例分析，提供一些建议。
@@ -143,4 +143,4 @@ kubelet未启动导致NotReady
 
 .. note::
 
-   我注意到 ``kali`` 节点使用的 ``INTERNAL-IP`` 是绑定在无线网卡上，这个无线网卡启动需要复杂认证，启动缓慢。我推测是这个导致kubelet无法正常启动，因为kueblet启动时无线网卡可能尚未就绪。具体原因后续再排查。
+   我注意到 ``kali`` 节点使用的 ``INTERNAL-IP`` 是绑定在无线网卡上，这个无线网卡启动需要复杂认证，启动缓慢。我推测是这个导致kubelet无法正常启动，因为kueblet启动时无线网卡可能尚未就绪。具体原因后续再排查。
diff --git a/source/kubernetes/debug/index.rst b/source/kubernetes/debug/index.rst
@@ -16,7 +16,8 @@ Kubernetes排查
    apiserver_cert_expire.rst
    k8s_crashloopbackoff.rst
    apiserver_port_2379_refuse.rst
-   k8s_notready_node.rst
+   debug_k8s_node_notready.rst
+   debug_k8s_notready_node_kubelet.rst
    k8s_master_pod_create_error.rst
    debug_k8s_restart_pods.rst
    get_pods_restart_time.rst

diff --git a/source/kubernetes/kubeflow/install_kubeflow_single_command.rst b/source/kubernetes/kubeflow/install_kubeflow_single_command.rst
@@ -68,9 +68,16 @@
    :language: yaml
    :caption: ``get pvc authservice-pvc`` 
 
-考虑到简单化部署，我采用 :ref:`zfs` 来实现 NFS 输出
+我最初以为这是一个简单的 :ref:`k8s_pvc_pv_bind` (类似我之前实践过的 :ref:`kube-prometheus-stack_persistent_volume` ) ，想正好实践一下 :ref:`zfs_nfs` 输出为 :ref:`k8s_nfs` 。
 
+但是仔细检查这个 ``authservice-pvc`` 就会发现和 ``pv/pvc`` 的静态配置有所不同: ``authservice-pvc`` 并没有提供 ``storageClassName`` 来对应绑定 ``pv`` 和 ``pvc`` 。也就是说，这里的实现是 :ref:`k8s_dynamic_volume_provisioning` 。
 
+如果我不是在云计算厂商的平台部署(通常云厂商会提供 :ref:`k8s_csi` ，并且只要配置好 :ref:`admission_plugins_DefaultStorageClass` 就能无需指定 ``sc`` storage class直接创建存储pv )，就必须自己部署实现:
+
+  - :ref:`openebs`
+  - :ref:`ceph-csi`
+
+然后通过指定 :ref:`admission_plugins_DefaultStorageClass` 实现为 ``kubeflow mainfest`` 提供 :ref:`k8s_dynamic_volume_provisioning`
 
 
 参考

diff --git a/source/kubernetes/security/admission_plugins.rst b/source/kubernetes/security/admission_plugins.rst
@@ -0,0 +1,37 @@
+.. _admission_plugins:
+
+==========================
+Admission 插件
+==========================
+
+检查默认启用的admission plugins
+==================================
+
+- ``kube-apiserver`` 提供了查询哪些插件是默认启用:
+
+.. literalinclude:: admission_plugins/apiserver_default_admission_plugins
+   :caption: 检查默认启用的Admission Plugins
+
+不过，需要注意的是，默认通过 :ref:`kubespray` 部署的 ``kube-apiserver`` 容器内部没有提供任何 ``sh`` 命令，所以我参考 `How to access kube-apiserver on command line? <https://stackoverflow.com/questions/56542351/how-to-access-kube-apiserver-on-command-line>`_ 想要登陆到容器内部并没有成功。不过，还是可以通过以下命令观察:
+
+.. literalinclude:: admission_plugins/kubectl_apiserver_default_admission_plugins
+   :caption: 通过 ``kubectl`` 运行pod内部的 ``kube-apiserver`` 检查默认启用的Admission Plugins
+
+输出类似(注意：输出实际是一行，我这里为了方便查看做了多行格式化)
+
+.. literalinclude:: admission_plugins/kubectl_apiserver_default_admission_plugins_output
+   :caption: 通过 ``kubectl`` 运行pod内部的 ``kube-apiserver`` 检查默认启用的Admission Plugins 输出
+
+.. _admission_plugins_DefaultStorageClass:
+
+Admission Plugin ``DefaultStorageClass``
+-------------------------------------------
+
+Admission Plugin ``DefaultStorageClass`` 为 **没有请求任何特定存储类** 的 ``PersistentVolumeClaim`` (PVC) 对象的创建请求，自动添加默认存储类。这样用户无需关心存储类型(很多用户也不care)，就可以自动完成配置。
+
+注意，当没有配置默认存储类是，这个Adminssion Controller不执行任何操作。而且，如果有多个存储类被标记为默认存储类，也会导致该控制器拒绝所有创建 PVC 的请求并返回错误。
+
+参考
+=========
+
+- `Using Admission Controllers <https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers>`_
diff --git a/source/kubernetes/security/admission_plugins/apiserver_default_admission_plugins b/source/kubernetes/security/admission_plugins/apiserver_default_admission_plugins
@@ -0,0 +1 @@
+kube-apiserver -h | grep enable-admission-plugins
diff --git a/source/kubernetes/security/admission_plugins/kubectl_apiserver_default_admission_plugins b/source/kubernetes/security/admission_plugins/kubectl_apiserver_default_admission_plugins
@@ -0,0 +1 @@
+kubectl exec -it kube-apiserver-y-k8s-m-1 -n  kube-system -- kube-apiserver -h | grep enable-admission-plugins
diff --git a/.../kubernetes/security/admission_plugins/kubectl_apiserver_default_admission_plugins_output b/.../kubernetes/security/admission_plugins/kubectl_apiserver_default_admission_plugins_output
@@ -0,0 +1,14 @@
+...
+      --enable-admission-plugins strings       admission plugins that should be enabled in addition to default enabled ones 
+      (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, 
+      StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, 
+
+     DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). 
+      Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, 
+      CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, 
+      ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, 
+      NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, 
+      PodNodeSelector, PodSecurity, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, 
+      StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. 
+
+      The order of plugins in this flag does not matter.
diff --git a/source/kubernetes/security/index.rst b/source/kubernetes/security/index.rst
@@ -8,6 +8,7 @@ Kubernetes安全
    :maxdepth: 1
 
    admission_controllers.rst
+   admission_plugins.rst
    mutating_admission_webhook.rst
    validating_admission_webhook.rst
    simple_k8s_admission_webook.rst

diff --git a/source/kubernetes/storage/index.rst b/source/kubernetes/storage/index.rst
@@ -7,6 +7,8 @@ Kubernetes存储
 .. toctree::
    :maxdepth: 1
 
+   k8s_csi/index
+   openebs/index
    k8s_hostpath.rst
    k8s_local.rst
    k8s_nfs.rst

diff --git a/source/kubernetes/storage/k8s_csi/ceph-csi/index.rst b/source/kubernetes/storage/k8s_csi/ceph-csi/index.rst
@@ -0,0 +1,15 @@
+.. _ceph-csi:
+
+============================================================
+Ceph CSI
+============================================================
+
+.. toctree::
+   :maxdepth: 1
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`
diff --git a/source/kubernetes/storage/k8s_csi/index.rst b/source/kubernetes/storage/k8s_csi/index.rst
@@ -0,0 +1,18 @@
+.. _k8s_csi:
+
+============================================================
+Kubernetes 容器存储接口(Container Storage Interface, CSI)
+============================================================
+
+.. toctree::
+   :maxdepth: 1
+
+   k8s_csi_arch.rst
+   ceph-csi/index
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`
diff --git a/source/kubernetes/storage/k8s_csi/k8s_csi_arch.rst b/source/kubernetes/storage/k8s_csi/k8s_csi_arch.rst
@@ -0,0 +1,10 @@
+.. _k8s_csi_arch:
+
+=======================
+Kubernetes CSI架构
+=======================
+
+参考
+=======
+
+- `Kubernetes CSI Specification <https://cctoctofx.netlify.app/post/cloud-computing/k8s-csi-interprete/>`_ 这篇文章较为全面清晰，提供了不少索引信息，可以作为学习起点
diff --git a/source/kubernetes/storage/k8s_rbd.rst b/source/kubernetes/storage/k8s_rbd.rst
@@ -4,3 +4,4 @@
 在Kubernetes中部署RBD存储
 =============================
 
+Kubernetes 的 RBD 存储制备器(Storage Provisioner)就是 :ref:`ceph_rbd` 内部驱动。但是， **Kubernetes v1.28已经废弃了Ceph RBD** ，改为采用 :ref:`ceph-csi` 。所以在最新的Kubernetes部署时，请顺应社区路线，采用标准 :ref:`k8s_csi` 实现 ``Ceph CSI`` 。
diff --git a/source/kubernetes/storage/openebs/cstor/cstor_arch.rst b/source/kubernetes/storage/openebs/cstor/cstor_arch.rst
@@ -0,0 +1,5 @@
+.. _cstor_arch:
+
+================
+cStor 架构
+================
diff --git a/source/kubernetes/storage/openebs/cstor/index.rst b/source/kubernetes/storage/openebs/cstor/index.rst
@@ -0,0 +1,17 @@
+.. _cstor:
+
+============================================================
+cStor
+============================================================
+
+.. toctree::
+   :maxdepth: 1
+
+   cstor_arch.rst
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`
diff --git a/source/kubernetes/storage/openebs/index.rst b/source/kubernetes/storage/openebs/index.rst
@@ -0,0 +1,20 @@
+.. _openebs:
+
+============================================================
+OpenEBS 基于容器的块存储(Container Attached Storage)
+============================================================
+
+.. toctree::
+   :maxdepth: 1
+
+   openebs_arch.rst
+   localpv/index
+   cstor/index
+   mayastor/index
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`
diff --git a/source/kubernetes/storage/openebs/localpv/index.rst b/source/kubernetes/storage/openebs/localpv/index.rst
@@ -0,0 +1,17 @@
+.. _localpv:
+
+============================================================
+LocalPV
+============================================================
+
+.. toctree::
+   :maxdepth: 1
+
+   localpv_arch.rst
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`
diff --git a/source/kubernetes/storage/openebs/localpv/localpv_arch.rst b/source/kubernetes/storage/openebs/localpv/localpv_arch.rst
@@ -0,0 +1,5 @@
+.. _localpv_arch:
+
+======================
+LocalPV 架构
+======================
diff --git a/source/kubernetes/storage/openebs/mayastor/index.rst b/source/kubernetes/storage/openebs/mayastor/index.rst
@@ -0,0 +1,17 @@
+.. _mayastor:
+
+============================================================
+Mayastor
+============================================================
+
+.. toctree::
+   :maxdepth: 1
+
+   mayastor_arch.rst
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`
diff --git a/source/kubernetes/storage/openebs/mayastor/mayastor_arch.rst b/source/kubernetes/storage/openebs/mayastor/mayastor_arch.rst
@@ -0,0 +1,10 @@
+.. _mayastor_arch:
+
+======================
+Mayastor架构
+======================
+
+参考
+=====
+
+- `Mayastor User Reference: Basic Architecture <https://mayastor.gitbook.io/introduction/basic-architecture>`_
diff --git a/source/kubernetes/storage/openebs/openebs_arch.rst b/source/kubernetes/storage/openebs/openebs_arch.rst
@@ -0,0 +1,21 @@
+.. _openebs_arch:
+
+=============================
+OpenEBS架构
+=============================
+
+**OpenEBS** 又称为 ``Container Attached Storage`` (基于容器的块存储):
+
+- OpenEBS遵循微服务架构，本身作为一组容器部署在Kubernetes工作节点上，使用Kubernetes编排管理OpenEBS组件
+- 完全构建于用户空间
+- OpenEBS支持一系列存储引擎(storage driver)，通常使用案例:
+
+  - Cassandra 这样的分布式应用程序可以使用 :ref:`localpv` 引擎实现低延迟写
+  - :ref:`mysql` 和 :ref:`pgsql` 可以使用 :ref:`zfs` 引擎( :ref:`cstor` )进行恢复
+  - :ref:`kafka` 这样的流媒体应用程序可以使用 :ref:`nvme` 引擎 :ref:`mayastor` (OpenEBS开发的NVMe-oF存储)
+
+参考
+=======
+
+- `OpenEBS中文版 README.md <https://github.com/openebs/openebs/blob/main/translations/README.zh.md>`_
+- `OpenEBS动态创建存储 <https://blog.51cto.com/liqingbiao/6051543>`_ 非常详尽的产品介绍，应该是官方文档的clone，我感觉需要多阅读几遍，并结合其包容的技术(例如 :ref:`zfs` :ref:`longhorn` 等)进行思考才能理解这个技术指南 
diff --git a/source/linux/storage/zfs/admin/zfs_startup_zcloud.rst b/source/linux/storage/zfs/admin/zfs_startup_zcloud.rst
@@ -19,6 +19,10 @@ ZFS快速起步(zcloud)
 .. literalinclude:: zfs_startup_zcloud/zpool_create
    :caption: 在磁盘 ``sda`` 上创建ZFS的存储池，名字为 ``zpool-data``
 
+.. note::
+
+   对于数据存储，启用 :ref:`zfs_compression` 节约存储空间
+
 - 检查 ``zpool`` :
 
 .. literalinclude:: zfs_startup_zcloud/zpool_list
@@ -35,19 +39,9 @@ ZFS快速起步(zcloud)
    :caption: zpool命令在完整磁盘上创建存储池的之后，就可以看到GPT分区以及2个ZFS分区
    :emphasize-lines: 10,11
 
-待续...
-
-- 准备在 ``zpool-data`` 下构建一个 ``home`` 卷，挂载到 ``/home`` 目录，这样大多数数据都能够得到有效保存
-
-- 首先以 ``root`` 身份登陆，并确保 ``/home`` 目录没有用户访问，将 ``/home`` 目录重命名:
-
-.. literalinclude:: zfs_startup/rename_home
-   :language: bash
-   :caption: 将/home目录重命名(备份)
-
-- 由于 ``zpool-data`` 存储池已经在 :ref:`mobile_cloud_x86_zfs` 构建好，所以忽略创建 ZFS 存储池步骤，直接创建卷 ``home`` ，并且创建 ``home`` 卷下面的子(用户目录):
+- 由于 ``zpool-data`` 存储池挂载在 ``/zpool-data`` ，所以后续创建的卷，默认都会挂载到这个目录下的子目录:
 
-.. literalinclude:: zfs_startup/zfs_create_volume
+.. literalinclude:: zfs_startup_zcloud/zfs_create_volume
    :language: bash
    :caption: 创建 zpool-data 存储池的 home 卷
 

diff --git a/source/linux/storage/zfs/admin/zfs_startup_zcloud/zfs_create_volume b/source/linux/storage/zfs/admin/zfs_startup_zcloud/zfs_create_volume
@@ -0,0 +1,6 @@
+# 采用一条命令创建挂载好目录的ZFS卷，这个目录卷用于 y-k8s
+zfs create zpool-data/y-k8s
+# 如果要挂载到其他可选目录，例如可以增加 -o mountpoint=/y-k8s
+
+# 为 y-k8s Kubernetes集群的不同用途pv创建子卷，例如这里为 install_kubeflow_single_command 提供PV
+zfs create zpool-data/y-k8s/authservice-pvc
diff --git a/source/linux/storage/zfs/admin/zfs_startup_zcloud/zpool_create b/source/linux/storage/zfs/admin/zfs_startup_zcloud/zpool_create
@@ -1 +1,3 @@
 zpool create zpool-data sda
+# 设置了从zpool的根开始激活压缩
+zfs set compression=lz4 zpool-data