feat(platform): support create superedge edge cluster (tkestack#1994)

Signed-off-by: 00pf00 <feeli202012@gmail.com>
00pf00 · Jul 12, 2022 · b762a29 · b762a29
1 parent c260f64
commit b762a29
Show file tree

Hide file tree

Showing 5 changed files with 254 additions and 206 deletions.
diff --git a/docs/design-proposals/edge-cluster.md b/docs/design-proposals/edge-cluster.md
@@ -1,6 +1,6 @@
-# 支持TKE创建边缘集群
+# 支持TKE创建SuperEdge边缘集群
 
-**Author**: [attlee wang](https://github.com/attlee-wang) 
+**Author**: [attlee wang](https://github.com/attlee-wang)
 
 **Status** (20220127): Done
 
@@ -59,29 +59,129 @@
 ```http
     CreateHandlers: []clusterprovider.Handler{
             ## TKEStack 创建TKEStack的逻辑, step按需要引用, 期间可能对部分函数会改写
-            p.EnsureCopyFiles,             
+            p.EnsureCopyFiles,
 			p.EnsurePreClusterInstallHook,
 			p.EnsurePreInstallHook,
 			...
+            // 准备创建SuperEdge边缘节点的逻辑
+            EnsurePrepareEgdeCluster
+            // Add-on SuperEdge 组件的逻辑
+            EnsureApplyEdgeApps
+            ...
 			p.EnsurePostInstallHook,
 			p.EnsurePostClusterInstallHook,
-			
-			// Addon SuperEdge 组件的逻辑
-			....
-			// 准备添加边缘节点的逻辑
 	},
 ```
 
-## 4. Plan
+## 4. 使用Demo
+
+### 4.1 创建边缘集群
+
+创建SuperEdge的边缘集群的demo superedge_edge.yaml具体如下：
+
+```yaml
+apiVersion: platform.tkestack.io/v1
+kind: Cluster
+metadata:
+  annotations:
+    superedge.io/edgeImageResository: superedge.tencentcloudcr.com/superedge ## 边缘节点上镜像仓库的地址
+    superedge.io/edge-version: v0.8.0 ## superedge 组件的版本
+    superedge.io/edge-virtual-addr: 169.254.20.11 ##边缘节点虚拟网卡的地址
+  name: attlee-superedge
+spec:
+  displayName: attlee-superedge-demo
+  etcd:
+    local:
+      dataDir: ""
+      serverCertSANs:
+        - etcd
+        - etcd.kube-system
+  features:
+    containerRuntime: docker
+    skipConditions:
+      - EnsureCilium
+  machines:
+    - ip: 10.0.200.98
+      password: PasswordBase64
+      port: 22
+      username: root
+  networkDevice: eth0
+  clusterCIDR: 192.168.0.0/16
+  properties:
+    maxClusterServiceNum: 256
+    maxNodePodNum: 256
+  publicAlternativeNames:
+    - 106.52.199.103
+  tenantID: default
+  type: Edge                           ## 集群类型，标识创建的是SuperEdge的边缘集群
+  version: 1.20.6-tke.2
+```
+
+更多参数的详细解析请参考TKEStack的官方文档，用`kubectl apply -f superedge_edge.yaml ` 提交集群yaml，集群处于`Running`表示创建SuperEdge 边缘集群成功，
+
+```powershell
+[root@attlee]# kubectl apply -f superedge_edge.yaml      ## 创建superedge edge边缘集群
+cluster.platform.tkestack.io/cls-tptvzrdk created
+
+[root@attlee]# kubectl get cluster                       ## 查看边缘集群的状态
+NAME           TYPE        VERSION        STATUS    AGE
+cls-tptvzrdk   Edge        1.21.4-tke.1   Running   8m   ## 处于Running表示创建边缘集群成功
+```
+
+失败可`kubectl get cluster cls-tptvzrdk -o yaml   `查看边缘集群创建失败细节。
+
+### 4.2 添加边缘节点
+
+**<1>.下载静态安装包**
+
+至[SuperEdge](https://github.com/superedge/superedge)社区在需要加入的节点上下载自己需要的K8s 静态安装包：
+
+> 注意修改"arch=amd64"参数，目前支持[amd64, arm64], kubernetes版本可选[1.20.6, 1.18.2], 其他参数不变
+
+```powershell
+arch=amd64 version=v0.7.0 kubernetesVersion=1.20.6 && rm -rf edgeadm-linux-* && wget https://superedge-1253687700.cos.ap-guangzhou.myqcloud.com/$version/$arch/edgeadm-linux-$arch-$version-k8s-$kubernetesVersion.tgz && tar -xzvf edgeadm-linux-* && cd edgeadm-linux-$arch-$version-k8s-$kubernetesVersion && ./edgeadm
+```
+
+**<2>. Join边缘节点**
+
+在自己新创建的SuperEdge边缘集群的Master执行如下命令获取加入边缘节点的Token：
+
+```powershell
+[root@attlee]# kubeadm token create --print-join-command
+kubeadm join 127.0.0.1:6443 --token n6hdw6.jbr9778iqoebekyv --discovery-token-ca-cert-hash sha256:b1a00ff33fba4f9a6e63ebbee777120bfbe252a27eeb86d78eaa44799c7a9415
+```
+
+> - kubeadm命令master默认已安装；
+>   - 127.0.0.1:6443 为访问边缘集群kube-api的地址，加入边缘节点可换成`<Master节点外网IP/Master节点内网IP/域名>`，端口、token、discovery-token-ca-cert-hash
+      sha256 不变；
+
+执行如下命令加入边缘节点：
+
+```powershell
+[root@attlee]# ./edgeadm join <Master节点外网IP/Master节点内网IP/域名>:Port --token xxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxx --install-pkg-path <edgeadm kube-*静态安装包路径> --enable-edge=true
+```
+
+> 注意通过 `--install-pkg-path` 带edgeadm join的kube-*静态安装包径路
+
+比如：
+
+```powershell
+[root@attlee]# ./edgeadm join 106.52.199.103:6443 --token n6hdw6.jbr9778iqoebekyv --discovery-token-ca-cert-hash sha256:b1a00ff33fba4f9a6e63ebbee777120bfbe252a27eeb86d78eaa44799c7a9415 --install-pkg-path ./kube-linux-amd64-v1.20.6.tar.gz
+```
+
+更好资料可产看[Addon SuperEdge 让原生Kuberntes集群具备边缘能力](https://github.com/superedge/superedge/blob/main/docs/installation/addon_superedge_CN.md)
+。
+
+## 5. Plan
 
 |    时间    | 关键节点                                             |   相关人员   |  进度  |
 | :--------: | ---------------------------------------------------- | :----------: | :----: |
 | 2022-01-27 | 输出Proposals, review 方案                           | @attlee-wang | 已完成 |
-| 2022-02-11 | 输出提交代码框架，分出steps，创建tasks               | @attlee-wang |        |
-| 2022-02-18 | 边缘独立集群能够创建出来                             |              |        |
-| 2022-02-25 | 边缘节点能够添加                                     |              |        |
-| 2022-03-04 | 能够把SuperEdge 打入TKEStack的Releases包             |              |        |
-| 2022-03-11 | 提供边缘集群的部署使用方式，补充入TKEStack的使用文档 |              |        |
+| 2022-02-11 | 输出提交代码框架，分出steps，创建tasks               | @attlee-wang | 已完成 |
+| 2022-02-18 | 边缘独立集群能够创建出来                             | @attlee-wang | 已完成 |
+| 2022-02-25 | 边缘节点能够添加                                     | @attlee-wang | 已完成 |
+|            | 能够把SuperEdge 打入TKEStack的Releases包             |              |        |
+|            | 提供边缘集群的部署使用方式，补充入TKEStack的使用文档 |              |        |
 |            | 添加必要的单元测试和e2e测试                          |              |        |
 |            | 支持TKEStack 创建边缘集群页面                        |              |        |
 
diff --git a/go.mod b/go.mod
@@ -7,9 +7,34 @@ replace (
 	github.com/chartmuseum/storage => github.com/leoryu/chartmuseum-storage v0.11.1-0.20211104032734-9da39e8f5170
 	github.com/deislabs/oras => github.com/deislabs/oras v0.8.0
 	google.golang.org/grpc => google.golang.org/grpc v1.38.0
+	k8s.io/api => k8s.io/api v0.22.3
+	k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.22.3
+	k8s.io/apimachinery => k8s.io/apimachinery v0.22.3
 	// this replace will be removed if https://github.com/kubernetes/kubernetes/pull/104920 is merged in 1.22
 	k8s.io/apiserver => github.com/leoryu/k8s-apiserver v0.22.4-0.20211110063743-0341ac1e5801
+	k8s.io/cli-runtime => k8s.io/cli-runtime v0.22.3
 	k8s.io/client-go => k8s.io/client-go v0.22.3
+	k8s.io/cloud-provider => k8s.io/cloud-provider v0.22.3
+	k8s.io/cluster-bootstrap => k8s.io/cluster-bootstrap v0.22.3
+	k8s.io/code-generator => k8s.io/code-generator v0.22.3
+	k8s.io/component-base => k8s.io/component-base v0.22.3
+	k8s.io/component-helpers => k8s.io/component-helpers v0.22.3
+	k8s.io/controller-manager => k8s.io/controller-manager v0.22.3
+	k8s.io/cri-api => k8s.io/cri-api v0.22.3
+	k8s.io/csi-translation-lib => k8s.io/csi-translation-lib v0.22.3
+	k8s.io/kube-aggregator => k8s.io/kube-aggregator v0.22.3
+	k8s.io/kube-controller-manager => k8s.io/kube-controller-manager v0.22.3
+	k8s.io/kube-proxy => k8s.io/kube-proxy v0.22.3
+	k8s.io/kube-scheduler => k8s.io/kube-scheduler v0.22.3
+	k8s.io/kubectl => k8s.io/kubectl v0.22.3
+	k8s.io/kubelet => k8s.io/kubelet v0.22.3
+	k8s.io/legacy-cloud-providers => k8s.io/legacy-cloud-providers v0.22.3
+	k8s.io/metrics => k8s.io/metrics v0.22.3
+	k8s.io/mount-utils => k8s.io/mount-utils v0.22.3
+	k8s.io/pod-security-admission => k8s.io/pod-security-admission v0.22.3
+	k8s.io/sample-apiserver => k8s.io/sample-apiserver v0.22.3
+	k8s.io/sample-cli-plugin => k8s.io/sample-cli-plugin v0.22.3
+	k8s.io/sample-controller => k8s.io/sample-controller v0.22.3
 )
 
 require (
@@ -69,6 +94,7 @@ require (
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.8.1
 	github.com/stretchr/testify v1.7.0
+	github.com/superedge/edgeadm v0.8.0-tkestack
 	github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/clb v1.0.194
 	github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common v1.0.194
 	github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/cvm v1.0.194
@@ -77,7 +103,7 @@ require (
 	go.etcd.io/etcd/client/v3 v3.5.0
 	go.uber.org/zap v1.17.0
 	golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a
-	golang.org/x/net v0.0.0-20210903162142-ad29c8ab022f
+	golang.org/x/net v0.0.0-20211209124913-491a49abca63
 	golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
 	golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
 	google.golang.org/grpc v1.40.0
@@ -103,8 +129,9 @@ require (
 	k8s.io/component-base v0.22.3
 	k8s.io/klog v1.0.0
 	k8s.io/kube-aggregator v0.22.3
-	k8s.io/kube-openapi v0.0.0-20210421082810-95288971da7e
+	k8s.io/kube-openapi v0.0.0-20211109043538-20434351676c
 	k8s.io/kubectl v0.22.3
+	k8s.io/kubernetes v1.22.6
 	k8s.io/metrics v0.22.3
 	k8s.io/utils v0.0.0-20210819203725-bdf08cb9a70a
 	sigs.k8s.io/controller-runtime v0.9.3