Table of Contents
- Introduction of OpenMCP®
- How To Install
- How To Join Cluster
- 1. How to join Kubernetes Cluster to OpenMCP
- (1) (option) Rename cluster name [In sub-cluster]
- (2) Check Status OpenMCP API Server [In openmcp-cluster]
- (3) Register DNS Server [In sub-cluster]
- (4) Register Region and Zone to All Nodes of sub-cluster [In sub-cluster]
- (5) Register sub-cluster to OpenMCP [In sub-cluster]
- (6) Check Registered OpenMCPCluster [In openmcp-cluster]
- (7) Join sub-cluster to OpenMCP [In openmcp-cluster]
- 2. How to join GKE Cluster to OpenMCP
- 3. How to join EKS Cluster to OpenMCP
- 4. How to join AKS Cluster to OpenMCP
- 1. How to join Kubernetes Cluster to OpenMCP
- OpenMCP EXAMPLE
Container control and management platform that provides flexible service movement and seamless automatic scaling of computing resources through mutual collaboration between locally isolated containers
Before you install OpenMCP, Federation and OpenMCP ExternalServer are required.
- GO v1.14
- Install federation (Version: 0.8.1)
- ExternalServer
- flannel CNI
[ Test Environment ]
OpenMCP Master IP : 10.0.3.30
Cluster1 Master IP : 10.0.3.40
Cluster2 Master IP : 10.0.3.50
Rename cluster name to 'openmcp' in kubeconfig file.
Default directory of kubeconfig file : $HOME/.kube/config
$ vi $HOME/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ...
server: https://10.0.3.30:6443
name: openmcp
contexts:
- context:
cluster: openmcp
user: openmcp-admin
name: openmcp
current-context: openmcp
kind: Config
preferences: {}
users:
- name: openmcp-admin
user:
client-certificate-data: ...
client-key-data: ...$ cd ./install_openmcp
$ vim settings.yaml
default:
# learning or debug
installType: debug
docker:
imagePullSecretName: regcred
imagePullPolicy: Always
master:
internal:
ip: 10.0.3.30
ports:
apiServerPort: 30000
clusterManagerPort: 30001
analyticEnginePort: 30002
metricCollectorPort: 30003
influxDBPort: 30004
public:
ip: 119.65.195.180
ports:
metricCollectorPort: 3212
APIServer:
AppKey: openmcp-apiserver
UserName: openmcp
UserPW: keti
metalLB:
rangeStartIP: 10.0.3.191
rangeEndIP: 10.0.3.200
powerDNS:
apiKey: 1234
internal:
ip: 10.0.3.12
ports:
pdnsPort: 53
public:
ip: 119.65.195.180
ports:
pdnsPort: 5353
$ ./create.shDeploy all init modules for OpenMCP.
$ cd master/
$ ./install.shlist of installed Deployment
- Cluster Manager
- Sync Controller
- Resource Controller (Deployment, HybridAutoScaler, Ingress, Service, Configmap, Secret)
- LoadBalancing Controller
- Scheduler
- Resource Manager (Analytic Engine, Metric Collector)
- Policy Engine
- DNS Controller
- API Server
- InfluxDB
Check state of pods.
$ kubectl get pods -n openmcp
NAME READY STATUS RESTARTS AGE
influxdb-55b9b9d8dd-6qghq 1/1 Running 0 18h
openmcp-analytic-engine-57f98db8fb-m4cxn 1/1 Running 0 18h
openmcp-apiserver-66d4f6d8f6-2k2wz 1/1 Running 0 18h
openmcp-cluster-manager-6959c679b4-6zjzh 1/1 Running 0 18h
openmcp-configmap-controller-668cdd9c87-q6jq8 1/1 Running 0 18h
openmcp-deployment-controller-65c8cddbcb-rd2h5 1/1 Running 0 18h
openmcp-dns-controller-54c6bbcfb7-t6cc4 1/1 Running 0 18h
openmcp-has-controller-d64db44fd-4nc9w 1/1 Running 0 18h
openmcp-ingress-controller-6c54847ddc-fn4kg 1/1 Running 0 18h
openmcp-job-controller-7fcdf79d66-8fqm8 1/1 Running 0 18h
openmcp-loadbalancing-controller-7dfd9dfb4b-frslj 1/1 Running 0 18h
openmcp-metric-collector-65c7775bbc-9crdk 1/1 Running 0 18h
openmcp-namespace-controller-fd6f485cb-5vq6c 1/1 Running 0 18h
openmcp-policy-engine-67986c6679-89jzp 1/1 Running 0 18h
openmcp-scheduler-5d4f48675c-5v55n 1/1 Running 0 18h
openmcp-secret-controller-b48497699-j6s4p 1/1 Running 0 18h
openmcp-service-controller-5f5c4994d6-dtsdf 1/1 Running 0 18h
openmcp-sync-controller-5c5f849496-xn89h 1/1 Running 0 18h
$ kubectl get openmcppolicy -n openmcp
NAME AGE
analytic-metrics-weight 19h
hpa-minmax-distribution-mode 19h
lb-scoring-weight 19h
log-level 19h
metric-collector-period 19h
post-scheduling-type 19h- How to join Kubernetes Cluster to OpenMCP
- How to join GKE Cluster to OpenMCP
- How to join EKS Cluster to OpenMCP
- How to join AKS Cluster to OpenMCP
(0) If the cluster you are trying to join is writing an internal network, you must change the certificate of port forwarding and API Server.
Port Forwarding
6443 -> 6443
15012 -> 32012
15021 -> 32021
15017 -> 32017
15443 -> 32443API Certificate
cd /etc/kubernetes/pki
sudo cp apiserver.crt apiserver.crt.old
sudo cp apiserver.key apiserver.key.old
sudo rm apiserver.crt apiserver.key
sudo kubeadm init phase certs apiserver --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=<EXTERNALIP>
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configKubeConfig Modify
$ vim $HOME/.kube/config
server: https://<EXTERNALIP>:6443Before you join the sub-cluster to OpenMCP, check for duplicate cluster names so it does not overlap.
Default directory of kubeconfig file : $HOME/.kube/config
$ vi $HOME/.kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ...
server: https://10.0.3.40:6443
name: cluster1
contexts:
- context:
cluster: cluster1
user: cluster1-admin
name: cluster1
current-context: cluster1
kind: Config
preferences: {}
users:
- name: cluster1-admin
user:
client-certificate-data: ...
client-key-data: ...$ kubectl get pod,svc -n openmcp
ME READY STATUS RESTARTS AGE
...
pod/openmcp-apiserver-ddf89465f-b77t9 1/1 Running 0 54s
...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
service/openmcp-apiserver LoadBalancer 10.96.65.179 XX.XX.XX.XX 8080:30000/TCP 54sIf you are using the same network, write the [<EXTERNAL_SERVER_IP>](https://github.com/openmcp/external of an external server at the top of the '/etc/resolv.conf' file. Join the OpenMCP cluster through the DNS domain of the API server. If you use a different network, write the authorized APISERVER IP information of the OpenMCP master in /etc/hosts.
# Same Network
$ vim /etc/resolv.conf
nameserver <EXTERNAL_SERVER_IP>
# Different Network
$ vim /etc/hosts
<PUBLIC_APISERVER_IP> openmcp-apiserver.openmcp.default-domain.svc.openmcp.example.orgTag labels(Region, Zone) on sub-cluster.
$ kubectl label nodes <node-name> topology.kubernetes.io/region=<region>
$ kubectl label nodes <node-name> topology.kubernetes.io/zone=<zone>
$ kubectl label nodes <node-name> topology.istio.io/subzone=<cluster-name>Region (ISO 3166-1 alpha-2) https://ko.wikipedia.org/wiki/ISO_3166-1
- KR (Korea)
- US (USA)
- CH (China)
- JP (Japan)
- IN (India)
Zone (locationInfo.csv)
Install 'kubectl request-join' plugin on sub-cluster.
$ git clone https://github.com/openmcp/Member_Plugin.git
$ cd Member_Plugin/kubectl_plugin
$ chmod +x kubectl-request_join
$ cp kubectl-request_join /usr/local/bin
$ kubectl request-join$ kubectl get openmcpcluster -n openmcp
NAME STATUS
cluster1 UNJOIN
cluster2 UNJOIN
Install 'kubectl join' plugin on sub-cluster. Before execute join command, you must set KUBECONFIG file and ~/.hosts file.
$ cd kubectl_plugin
$ chmod +x kubectl-join
$ cp kubectl-join /usr/local/bin
$ kubectl join <CLUSTERNAME>https://cloud.google.com/sdk/docs/downloads-apt-get?hl=ko
$ gcloud init
$ gcloud container clusters list
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
cluster3 asia-east1-a 1.16.13-gke.1 35.201.135.105 e2-medium 1.16.13-gke.1 2 RUNNING
$ gcloud config set compute/region <GKE_REGION>
Install 'kubectl regist' plugin on OpenMCP.
$ cd kubectl_plugin
$ chmod +x kubectl-regist_join
$ cp kubectl-regist_join /usr/local/bin
$ kubectl regist-join GKE <CLUSTERNAME>
Install 'kubectl join' plugin on OpenMCP.
$ cd kubectl_plugin
$ chmod +x kubectl-join
$ cp kubectl-join /usr/local/bin
$ kubectl join <CLUSTERNAME>
https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html
$ aws configure
AWS Access Key ID [None]: ...
AWS Secret Access Key [None]: ...
Default region name [None]: eu-west-2
Default output format [None]: json
$ aws eks list-clusters
{
"clusters": [
"cluster4"
]
}
Install 'kubectl regist' plugin on OpenMCP.
$ cd kubectl_plugin
$ chmod +x kubectl-regist_join
$ cp kubectl-regist_join /usr/local/bin
$ kubectl regist-join EKS <CLUSTERNAME>
Install 'kubectl join' plugin on OpenMCP.
$ cd kubectl_plugin
$ chmod +x kubectl-join
$ cp kubectl-join /usr/local/bin
$ kubectl join <CLUSTERNAME>
https://docs.microsoft.com/ko-kr/cli/azure/install-azure-cli-linux?pivots=apt
$ az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code ... to authenticate.
[
{
"cloudName": "AzureCloud",
"homeTenantId": "...",
"id": "...",
"isDefault": true,
"managedByTenants": [],
"name": "Microsoft Azure",
"state": "Enabled",
"tenantId": "...",
"user": {
"name": "admin@openmcp.onmicrosoft.com",
"type": "user"
}
}
]
$ az aks list
[
{
...
"kubernetesVersion": "1.17.11",
"linuxProfile": null,
"location": "australiaeast",
"maxAgentPools": 10,
"name": "aks-cluster1",
...
}
]
Install 'kubectl regist' plugin on OpenMCP.
$ cd kubectl_plugin
$ chmod +x kubectl-regist_join
$ cp kubectl-regist_join /usr/local/bin
$ kubectl regist-join AKS <CLUSTERNAME> <CLUSTERLOCATION>
Install 'kubectl join' plugin on OpenMCP.
$ cd kubectl_plugin
$ chmod +x kubectl-join
$ cp kubectl-join /usr/local/bin
$ kubectl join <CLUSTERNAME>
$ kubectl get kubefedcluster -n kube-federation-system
NAME READY AGE
cluster1 True 23h
cluster2 True 23hAs you deploy OpenMCPDeployment resource, pods will be scheduled to cluster1, cluster2 as Deployment resource.
$ kubectl create -f sample/openmcpdeployment/.$ kubectl get openmcpdeployment -n openmcp
NAME AGE
openmcp-deployment 72s
$ kubectl get deploy -n openmcp --context cluster1
NAME READY UP-TO-DATE AVAILABLE AGE
openmcp-deployment 2/2 2 2 79s
$ kubectl get deploy -n openmcp --context cluster2
NAME READY UP-TO-DATE AVAILABLE AGE
openmcp-deployment 2/2 2 2 80sAs you deploy OpenMCPService, service resource will be deployed to cluster1, cluster2.
$ kubectl create -f sample/openmcpservice/.$ kubectl get openmcpservice -n openmcp
NAME AGE
openmcp-service 18s
$ kubectl get service -n openmcp --context cluster1
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
openmcp-service LoadBalancer 10.110.248.190 10.0.3.200 80:31890/TCP 36s
$ kubectl get service -n openmcp --context cluster2
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
openmcp-service LoadBalancer 10.103.151.112 10.0.3.180 80:30569/TCP 34sWhen you deploy OpenMCPIngress, it will search for cluster with Target Service and deploy Ingress resource to the cluster.
$ kubectl create -f sample/openmcpingress/.$ kubectl get openmcpingress -n openmcp
NAME AGE
openmcp-ingress 4s
$ kubectl get ingress -n openmcp --context cluster1
NAME HOSTS ADDRESS PORTS AGE
openmcp-ingress cluster1.loadbalancing.openmcp.org 80 18s
$ kubectl get ingress -n openmcp --context cluster2
NAME HOSTS ADDRESS PORTS AGE
openmcp-ingress cluster2.loadbalancing.openmcp.org 80 18s$ kubectl create -f sample/openmcpdns/.$ kubectl get openmcpdomain -n kube-federation-system
NAME AGE
openmcp-service-domain 16h$ kubectl get openmcpservicednsrecord -n openmcp
NAME AGE
openmcp-service 16h$ kubectl get openmcpingressdnsrecord -n openmcp
NAME AGE
openmcp-ingress 16h$ kubectl get openmcpdnsendpoint -n openmcp
NAME AGE
ingress-openmcp-ingress 16h
service-openmcp-service 16h$ curl -L http://openmcp.service.orgWhen you deploy OpenMCPHybridAutoScaler, it will search for cluster with Target Deployment and deploy HorizontalPodAutoscaler, VerticalPodAutoscaler resource to the cluster.
$ kubectl create -f sample/openmcphybridautoscaler/.$ kubectl get openmcphybridautoscaler -n openmcp
NAME AGE
openmcp-has 6m51s
$ kubectl get hpa,vpa -n openmcp --context cluster1
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/openmcp-has Deployment/openmcp-deployment 56/100, 0%/50% 2 4 2 12m
NAME AGE
verticalpodautoscaler.autoscaling.k8s.io/openmcp-has 11m
$ kubectl get hpa,vpa -n openmcp --context cluster2
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/openmcp-has Deployment/openmcp-deployment 42/100, 0%/50% 2 4 2 11m
NAME AGE
verticalpodautoscaler.autoscaling.k8s.io/openmcp-has 11mThis project was supported by Institute of Information & communications Technology Planning & evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00052, Development of Distributed and Collaborative Container Platform enabling Auto Scaling and Service Mobility)