Add Manifest resource and support export an artifact using the manifest

[24sama]

What does this PR do?

• Add a Manifest resource to define cluster dependencies(images, binaries, Linux package e.g.).
• Add a command ./kk create manifest --kubeconfig "~/.kube/config" to generate a manifest-sample.yaml that related the config file specified cluster.
• Add a command ./kk artifact export -m manifest-sample.yaml to export a KubeKey artifact(kubekey-artifact.tar.gz) that is used to offline create a cluster. For now, the artifact contains images, binaries, Linux repository iso files.
• Support ./kk create cluster command uses an artifact to create a cluster.

Examples:

• Create a manifest file according to the kubeconfig. The KubeKey will detect all dependencies from the specified k8s cluster.

# The default value will use the `~/.kube/config`
./kk create manifest

# Or specify a kubeconfig and the manifest file's name
./kk create manifest -f my-manifest.yaml --kubeconfig ./mykube/config

• Export a KubeKey artifact by using a manifest file. The KubeKey will base on the manifest file to pull all images, download the specified binaries, then archive them as a kubekey-artifact.tar.gz file.

At first, containerd or docker must be installed in your environment.

./kk artifact export -m my-manifest.yaml -o my-artifact.tar.gz

• Create a cluster using the artifact. (Air-gapped Installation)

Need to have a private registry at first.

# I. Create a config and customize it. Especially, set up the .spec.registry.privateRegistry.
./kk create config

# II. Create a cluster in an air-gapped environment. There will be some following steps:
#     1. Unarchive the artifact
#     2. Load these images locally, and push the images to the private registry
#     3. Sync the iso files and k8s binaries to all nodes
#     4. Then KubeKey uses these files to install the Linux packages and k8s, certainly, it downloads nothing from the Internet.
./kk create cluster -f config-sample.yaml -a kubekey-artifact.tar.gz

# If the private registry already has these images
./kk create cluster -f config-sample.yaml -a kubekey-artifact.tar.gz --skip-push-images

# If the environment already has these Linux packages (conntrack, socat, ipset, ebtables, chrony .e.g) 
./kk create cluster -f config-sample.yaml -a kubekey-artifact.tar.gz --skip-install-packages

Details

Manifest

The manifest file content:

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:
  name: sample
spec:
  arches: # Contains all arches which are in the k8s cluster.
  - amd64
  operationSystems: # Contains all OS which is in the k8s cluster.
  - arch: amd64
    type: linux
    id: ubuntu
    version: 20.04
    osImage: Ubuntu 20.04.3 LTS
    repository: # Linux repostiory like `yum`, `apt` e.g.
      iso:
        localPath: "" # KubeKey will copy the local Linux repository iso file into the artifact from the local path.
        url: https://github.com/pixiake/k8s-dependencies/releases/download/v1.0.0/ubuntu-20.04-amd64-debs.iso # KubeKey will download the Linux repository iso file for this URL.
  kubernetesDistribution:
    type: kubernetes
    version: v1.21.5
  components:
    helm:
      version: v3.6.3
    cni:
      version: v0.9.1
    etcd:
      version: v3.4.13
    containerRuntime:
      type: docker
      version: 20.10.8
    crictl:
      version: v1.22.0
  images: # Contains all images which are in the each of k8s cluster nodes
  - docker.io/calico/cni:v3.20.0
  - docker.io/calico/kube-controllers:v3.20.0
  - docker.io/calico/node:v3.20.0
  - docker.io/calico/pod2daemon-flexvol:v3.20.0
  - docker.io/coredns/coredns:1.8.0
  - docker.io/kubesphere/k8s-dns-node-cache:1.15.12
  - docker.io/kubesphere/kube-apiserver:v1.21.5
  - docker.io/kubesphere/kube-controller-manager:v1.21.5
  - docker.io/kubesphere/kube-proxy:v1.21.5
  - docker.io/kubesphere/kube-scheduler:v1.21.5
  - docker.io/kubesphere/pause:3.4.1

For now, there are three todo lists:

• KubeKey can't detect the helm, cni, etcd according to kubeconfig. These components' versions are generated by a default value. Besides. KubeKey doesn't support customizing these components' versions to create a k8s cluster.
• When the specified k8s cluster container runtime is containerd，the field .spec.components.containerRuntime.type will also be the docker and the version will be the default version(20.10.8). This is because of KubeKey install a containerd based k8s cluster by installing a docker and making the kubelet connect the /run/containerd/containerd.sock.
• KubeKey will try to auto-complete the image's repo/namespace/ when these fields are missing. Users may need to check if these images' full names are correct manually.

Artifact

The artifact's dir tree:

artifact/
├── images
│   ├── docker.io-calico-cni-v3.20.0.tar
│   ├── docker.io-calico-kube-controllers-v3.20.0.tar
│   ├── docker.io-calico-node-v3.20.0.tar
│   ├── docker.io-calico-pod2daemon-flexvol-v3.20.0.tar
│   ├── docker.io-coredns-coredns-1.8.0.tar
│   ├── docker.io-kubesphere-k8s-dns-node-cache-1.15.12.tar
│   ├── docker.io-kubesphere-kube-apiserver-v1.21.5.tar
│   ├── docker.io-kubesphere-kube-controller-manager-v1.21.5.tar
│   ├── docker.io-kubesphere-kube-proxy-v1.21.5.tar
│   ├── docker.io-kubesphere-kube-scheduler-v1.21.5.tar
│   └── docker.io-kubesphere-pause-3.4.1.tar
├── repository
│   └── amd64
│       └── ubuntu
│           └── 20.04
│               └── ubuntu-20.04-amd64.iso
└── v1.21.5
    └── amd64
        ├── cni-plugins-linux-amd64-v0.9.1.tgz
        ├── docker-20.10.8.tgz
        ├── etcd-v3.4.13-linux-amd64.tar.gz
        ├── helm
        ├── kubeadm
        ├── kubectl
        └── kubelet

7 directories, 19 files

Total TODO Lists

• Support K3s artifact

[ks-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 24sama

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [24sama]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[24sama]

/kind do-not-merge

[ks-ci-bot]

@24sama: The label(s) kind/do-not-merge cannot be applied, because the repository doesn't have them.

In response to this:

/kind do-not-merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pixiake]

Wonderful，I think this feature will further enhance the usability of KubeKey.

[tanguofu]

awesome！ this feature is our urgently needed

[24sama]

Reopen this PR, because the GitHub CI workflow has been modified.

[pixiake]

/lgtm

[ks-ci-bot]

LGTM label has been added.

Git tree hash: ecabc4ba40d146fa5a744f7142de41486e95c989

[24sama]

/cherrypick release-2.0

[ks-ci-bot]

@24sama: new pull request created: #925

In response to this:

/cherrypick release-2.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[24sama]

I think this PR can fix this issue #597