Add build support for riscv64 arch

[carlosedp]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Adds support to build Kubernetes binaries for riscv64 architecture by using makefiles as:

make WHAT=./cmd/binary KUBE_BUILD_PLATFORMS=linux/riscv64

Output dir:

❯ ll _output/local/bin/linux/riscv64
total 1.1G
-rwxr-xr-x 1 carlosedp carlosedp 5.6M Dec  6 14:17 linkcheck*
-rwxr-xr-x 1 carlosedp carlosedp  43M Dec  6 14:17 kube-scheduler*
-rwxr-xr-x 1 carlosedp carlosedp 1.9M Dec  6 14:17 mounter*
-rwxr-xr-x 1 carlosedp carlosedp 108M Dec  6 14:17 kubelet*
-rwxr-xr-x 1 carlosedp carlosedp 154M Dec  6 14:17 genman*
-rwxr-xr-x 1 carlosedp carlosedp 107M Dec  6 14:17 kubemark*
-rwxr-xr-x 1 carlosedp carlosedp  40M Dec  6 14:17 kubeadm*
-rwxr-xr-x 1 carlosedp carlosedp 147M Dec  6 14:17 genkubedocs*
-rwxr-xr-x 1 carlosedp carlosedp  43M Dec  6 14:17 gendocs*
-rwxr-xr-x 1 carlosedp carlosedp 2.2M Dec  6 14:17 go-runner*
-rwxr-xr-x 1 carlosedp carlosedp  38M Dec  6 14:18 kube-proxy*
-rwxr-xr-x 1 carlosedp carlosedp  44M Dec  6 14:18 kubectl*
-rwxr-xr-x 1 carlosedp carlosedp 116M Dec  6 14:18 kube-apiserver*
-rwxr-xr-x 1 carlosedp carlosedp 8.5M Dec  6 14:18 genswaggertypedocs*
-rwxr-xr-x 1 carlosedp carlosedp 8.9M Dec  6 14:18 ginkgo*
-rwxr-xr-x 1 carlosedp carlosedp  47M Dec  6 14:18 apiextensions-apiserver*
-rwxr-xr-x 1 carlosedp carlosedp  43M Dec  6 14:18 genyaml*
-rwxr-xr-x 1 carlosedp carlosedp 106M Dec  6 14:18 kube-controller-manager*

❯ ./_output/local/bin/linux/riscv64/kubectl version
Client Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1512+616fce7839b16d-dirty", GitCommit:"616fce7839b16dc61c9079cbae3b0ce1e47060f7", GitTreeState:"dirty", BuildDate:"2019-12-06T19:03:17Z", GoVersion:"devel +1c193b20a6 3 weeks ago", Compiler:"gc", Platform:"linux/riscv64"}

Special notes for your reviewer:

Ginkgo has been bumped after PR onsi/ginkgo#632 was merged to be able to build on riscv64.

Does this PR introduce a user-facing change?:

NONE

[k8s-ci-robot]

Hi @carlosedp. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[carlosedp]

/assign @BenTheElder

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: carlosedp
To complete the pull request process, please assign bentheelder
You can assign the PR to them by writing /assign @bentheelder in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• hack/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[feiskyer]

/ok-to-test

[carlosedp]

Waiting for K8s to adopt go1.14 that supports riscv64 architecture the I'll proceed with the PR.
/remove-lifecycle stale

[cblecker]

@carlosedp As this deals with building/releasing artifacts and supporting additional architectures, a KEP (https://git.k8s.io/enhancements/keps/README.md) via sig-release should be needed.

This hasn't had any movement, primarily because the Kubernetes project hasn't decided to take on any additional architectures. I'm going to go ahead and close this change for now, but if a KEP is accepted, we can look at re-opening this PR.

/close

[k8s-ci-robot]

@cblecker: Closed this PR.

In response to this:

@carlosedp As this deals with building/releasing artifacts and supporting additional architectures, a KEP (https://git.k8s.io/enhancements/keps/README.md) via sig-release should be needed.

This hasn't had any movement, primarily because the Kubernetes project hasn't decided to take on any additional architectures. I'm going to go ahead and close this change for now, but if a KEP is accepted, we can look at re-opening this PR.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dims]

/sig architecture
/reopen

@carlosedp please review this WIP document on guidance kubernetes/community#5014

[k8s-ci-robot]

@dims: Reopened this PR.

In response to this:

/sig architecture
/reopen

@carlosedp please review this WIP document on guidance kubernetes/community#5014

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dims]

i've a WIP for the guidance kubernetes/community#5014

[carlosedp]

Thanks @dims. I'll revisit this now as Go for RISCV has been upstreamed and most pre-reqs are already upstream as well!

[k8s-ci-robot]

@carlosedp: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-cross	20d471b	link	/test pull-kubernetes-cross
pull-kubernetes-e2e-kind	20d471b	link	/test pull-kubernetes-e2e-kind
pull-kubernetes-kubemark-e2e-gce-big	20d471b	link	/test pull-kubernetes-kubemark-e2e-gce-big

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[andrewsykim]

/remove-sig cloud-provider

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jiangxiaobin96]

if i want to build Kubernetes binaries for riscv64 architecture, which branch should i git clone?
i follow make WHAT=./cmd/binary KUBE_BUILD_PLATFORMS=linux/riscv64 and get the following response.

!!! [1029 07:47:14] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.
!!! [1029 07:47:52] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.
!!! [1029 07:49:06] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.
!!! [1029 07:49:10] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.
!!! [1029 07:50:08] Unsupported host arch. Must be x86_64, 386, arm, arm64, s390x or ppc64le.
+++ [1029 07:50:14] Building go targets for linux/riscv64:
    ./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
> non-static build: k8s.io/kubernetes/./vendor/k8s.io/code-generator/cmd/prerelease-lifecycle-gen
touch: cannot touch '_output/bin/prerelease-lifecycle-gen': No such file or directory
make[1]: *** [Makefile.generated_files:209: _output/bin/prerelease-lifecycle-gen] Error 1
make: *** [Makefile:553: generated_files] Error 2

[BenTheElder]

@jiangxiaobin96 unfortunately building for a new platform requires patching the bash scripts to allow this platform. see:

• Allow fast builds on ppc64le #105399
• https://github.com/kubernetes/kubernetes/pull/97743/files#r586161189

[jiangxiaobin96]

@jiangxiaobin96 unfortunately building for a new platform requires patching the bash scripts to allow this platform. see:

• Allow fast builds on ppc64le #105399
• https://github.com/kubernetes/kubernetes/pull/97743/files#r586161189

I succeeded in build Kubernetes binaries for riscv64. But how to install it? I used executable file directly like ./kubeadm init and met the failure.

[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

        Unfortunately, an error has occurred:
                timed out waiting for the condition

        This error is likely caused by:
                - The kubelet is not running
                - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                - 'systemctl status kubelet'
                - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI.

        Here is one example how you may list all Kubernetes containers running in docker:
                - 'docker ps -a | grep kube | grep -v pause'
                Once you have found the failing container, you can inspect its logs with:
                - 'docker logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

Because i didn't know how to install it, so i can not find the kubelet.service to watch the failure log. So how can i install it and generate service file?

[neolit123]

Because i didn't know how to install it, so i can not find the kubelet.service to watch the failure log. So how can i install it and generate service file?

if you wish to use kubeadm build from source (without apt, rpm package managers) you can check here:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

tab: "Without a package manager..."

[jiangxiaobin96]

I found where is my problem. I just run make KUBE_BUILD_PLATFORMS=linux/riscv64 and generate

[root@master0 riscv64]# ls -l
total 1166682
-rwxr-xr-x 1 root root  41711688 Nov  4 21:22 apiextensions-apiserver
-rwxr-xr-x 1 root root   6227408 Nov  2 14:15 conversion-gen
-rwxr-xr-x 1 root root   5961136 Nov  2 14:09 deepcopy-gen
-rwxr-xr-x 1 root root   5944720 Nov  2 14:12 defaulter-gen
-rwxr-xr-x 1 root root 107038704 Nov  4 21:23 e2e.test
-rwxr-xr-x 1 root root 116796584 Nov  4 21:22 e2e_node.test
-rwxr-xr-x 1 root root  38709064 Nov  4 21:23 gendocs
-rwxr-xr-x 1 root root 129439752 Nov  4 21:24 genkubedocs
-rwxr-xr-x 1 root root 135353224 Nov  4 21:22 genman
-rwxr-xr-x 1 root root   3285672 Nov  4 21:22 genswaggertypedocs
-rwxr-xr-x 1 root root  38704744 Nov  4 21:24 genyaml
-rwxr-xr-x 1 root root   7187536 Nov  4 21:21 ginkgo
-rwxr-xr-x 1 root root   1900544 Nov  2 14:26 go-bindata
-rwxr-xr-x 1 root root   1801424 Nov  4 21:22 go-runner
-rwxr-xr-x 1 root root   3366768 Nov  2 14:09 go2make
-rwxr-xr-x 1 root root 102367232 Nov  4 21:24 kube-apiserver
-rwxr-xr-x 1 root root  95092736 Nov  4 21:22 kube-controller-manager
-rwxr-xr-x 1 root root  33816576 Nov  4 21:21 kube-proxy
-rwxr-xr-x 1 root root  37289984 Nov  4 21:24 kube-scheduler
-rwxr-xr-x 1 root root  35192832 Nov  4 21:24 kubeadm
-rwxr-xr-x 1 root root  39321600 Nov  4 21:23 kubectl
-rwxr-xr-x 1 root root  96825128 Nov  4 21:22 kubelet
-rwxr-xr-x 1 root root  95325544 Nov  4 21:23 kubemark
-rwxr-xr-x 1 root root   4674000 Nov  4 21:23 linkcheck
-rwxr-xr-x 1 root root   1572864 Nov  4 21:22 mounter
-rwxr-xr-x 1 root root   9751176 Nov  2 14:22 openapi-gen

But don't release the kubernetes. I try to KUBE_BUILD_PLATFORMS=linux/riscv64 make update && KUBE_BUILD_PLATFORMS=linux/riscv64 make release , there are still some dependency which have not supported RISCV64.
If i don't make update and release and just want to use k8s directly, i should put all config files like kubelet.service to proper path. I look the file build/rpm/BUILD like

[pkg_rpm_for_goarch(
    name = "kubeadm",
    changelog = "//:CHANGELOG.md",
    data = [
        "10-kubeadm.conf",
        "50-kubeadm.conf",
        "kubeadm.conf",
        "kubelet.env",
        "//cmd/kubeadm",
    ],
    goarch = arch,
    spec_file = "kubeadm.spec",
    tags = ["manual"],
    version_file = "//build:os_package_version",
) for arch in NODE_PLATFORMS["linux"]]

If I want to use k8s correctly, where can i refer to to place all config files?