Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Cilium to version 1.12.3 #15242

Merged
merged 1 commit into from
Nov 14, 2022

Conversation

rastaman
Copy link
Contributor

@rastaman rastaman commented Oct 29, 2022

Context

The Cilium version deployed by minikube is quite old and doesn't support arm64 setups.

This PR update Cilium to version 1.12.3.

Implementation

Update Kubernetes template in minikube/pkg/cni/cilium.go to:

  • allow cilium-operator to list and watch nodes as it is also done in upstream template
  • use a cluster name and cluster id as it looks it is required in cilium 1.12.3
  • update the agent health port as it changed 5 months ago in cilium

Tests

This has been tested by running current minikube code from a Mac M1 with the podman driver in a Qemu VM (limavm) runnning Fedora Core 36 (Fedora-Cloud-Base-36-1.5.aarch64):

./out/minikube -p miniforge start --rootless=false --container-runtime=containerd --cpus=4 --cni=cilium --disk-size=20000mb  --dns-domain=cluster.local --driver=podman --memory=7916m --nodes=1 
😄  [miniforge] minikube v1.27.1 sur Darwin 12.6 (arm64)
    ▪ MINIKUBE_ROOTLESS=false
✨  Utilisation du pilote podman (expérimental) basé sur la configuration de l'utilisateur
📌  Utilisation du pilote Podman avec le privilège root
👍  Démarrage du noeud de plan de contrôle miniforge dans le cluster miniforge
🚜  Extraction de l'image de base...
    > gcr.io/k8s-minikube/kicbase...:  347.52 MiB / 347.52 MiB  100.00% 9.10 Mi
E1029 11:34:47.234886   77006 cache.go:203] Error downloading kic artifacts:  not yet implemented, see issue #8426
🔥  Création de podman container (CPUs=4, Memory=7916Mo) ...
📦  Préparation de Kubernetes v1.25.3 sur containerd 1.6.9...
E1029 11:35:16.036962   77006 start.go:130] Unable to get host IP: RoutableHostIPFromInside is currently only implemented for linux
    ▪ Génération des certificats et des clés
    ▪ Démarrage du plan de contrôle ...
    ▪ Configuration des règles RBAC ...
🔗  Configuration de Cilium (Container Networking Interface)...
🔎  Vérification des composants Kubernetes...
    ▪ Utilisation de l'image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Modules activés: storage-provisioner, default-storageclass
🏄  Terminé ! kubectl est maintenant configuré pour utiliser "miniforge" cluster et espace de noms "default" par défaut.
$ kubectl get pods -n kube-system
NAME                                READY   STATUS    RESTARTS   AGE
cilium-mbzrq                        1/1     Running   0          46s
cilium-operator-6b885c4575-6f954    1/1     Running   0          46s
coredns-565d847f94-6f6vm            1/1     Running   0          46s
etcd-miniforge                      1/1     Running   0          58s
kube-apiserver-miniforge            1/1     Running   0          59s
kube-controller-manager-miniforge   1/1     Running   0          58s
kube-proxy-72czb                    1/1     Running   0          46s
kube-scheduler-miniforge            1/1     Running   0          59s
storage-provisioner                 1/1     Running   0          57s
$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium             Running: 1
                  cilium-operator    Running: 1
Cluster Pods:     0/1 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.3@sha256:30de50c4dc0a1e1077e9e7917a54d5cab253058b3f779822aec00f5c817ca826: 1
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.3@sha256:816ec1da586139b595eeb31932c61a7c13b07fb4a0255341c0e0f18608e84eff: 1
$ kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium status
KVStore:                 Ok   Disabled
Kubernetes:              Ok   1.25 (v1.25.3) [linux/arm64]
Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:    Probe   [eth0 192.168.49.2]
Host firewall:           Disabled
CNI Chaining:            none
Cilium:                  Ok   1.12.3 (v1.12.3-1c466d2)
NodeMonitor:             Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok   
IPAM:                    IPv4: 2/254 allocated from 10.244.0.0/24, 
ClusterMesh:             0/0 clusters ready, 0 global-services
BandwidthManager:        Disabled
Host Routing:            BPF
Masquerading:            BPF   [eth0]   10.244.0.0/24 [IPv4: Enabled, IPv6: Disabled]
Controller Status:       18/18 healthy
Proxy Status:            OK, ip 10.244.0.214, 0 redirects active on ports 10000-20000
Global Identity Range:   min 256, max 65535
Hubble:                  Ok   Current/Max Flows: 36/4095 (0.88%), Flows/s: 0.20   Metrics: Disabled
Encryption:              Disabled
Cluster health:          1/1 reachable   (2022-10-29T09:37:02Z)

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Oct 29, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Oct 29, 2022
@k8s-ci-robot
Copy link
Contributor

Welcome @rastaman!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 29, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @rastaman. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 29, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rastaman
Once this PR has been reviewed and has the lgtm label, please assign prezha for approval by writing /assign @prezha in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Oct 29, 2022
@spowelljr
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 31, 2022
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15242) |
+----------------+----------+---------------------+
| minikube start | 55.3s    | 55.2s               |
| enable ingress | 28.2s    | 28.6s               |
+----------------+----------+---------------------+

Times for minikube start: 54.5s 54.0s 55.7s 55.6s 56.7s
Times for minikube (PR 15242) start: 54.0s 54.2s 54.5s 57.4s 55.9s

Times for minikube (PR 15242) ingress: 28.3s 29.2s 28.7s 28.8s 28.2s
Times for minikube ingress: 28.3s 29.7s 28.7s 28.8s 25.7s

docker driver with docker runtime
error collecting results for docker driver: timing run 0 with minikube: timing cmd: [out/minikube addons enable ingress]: waiting for minikube: exit status 10
docker driver with containerd runtime
error downloading artifacts: artifact download start: exit status 90

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
KVM_Linux TestKubernetesUpgrade (gopogh) 0.63 (chart)
KVM_Linux_containerd TestFunctional/parallel/ConfigCmd (gopogh) 1.27 (chart)
Docker_Linux_containerd TestPause/serial/Start (gopogh) 18.99 (chart)
Docker_Linux_containerd TestOffline (gopogh) 24.05 (chart)
Docker_Linux_containerd TestAddons/Setup (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/DashboardCmd (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/MountCmd/any-port (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/MySQL (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/PersistentVolumeClaim (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/ServiceCmd (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/ServiceCmdConnect (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/serial/ExtraConfig (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/serial/SoftStart (gopogh) 27.85 (chart)
Docker_Linux_containerd TestFunctional/serial/StartWithProxy (gopogh) 27.85 (chart)
Docker_Linux_containerd TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh) 27.85 (chart)
Docker_Linux_containerd TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh) 27.85 (chart)
Docker_Linux_containerd TestJSONOutput/start/Command (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/AddNode (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/DeleteNode (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/DeployApp2Nodes (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/FreshStart2Nodes (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/PingHostFrom2Pods (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/ProfileList (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/StartAfterStop (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/StopNode (gopogh) 27.85 (chart)
Docker_Linux_containerd TestMultiNode/serial/RestartMultiNode (gopogh) 28.48 (chart)
Docker_Linux_containerd TestMultiNode/serial/RestartKeepsNodes (gopogh) 29.75 (chart)
Docker_Windows TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 35.06 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@rastaman
Copy link
Contributor Author

rastaman commented Nov 4, 2022

I will look at the tests results.

@medyagh
Copy link
Member

medyagh commented Nov 14, 2022

@rastaman thank you for this PR ! look forward for more contributions from you

@medyagh medyagh merged commit 84d6bc8 into kubernetes:master Nov 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants