multi-node: Add support for volumes using CSI addon #15829

denisok · 2023-02-10T19:09:19Z

k8s-ci-robot · 2023-02-10T19:09:28Z

Welcome @denisok!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2023-02-10T19:09:29Z

Hi @denisok. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

minikube-bot · 2023-02-10T19:12:05Z

Can one of the admins verify this patch?

denisok · 2023-02-10T23:02:52Z

deploy/addons/csi-hostpath-driver/deploy/csi-hostpath-plugin.yaml.tmpl

      containers:
-        - name: csi-external-health-monitor-agent


health-agent was merged to kubelet: https://github.com/kubernetes-csi/external-health-monitor#overview

denisok · 2023-02-10T23:04:52Z

deploy/addons/csi-hostpath-driver/rbac/rbac-external-provisioner.yaml.tmpl

@@ -92,12 +96,24 @@ metadata:
 rules:
 # Only one of the following rules for endpoints or leases is required based on
 # what is set for `--leader-election-type`. Endpoints are deprecated in favor of Leases.
- apiGroups: [""]


https://github.com/kubernetes-csi/external-provisioner/blob/v3.3.0/deploy/kubernetes/rbac.yaml

medyagh

do you mind adding output of minikube Before / After this PR?

also, it would be nice to add a section on multinode use example in our website

https://minikube.sigs.k8s.io/docs/tutorials/volume_snapshots_and_csi/

medyagh · 2023-02-12T02:10:26Z

/ok-to-test

minikube-pr-bot · 2023-02-12T03:07:05Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 51.1s    | 53.2s               |
| enable ingress | 26.0s    | 25.1s               |
+----------------+----------+---------------------+

Times for minikube ingress: 27.7s 26.7s 24.7s 26.7s 24.2s
Times for minikube (PR 15829) ingress: 28.2s 23.7s 24.2s 25.2s 24.2s

Times for minikube (PR 15829) start: 55.0s 53.9s 53.3s 49.9s 53.7s
Times for minikube start: 49.8s 50.1s 55.1s 50.8s 49.6s

docker driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15829) |
+-------------------+----------+---------------------+
| minikube start    | 26.2s    | 26.2s               |
| ⚠️  enable ingress | 33.7s    | 44.6s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 25.1s 25.6s 26.2s 27.4s 26.9s
Times for minikube (PR 15829) start: 26.8s 26.2s 26.9s 26.7s 24.5s

Times for minikube ingress: 20.0s 22.0s 23.0s 21.0s 82.6s
Times for minikube (PR 15829) ingress: 19.6s 20.0s 81.6s 81.6s 20.1s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 21.4s    | 21.4s               |
| enable ingress | 38.7s    | 32.1s               |
+----------------+----------+---------------------+

Times for minikube (PR 15829) start: 20.8s 21.1s 21.6s 21.7s 22.0s
Times for minikube start: 21.7s 21.4s 22.3s 20.9s 20.9s

Times for minikube ingress: 80.0s 18.6s 31.6s 31.6s 31.6s
Times for minikube (PR 15829) ingress: 33.0s 32.5s 31.6s 31.6s 31.6s

minikube-pr-bot · 2023-02-12T03:58:27Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/AddonExistsAfterStop (gopogh)	2.10 (chart)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/Pause (gopogh)	2.10 (chart)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/SecondStart (gopogh)	2.10 (chart)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/VerifyKubernetesImages (gopogh)	2.10 (chart)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh)	2.80 (chart)
KVM_Linux	TestMultiNode/serial/StartAfterStop (gopogh)	6.25 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh)	98.03 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	100.00 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh)	100.00 (chart)
Docker_macOS	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	100.00 (chart)
Docker_macOS	TestKubernetesUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestMissingContainerUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestRunningBinaryUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh)	100.00 (chart)
Docker_macOS	TestStoppedBinaryUpgrade/Upgrade (gopogh)	100.00 (chart)
KVM_Linux_containerd	TestPreload (gopogh)	100.00 (chart)

To see the flake rates of all tests by environment, click here.

denisok · 2023-02-12T11:13:02Z

do you mind adding output of minikube Before / After this PR?

sure, here evertyhing with ./out/ is a new version:

[dkondratenko@denlen minikube]$ minikube addons enable volumesnapshots 
💡  volumesnapshots is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0
🌟  The 'volumesnapshots' addon is enabled
[dkondratenko@denlen minikube]$ minikube addons disable volumesnapshots 
🌑  "The 'volumesnapshots' addon is disabled

[dkondratenko@denlen minikube]$ ./out/minikube addons enable volumesnapshots 
💡  volumesnapshots is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/sig-storage/snapshot-controller:v6.1.0
🌟  The 'volumesnapshots' addon is enabled
[dkondratenko@denlen minikube]$ ./out/minikube addons disable volumesnapshots 
🌑  "The 'volumesnapshots' addon is disabled

[dkondratenko@denlen minikube]$ minikube addons enable csi-hostpath-driver
💡  csi-hostpath-driver is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
❗  [WARNING] For full functionality, the 'csi-hostpath-driver' addon requires the 'volumesnapshots' addon to be enabled.

You can enable 'volumesnapshots' addon by running: 'minikube addons enable volumesnapshots'

    ▪ Using image k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0
    ▪ Using image k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1
    ▪ Using image k8s.gcr.io/sig-storage/livenessprobe:v2.2.0
    ▪ Using image k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0
    ▪ Using image k8s.gcr.io/sig-storage/csi-external-health-monitor-controller:v0.2.0
    ▪ Using image k8s.gcr.io/sig-storage/hostpathplugin:v1.6.0
    ▪ Using image k8s.gcr.io/sig-storage/csi-resizer:v1.1.0
    ▪ Using image k8s.gcr.io/sig-storage/csi-attacher:v3.1.0
    ▪ Using image k8s.gcr.io/sig-storage/csi-external-health-monitor-agent:v0.2.0
🔎  Verifying csi-hostpath-driver addon...
🌟  The 'csi-hostpath-driver' addon is enabled
[dkondratenko@denlen minikube]$ minikube addons disable csi-hostpath-driver
🌑  "The 'csi-hostpath-driver' addon is disabled

[dkondratenko@denlen minikube]$ ./out/minikube addons enable csi-hostpath-driver
💡  csi-hostpath-driver is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
❗  [WARNING] For full functionality, the 'csi-hostpath-driver' addon requires the 'volumesnapshots' addon to be enabled.

You can enable 'volumesnapshots' addon by running: 'minikube addons enable volumesnapshots'

    ▪ Using image registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.0
    ▪ Using image registry.k8s.io/sig-storage/hostpathplugin:v1.9.0
    ▪ Using image registry.k8s.io/sig-storage/livenessprobe:v2.8.0
    ▪ Using image registry.k8s.io/sig-storage/csi-resizer:v1.6.0
    ▪ Using image registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0
    ▪ Using image registry.k8s.io/sig-storage/csi-provisioner:v3.3.0
    ▪ Using image registry.k8s.io/sig-storage/csi-attacher:v4.0.0
    ▪ Using image registry.k8s.io/sig-storage/csi-external-health-monitor-controller:v0.7.0
🔎  Verifying csi-hostpath-driver addon...
🌟  The 'csi-hostpath-driver' addon is enabled
[dkondratenko@denlen minikube]$ ./out/minikube addons disable csi-hostpath-driver
🌑  "The 'csi-hostpath-driver' addon is disabled

also, it would be nice to add a section on multinode use example in our website

https://minikube.sigs.k8s.io/docs/tutorials/volume_snapshots_and_csi/
yes, will do later.

denisok · 2023-02-12T11:17:41Z

I see some failures related to csi, will check it.

denisok · 2023-02-13T19:12:46Z

OK, I looked to some of the tests, and those failures related to current state: Pending for many containers not just csi drivers. Same in other PRs currently open.

I tried different confirurations and don't see any issues with kvm or podman driver,

Once I seen the issue minikube starting with podman and cri-o - Node wasn't ready and thus containers were in Pending state, looks like #15676 .

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Mon, 13 Feb 2023 19:39:17 +0100   Mon, 13 Feb 2023 19:39:12 +0100   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 13 Feb 2023 19:39:17 +0100   Mon, 13 Feb 2023 19:39:12 +0100   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 13 Feb 2023 19:39:17 +0100   Mon, 13 Feb 2023 19:39:12 +0100   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Mon, 13 Feb 2023 19:39:17 +0100   Mon, 13 Feb 2023 19:39:12 +0100   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?

But it was only once and I can't reproduce it.

PR / functional_podman_ubuntu (pull_request) fails due to network issue with GH or network:

10s
Run curl -LO https://github.com/medyagh/gopogh/releases/download/v0.13.0/gopogh-linux-amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  0 3848k    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 73 3848k   73 2831k    0     0  1985k      0  0:00:01  0:00:01 --:--:-- 2686k
 73 3848k   73 2831k    0     0  1166k      0  0:00:03  0:00:02  0:00:01 1377k
 73 3848k   73 2831k    0     0   825k      0  0:00:04  0:00:03  0:00:01  926k
 73 3848k   73 2831k    0     0   639k      0  0:00:06  0:00:04  0:00:02  697k
 73 3848k   73 2831k    0     0   521k      0  0:00:07  0:00:05  0:00:02  559k
 73 3848k   73 2831k    0     0   440k      0  0:00:08  0:00:06  0:00:02     0
 73 3848k   73 2831k    0     0   380k      0  0:00:10  0:00:07  0:00:03     0
 73 3848k   73 2831k    0     0   335k      0  0:00:11  0:00:08  0:00:03     0
 73 3848k   73 2831k    0     0   300k      0  0:00:12  0:00:09  0:00:03     0
 73 3848k   73 2831k    0     0   272k      0  0:00:14  0:00:10  0:00:04     0
 73 3848k   73 2831k    0     0   272k      0  0:00:14  0:00:10  0:00:04     0
curl: (18) transfer closed with 1041185 bytes remaining to read
Error: Process completed with exit code 18.

needs re-run.

denisok · 2023-02-13T21:33:17Z

Hi @medyagh,

I added docs and checked tests. So far I found no regressions.

Thanks,
Denys

minikube-pr-bot · 2023-02-13T21:40:44Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 53.4s    | 54.1s               |
| enable ingress | 26.6s    | 27.1s               |
+----------------+----------+---------------------+

Times for minikube start: 54.0s 53.6s 53.0s 52.9s 53.3s
Times for minikube (PR 15829) start: 55.4s 53.4s 55.4s 52.8s 53.3s

Times for minikube ingress: 25.7s 27.7s 29.2s 24.2s 26.4s
Times for minikube (PR 15829) ingress: 28.2s 25.2s 25.7s 27.7s 28.7s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 26.3s    | 27.0s               |
| enable ingress | 21.3s    | 21.1s               |
+----------------+----------+---------------------+

Times for minikube start: 26.4s 26.0s 25.9s 26.6s 26.6s
Times for minikube (PR 15829) start: 27.2s 26.0s 26.6s 27.8s 27.2s

Times for minikube ingress: 21.1s 20.1s 22.1s 21.1s 22.1s
Times for minikube (PR 15829) ingress: 20.1s 22.1s 21.1s 20.1s 22.1s

docker driver with containerd runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15829) |
+-------------------+----------+---------------------+
| minikube start    | 22.2s    | 22.1s               |
| ⚠️  enable ingress | 30.0s    | 46.8s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 22.3s 21.8s 23.1s 21.9s 22.0s
Times for minikube (PR 15829) start: 21.5s 22.0s 22.3s 22.4s 22.3s

Times for minikube ingress: 32.6s 32.6s 20.6s 31.6s 32.6s
Times for minikube (PR 15829) ingress: 31.6s 20.1s 81.1s 80.6s 20.6s

minikube-pr-bot · 2023-02-13T22:50:39Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 56.2s    | 55.8s               |
| enable ingress | 27.5s    | 27.7s               |
+----------------+----------+---------------------+

Times for minikube (PR 15829) start: 57.1s 55.0s 54.5s 56.7s 56.0s
Times for minikube start: 55.2s 56.2s 54.9s 58.5s 56.3s

Times for minikube ingress: 28.4s 29.8s 28.9s 24.8s 25.8s
Times for minikube (PR 15829) ingress: 28.4s 28.9s 24.3s 27.9s 28.9s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 28.3s    | 27.7s               |
| enable ingress | 33.5s    | 33.3s               |
+----------------+----------+---------------------+

Times for minikube start: 28.5s 28.9s 28.7s 27.0s 28.4s
Times for minikube (PR 15829) start: 27.2s 27.7s 28.9s 27.1s 27.5s

Times for minikube ingress: 21.1s 22.6s 82.6s 22.2s 19.1s
Times for minikube (PR 15829) ingress: 82.2s 20.2s 21.1s 21.2s 22.1s

docker driver with containerd runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15829) |
+-------------------+----------+---------------------+
| minikube start    | 22.6s    | 23.3s               |
| ⚠️  enable ingress | 29.8s    | 37.2s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube (PR 15829) ingress: 79.6s 21.6s 31.6s 20.6s 32.7s
Times for minikube ingress: 31.6s 32.6s 32.7s 31.6s 20.6s

Times for minikube (PR 15829) start: 23.2s 23.1s 23.4s 23.5s 23.1s
Times for minikube start: 22.5s 22.7s 23.6s 21.9s 22.5s

minikube-pr-bot · 2023-02-13T23:03:31Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Windows	TestRunningBinaryUpgrade (gopogh)	0.00 (chart)
Docker_Linux	TestPause/serial/SecondStartNoReconfiguration (gopogh)	7.02 (chart)
Hyper-V_Windows	TestPause/serial/SecondStartNoReconfiguration (gopogh)	56.25 (chart)
Hyper-V_Windows	TestNoKubernetes/serial/StartWithK8s (gopogh)	62.89 (chart)
Hyper-V_Windows	TestRunningBinaryUpgrade (gopogh)	64.95 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh)	98.15 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	100.00 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh)	100.00 (chart)
Docker_macOS	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	100.00 (chart)
Docker_macOS	TestKubernetesUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestMissingContainerUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestRunningBinaryUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh)	100.00 (chart)
Docker_macOS	TestStoppedBinaryUpgrade/Upgrade (gopogh)	100.00 (chart)
Docker_Windows	TestFunctional/parallel/ServiceCmd (gopogh)	100.00 (chart)
Hyper-V_Windows	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	100.00 (chart)
Hyper-V_Windows	TestMultiNode/serial/RestartKeepsNodes (gopogh)	100.00 (chart)
Hyper-V_Windows	TestStoppedBinaryUpgrade/Upgrade (gopogh)	100.00 (chart)
KVM_Linux_containerd	TestPreload (gopogh)	100.00 (chart)

To see the flake rates of all tests by environment, click here.

minikube-pr-bot · 2023-02-13T23:56:11Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Windows	TestRunningBinaryUpgrade (gopogh)	0.00 (chart)
Docker_Linux_containerd	TestFunctional/serial/LogsFileCmd (gopogh)	1.75 (chart)
Docker_macOS	TestPause/serial/SecondStartNoReconfiguration (gopogh)	3.97 (chart)
KVM_Linux	TestMultiNode/serial/RestartMultiNode (gopogh)	8.82 (chart)
Hyperkit_macOS	TestRunningBinaryUpgrade (gopogh)	10.46 (chart)
Hyperkit_macOS	TestFunctional/parallel/ConfigCmd (gopogh)	15.03 (chart)
Hyper-V_Windows	TestPause/serial/SecondStartNoReconfiguration (gopogh)	56.25 (chart)
Hyper-V_Windows	TestNoKubernetes/serial/StartWithK8s (gopogh)	62.89 (chart)
Hyper-V_Windows	TestRunningBinaryUpgrade (gopogh)	64.95 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh)	98.15 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	100.00 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh)	100.00 (chart)
Docker_macOS	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	100.00 (chart)
Docker_macOS	TestKubernetesUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestMissingContainerUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestRunningBinaryUpgrade (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh)	100.00 (chart)
Docker_macOS	TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh)	100.00 (chart)
Docker_macOS	TestStoppedBinaryUpgrade/Upgrade (gopogh)	100.00 (chart)
Docker_Windows	TestFunctional/parallel/ServiceCmd (gopogh)	100.00 (chart)
Hyper-V_Windows	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	100.00 (chart)
Hyper-V_Windows	TestMultiNode/serial/RestartKeepsNodes (gopogh)	100.00 (chart)
Hyper-V_Windows	TestStoppedBinaryUpgrade/Upgrade (gopogh)	100.00 (chart)
KVM_Linux_containerd	TestPreload (gopogh)	100.00 (chart)
none_Linux	TestAddons/Setup (gopogh)	100.00 (chart)
none_Linux	TestCertExpiration (gopogh)	100.00 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

denisok · 2023-02-14T09:03:27Z

I can't make any sense from test results.

could someone help me to see where should I look into? Should be they all green?
What should I pay attention to ?

minikube-pr-bot · 2023-02-14T10:38:00Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 58.9s    | 59.3s               |
| enable ingress | 30.5s    | 30.5s               |
+----------------+----------+---------------------+

Times for minikube start: 57.9s 59.9s 57.6s 59.6s 59.3s
Times for minikube (PR 15829) start: 60.7s 58.7s 59.3s 59.7s 58.0s

Times for minikube ingress: 31.4s 30.5s 29.4s 30.9s 30.4s
Times for minikube (PR 15829) ingress: 29.4s 30.5s 31.5s 30.4s 30.5s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 29.1s    | 29.0s               |
| enable ingress | 34.4s    | 22.4s               |
+----------------+----------+---------------------+

Times for minikube start: 29.1s 28.6s 28.8s 29.6s 29.3s
Times for minikube (PR 15829) start: 29.3s 29.6s 29.3s 28.4s 28.3s

Times for minikube ingress: 21.2s 25.7s 22.2s 81.7s 21.3s
Times for minikube (PR 15829) ingress: 22.2s 21.2s 24.2s 22.2s 22.2s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15829) |
+----------------+----------+---------------------+
| minikube start | 24.5s    | 24.4s               |
| enable ingress | 37.5s    | 29.6s               |
+----------------+----------+---------------------+

Times for minikube start: 23.6s 24.8s 24.6s 24.2s 25.2s
Times for minikube (PR 15829) start: 24.5s 24.2s 23.9s 24.2s 25.2s

Times for minikube ingress: 47.8s 47.7s 32.3s 28.2s 31.7s
Times for minikube (PR 15829) ingress: 31.7s 31.7s 32.7s 19.2s 32.8s

minikube-pr-bot · 2023-02-14T12:29:09Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Windows	TestStartStop/group/default-k8s-diff-port/serial/Pause (gopogh)	0.00 (chart)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/DeployApp (gopogh)	0.00 (chart)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/EnableAddonWhileActive (gopogh)	0.00 (chart)
Hyperkit_macOS	TestStartStop/group/embed-certs/serial/FirstStart (gopogh)	0.00 (chart)
Docker_Linux_containerd	TestMultiNode/serial/RestartMultiNode (gopogh)	0.60 (chart)
KVM_Linux_containerd	TestFunctional/parallel/ConfigCmd (gopogh)	0.61 (chart)
Hyper-V_Windows	TestAddons/parallel/CloudSpanner (gopogh)	9.86 (chart)
Docker_Windows	TestStartStop/group/old-k8s-version/serial/Pause (gopogh)	10.00 (chart)
Docker_Windows	TestFunctional/parallel/ServiceCmd (gopogh)	100.00 (chart)
Hyper-V_Windows	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	100.00 (chart)
Hyper-V_Windows	TestMultiNode/serial/RestartKeepsNodes (gopogh)	100.00 (chart)
Hyper-V_Windows	TestNoKubernetes/serial/StartWithStopK8s (gopogh)	100.00 (chart)
Hyper-V_Windows	TestStoppedBinaryUpgrade/Upgrade (gopogh)	100.00 (chart)
KVM_Linux_containerd	TestPreload (gopogh)	100.00 (chart)

To see the flake rates of all tests by environment, click here.

medyagh

Thank you for this PR, please consider making a follow UP PR that when users use multi-node, minikube automatically enable this addon for them or suggests them to do so

k8s-ci-robot · 2023-02-14T18:51:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: denisok, medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 10, 2023

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 10, 2023

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 10, 2023

k8s-ci-robot requested review from afbjorklund and sharifelgamal February 10, 2023 19:09

denisok mentioned this pull request Feb 10, 2023

add distributed snapshotter kubernetes-csi/csi-driver-host-path#392

Closed

update CSI driver, enable multi-node

0ba0122

denisok force-pushed the master branch from 2bd8853 to 0ba0122 Compare February 10, 2023 22:56

denisok marked this pull request as ready for review February 10, 2023 23:00

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 10, 2023

k8s-ci-robot requested review from medyagh and prezha February 10, 2023 23:00

denisok commented Feb 10, 2023

View reviewed changes

medyagh reviewed Feb 11, 2023

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 12, 2023

denisok and others added 2 commits February 13, 2023 21:30

Merge branch 'kubernetes:master' into master

76ccf5c

add multi-node cluster doc for the csi hostpath driver

874de40

denisok requested review from medyagh and removed request for prezha, afbjorklund and sharifelgamal February 13, 2023 21:46

Merge branch 'kubernetes:master' into master

a1a8b7d

medyagh approved these changes Feb 14, 2023

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 14, 2023

medyagh merged commit 5ea80d0 into kubernetes:master Feb 14, 2023

medyagh changed the title ~~update CSI driver, enable multi-node~~ multi-node: Add support for volumes using CSI addon Feb 14, 2023

denisok mentioned this pull request Feb 14, 2023

Persistent Volumes permission issue for 2+ node clutsters #12360

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi-node: Add support for volumes using CSI addon #15829

multi-node: Add support for volumes using CSI addon #15829

denisok commented Feb 10, 2023

k8s-ci-robot commented Feb 10, 2023

k8s-ci-robot commented Feb 10, 2023

minikube-bot commented Feb 10, 2023

denisok Feb 10, 2023

denisok Feb 10, 2023

medyagh left a comment

medyagh commented Feb 12, 2023

minikube-pr-bot commented Feb 12, 2023

minikube-pr-bot commented Feb 12, 2023

denisok commented Feb 12, 2023 •

edited

denisok commented Feb 12, 2023

denisok commented Feb 13, 2023

denisok commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

denisok commented Feb 14, 2023

minikube-pr-bot commented Feb 14, 2023

minikube-pr-bot commented Feb 14, 2023

medyagh left a comment •

edited

k8s-ci-robot commented Feb 14, 2023

multi-node: Add support for volumes using CSI addon #15829

multi-node: Add support for volumes using CSI addon #15829

Conversation

denisok commented Feb 10, 2023

k8s-ci-robot commented Feb 10, 2023

k8s-ci-robot commented Feb 10, 2023

minikube-bot commented Feb 10, 2023

denisok Feb 10, 2023

Choose a reason for hiding this comment

denisok Feb 10, 2023

Choose a reason for hiding this comment

medyagh left a comment

Choose a reason for hiding this comment

medyagh commented Feb 12, 2023

minikube-pr-bot commented Feb 12, 2023

minikube-pr-bot commented Feb 12, 2023

denisok commented Feb 12, 2023 • edited

denisok commented Feb 12, 2023

denisok commented Feb 13, 2023

denisok commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

minikube-pr-bot commented Feb 13, 2023

denisok commented Feb 14, 2023

minikube-pr-bot commented Feb 14, 2023

minikube-pr-bot commented Feb 14, 2023

medyagh left a comment • edited

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 14, 2023

denisok commented Feb 12, 2023 •

edited

medyagh left a comment •

edited