ISO: persistently mount /var/lib/containerd #7843

medyagh · 2020-04-22T04:55:07Z

before this PR:
/var/lib/containerd was stored in tmpfs

$  grep /var /proc/mounts | egrep -v "^(overlay|shm)"
/dev/vda1 /var/lib/boot2docker ext4 rw,relatime 0 0
/dev/vda1 /var/lib/docker ext4 rw,relatime 0 0
/dev/vda1 /var/lib/containers ext4 rw,relatime 0 0
/dev/vda1 /var/log ext4 rw,relatime 0 0
/dev/vda1 /var/lib/kubelet ext4 rw,relatime 0 0
/dev/vda1 /var/lib/cni ext4 rw,relatime 0 0
/dev/vda1 /var/lib/minikube ext4 rw,relatime 0 0
/dev/vda1 /var/lib/toolbox ext4 rw,relatime 0 0
/dev/vda1 /var/lib/minishift ext4 rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/6a9c4fa6-7027-48db-8f80-1ecb3c6e568b/volumes/kubernetes.io~secret/storage-provisioner-token-th5bp tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/6a9c4fa6-7027-48db-8f80-1ecb3c6e568b/volumes/kubernetes.io~secret/storage-provisioner-token-th5bp tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/615c7df5-e19c-440c-8688-000febf34ade/volumes/kubernetes.io~secret/kube-proxy-token-gscfv tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/615c7df5-e19c-440c-8688-000febf34ade/volumes/kubernetes.io~secret/kube-proxy-token-gscfv tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/1bc7a49f-4420-4e8e-a9d1-f2d100558136/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/1bc7a49f-4420-4e8e-a9d1-f2d100558136/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/b96c5a5a-5e2c-4373-a481-2a4a571a936e/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/b96c5a5a-5e2c-4373-a481-2a4a571a936e/volumes/kubernetes.io~secret/coredns-token-mbzzs tmpfs rw,relatime 0 0

After this PR

$ grep /var /proc/mounts | egrep -v "^(overlay|shm)"
/dev/vda1 /var/lib/boot2docker ext4 rw,relatime 0 0
/dev/vda1 /var/lib/docker ext4 rw,relatime 0 0
/dev/vda1 /var/lib/containerd ext4 rw,relatime 0 0
/dev/vda1 /var/lib/containers ext4 rw,relatime 0 0
/dev/vda1 /var/log ext4 rw,relatime 0 0
/dev/vda1 /var/tmp ext4 rw,relatime 0 0
/dev/vda1 /var/lib/kubelet ext4 rw,relatime 0 0
/dev/vda1 /var/lib/kubelet/pods ext4 rw,relatime 0 0
/dev/vda1 /mnt/vda1/var/lib/kubelet/pods ext4 rw,relatime 0 0
/dev/vda1 /var/lib/cni ext4 rw,relatime 0 0
/dev/vda1 /var/lib/minikube ext4 rw,relatime 0 0
/dev/vda1 /var/lib/toolbox ext4 rw,relatime 0 0
/dev/vda1 /var/lib/minishift ext4 rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/e498f978-13ca-4561-bd82-f729a79e0b22/volumes/kubernetes.io~secret/kube-proxy-token-5fb45 tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/e498f978-13ca-4561-bd82-f729a79e0b22/volumes/kubernetes.io~secret/kube-proxy-token-5fb45 tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/e498f978-13ca-4561-bd82-f729a79e0b22/volumes/kubernetes.io~secret/kube-proxy-token-5fb45 tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/e498f978-13ca-4561-bd82-f729a79e0b22/volumes/kubernetes.io~secret/kube-proxy-token-5fb45 tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/76633a80-973a-4b68-9f5e-f39d4c781709/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/76633a80-973a-4b68-9f5e-f39d4c781709/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/76633a80-973a-4b68-9f5e-f39d4c781709/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/76633a80-973a-4b68-9f5e-f39d4c781709/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/ce976354-8acc-4f4c-b63a-9c7bef5ad986/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/ce976354-8acc-4f4c-b63a-9c7bef5ad986/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/ce976354-8acc-4f4c-b63a-9c7bef5ad986/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/ce976354-8acc-4f4c-b63a-9c7bef5ad986/volumes/kubernetes.io~secret/coredns-token-qcxrj tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/pods/bfd58ce3-5687-4b13-bd10-ece1ff1d9dcb/volumes/kubernetes.io~secret/storage-provisioner-token-zzsbh tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/bfd58ce3-5687-4b13-bd10-ece1ff1d9dcb/volumes/kubernetes.io~secret/storage-provisioner-token-zzsbh tmpfs rw,relatime 0 0
tmpfs /var/lib/kubelet/bfd58ce3-5687-4b13-bd10-ece1ff1d9dcb/volumes/kubernetes.io~secret/storage-provisioner-token-zzsbh tmpfs rw,relatime 0 0
tmpfs /mnt/vda1/var/lib/kubelet/pods/bfd58ce3-5687-4b13-bd10-ece1ff1d9dcb/volumes/kubernetes.io~secret/storage-provisioner-token-zzsbh tmpfs rw,relatime 0 0

this PR will add /var/lib/containerd to disk instead of tmpfs that caused preload containerd to fail
and possible source of other mysterious containerd test failures.

also put /kubelet/pods there too since that was only folder on tempfs before this PR. leaving the rest of /var to be presitant in following up PRs

k8s-ci-robot · 2020-04-22T04:55:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

medyagh · 2020-04-22T05:02:53Z

/ok-to-test

minikube-pr-bot · 2020-04-22T05:05:25Z

kvm2 Driver
docker Driver

afbjorklund · 2020-04-22T05:37:18Z

Would be a less intrusive change to only move /var/lib/containerd, but

afbjorklund · 2020-04-22T05:44:49Z

This needs a rebase, without the podman revert

deploy/iso/minikube-iso/package/podman/podman.mk

medyagh · 2020-04-22T05:47:15Z

once jenkins is done the URL will be https://storage.cloud.google.com/minikube-builds/7843/minikube.iso

update i tried the ISO it doesnt seem to work on hyperkit

xit 0
I0421 23:47:05.479677    1920 main.go:110] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 192.168.64.1:49565->192.168.64.21:22: read: connection reset by peer
I0421 23:47:08.490846    1920 main.go:110] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 192.168.64.1:49566->192.168.64.21:22: read: connection reset by peer
I0421 23:47:11.498056    1920 main.go:110] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 192.168.64.1:49567->192.168.64.21:22: read: connection reset by peer
I0421 23:47:14.507819    1920 main.go:110] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 192.168.64.1:49569->192.168.64.21:22: read: connection reset by peer
^C

afbjorklund · 2020-04-22T17:13:55Z

I think you should leave re-organizing /var to the next release, and do a simple fix like in #7740

minikube-pr-bot · 2020-04-22T20:18:00Z

kvm2 Driver
Times for minikube: [66.938969361 66.616388846 65.215175756]
Average time for minikube: 66.25684465433334

Times for Minikube (PR 7843): [65.129501025 66.45322463000001 65.24051424299999]
Average time for Minikube (PR 7843): 65.60774663266666

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7843) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.0 on   |  0.063066 |           0.064577 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.020695 |           0.022936 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.012422 |           0.003397 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 42.433002 |          40.813116 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 21.720756 |          22.505028 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.735268 |           1.471351 |
| components...                  |           |                    |
| * Enabled addons:              |  0.178142 |           0.644432 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.088962 |           0.078013 |
| configured to use "minikube"   |           |                    |
|                                |  0.004532 |           0.004897 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.306763285000002 26.815962518 27.123464299]
Average time for minikube: 27.082063367333337

Times for Minikube (PR 7843): [28.103371672 28.228692250999995 27.330325618000003]
Average time for Minikube (PR 7843): 27.88746318033333

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7843) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.0 on           |  0.082790 |           0.075937 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002509 |           0.003110 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.066701 |           0.062513 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.974504 |           8.027296 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000265 |           0.000244 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.938833 |          18.083246 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.885722 |           1.562088 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.052445 |           0.002705 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.073951 |           0.064553 |
| configured to use "minikube"           |           |                    |
|                                        |  0.004345 |           0.005770 |
+----------------------------------------+-----------+--------------------+

deploy/iso/minikube-iso/package/automount/minikube-automount

codecov-io · 2020-04-22T21:47:31Z

Codecov Report

Merging #7843 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #7843   +/-   ##
=======================================
  Coverage   35.57%   35.57%           
=======================================
  Files         148      148           
  Lines        9297     9297           
=======================================
  Hits         3307     3307           
  Misses       5593     5593           
  Partials      397      397

minikube-pr-bot · 2020-04-22T22:11:38Z

kvm2 Driver
Times for minikube: [65.75042828499998 64.8755544 66.97153308300001]
Average time for minikube: 65.86583858933334

Times for Minikube (PR 7843): [64.48972665299999 65.079870305 65.495530272]
Average time for Minikube (PR 7843): 65.02170907666665

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7843) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.0 on   |  0.061767 |           0.066547 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.019822 |           0.022830 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003221 |           0.003771 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.175087 |          41.039294 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 22.241828 |          21.516541 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.441944 |           1.504659 |
| components...                  |           |                    |
| * Enabled addons:              |  0.847953 |           0.789408 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.070442 |           0.073860 |
| configured to use "minikube"   |           |                    |
|                                |  0.003773 |           0.004799 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.251727758 27.382844426 27.680943432999992]
Average time for minikube: 27.43850520566667

Times for Minikube (PR 7843): [27.806755057000004 27.700561630999996 26.297996064]
Average time for Minikube (PR 7843): 27.268437583999997

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7843) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.0 on           |  0.080379 |           0.081022 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.003188 |           0.002780 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.060851 |           0.061332 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.868635 |           8.075418 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000192 |           0.000217 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 18.540341 |          18.000425 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.813700 |           0.896130 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.002865 |           0.072123 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.064693 |           0.075305 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003662 |           0.003688 |
+----------------------------------------+-----------+--------------------+

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 22, 2020

k8s-ci-robot requested review from afbjorklund and sharifelgamal April 22, 2020 04:55

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 22, 2020

medyagh changed the title ~~make all /var presistant on VM ISO~~ ISO: persistently mount /var Apr 22, 2020

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Apr 22, 2020

afbjorklund requested changes Apr 22, 2020

View reviewed changes

deploy/iso/minikube-iso/package/podman/podman.mk Outdated Show resolved Hide resolved

afbjorklund mentioned this pull request Apr 22, 2020

wip: fix mounting /var/lib/contaired to tmpfs in VM ISO #7834

Closed

medyagh force-pushed the fix_iso_var_lib branch from ade5a44 to 4823728 Compare April 22, 2020 20:03

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 22, 2020

moviong containerd and kublet pods to presistant mount

27e5690

medyagh force-pushed the fix_iso_var_lib branch from 4823728 to 27e5690 Compare April 22, 2020 20:03

remove extra line

679e4ca

medyagh changed the title ~~ISO: persistently mount /var~~ ISO: persistently mount /var/lib/containrerd and /var/lib/kubelet/pods Apr 22, 2020

medyagh changed the title ~~ISO: persistently mount /var/lib/containrerd and /var/lib/kubelet/pods~~ ISO: persistently mount /var/lib/containrerd and kubelet/pods Apr 22, 2020

medyagh changed the title ~~ISO: persistently mount /var/lib/containrerd and kubelet/pods~~ ISO: persistently mount /var/lib/containrerd & kubelet/pods Apr 22, 2020

tstromberg reviewed Apr 22, 2020

View reviewed changes

deploy/iso/minikube-iso/package/automount/minikube-automount Outdated Show resolved Hide resolved

address review comments

214bf43

medyagh changed the title ~~ISO: persistently mount /var/lib/containrerd & kubelet/pods~~ ISO: persistently mount /var/lib/containrerd Apr 22, 2020

tstromberg changed the title ~~ISO: persistently mount /var/lib/containrerd~~ ISO: persistently mount /var/lib/containerd Apr 22, 2020

tstromberg approved these changes Apr 22, 2020

View reviewed changes

medyagh merged commit 2ffcec9 into kubernetes:master Apr 22, 2020

medyagh deleted the fix_iso_var_lib branch May 2, 2020 22:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ISO: persistently mount /var/lib/containerd #7843

ISO: persistently mount /var/lib/containerd #7843

medyagh commented Apr 22, 2020 •

edited

Loading

k8s-ci-robot commented Apr 22, 2020

medyagh commented Apr 22, 2020

minikube-pr-bot commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

medyagh commented Apr 22, 2020 •

edited

Loading

afbjorklund commented Apr 22, 2020

minikube-pr-bot commented Apr 22, 2020

codecov-io commented Apr 22, 2020

minikube-pr-bot commented Apr 22, 2020

ISO: persistently mount /var/lib/containerd #7843

ISO: persistently mount /var/lib/containerd #7843

Conversation

medyagh commented Apr 22, 2020 • edited Loading

After this PR

k8s-ci-robot commented Apr 22, 2020

medyagh commented Apr 22, 2020

minikube-pr-bot commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

medyagh commented Apr 22, 2020 • edited Loading

afbjorklund commented Apr 22, 2020

minikube-pr-bot commented Apr 22, 2020

codecov-io commented Apr 22, 2020

Codecov Report

minikube-pr-bot commented Apr 22, 2020

medyagh commented Apr 22, 2020 •

edited

Loading

medyagh commented Apr 22, 2020 •

edited

Loading