Make GCI nodes mount non tmpfs, ext* & bind mounts using an external mounter #36267

vishh · 2016-11-04T22:59:31Z

This PR downloads the stage1 & gci-mounter ACIs as part of cluster bring up instead of downloading them dynamically from gcr.io, which was the cause for #36206.

I have also optimized the containerized mounter to pre-load the mounter image once to avoid fetch latency while using it.

Original PR which got reverted: #35821

GCI nodes use an external mounter script to mount NFS & GlusterFS storage volumes

@mtaufen Node e2e is not re-enabled in this PR.

cc @jingxu97

This change is

jingxu97 · 2016-11-04T23:07:16Z

cluster/gce/gci/configure.sh

@@ -185,7 +192,7 @@ function install-kube-binary-config {
  chmod -R 755 "${kube_bin}"

  # Install rkt binary to allow mounting storage volumes in GCI
-  install-rkt
+  install-gci-mounter-tools


change the comment too?

jingxu97 · 2016-11-04T23:07:38Z

@mtaufen could you also help review this PR? Thanks!

jingxu97 · 2016-11-07T17:25:24Z

@k8s-bot gci gke e2e test this

mtaufen · 2016-11-07T19:40:29Z

For the following tests the cluster fails to initialize within 300 seconds:

pull-kubernetes-e2e-gce
pull-kubernetes-e2e-gce-etcd3
pull-kubernetes-e2e-gce-gci
pull-kubernetes-kubemark-e2e-gce

The verification test is wants us to run ./hack/update_owners.py:

I1107 08:42:07.156] Verifying hack/make-rules/../../hack/verify-test-owners.sh
I1107 08:42:08.482] # OUTDATED TESTS (6):
I1107 08:42:08.482] Generated release_1_5 clientset should create v2alpha1 scheduleJobs, delete scheduleJobs, watch scheduleJobs -- caesarxuchao
I1107 08:42:08.483] ScheduledJob should not emit unexpected warnings -- soltysh
I1107 08:42:08.483] ScheduledJob should not schedule jobs when suspended -- soltysh
I1107 08:42:08.483] ScheduledJob should not schedule new jobs when ForbidConcurrent -- soltysh
I1107 08:42:08.483] ScheduledJob should replace jobs when ReplaceConcurrent -- soltysh
I1107 08:42:08.483] ScheduledJob should schedule multiple jobs concurrently -- soltysh
I1107 08:42:08.483] # NEW TESTS (11):
I1107 08:42:08.484] CronJob should not emit unexpected warnings
I1107 08:42:08.484] CronJob should not schedule jobs when suspended
I1107 08:42:08.484] CronJob should not schedule new jobs when ForbidConcurrent
I1107 08:42:08.484] CronJob should replace jobs when ReplaceConcurrent
I1107 08:42:08.484] CronJob should schedule multiple jobs concurrently
I1107 08:42:08.484] Federation daemonsets DaemonSet objects should be created and deleted successfully
I1107 08:42:08.485] Federation deployments Deployment objects should be created and deleted successfully
I1107 08:42:08.485] Federation deployments Federated Deployment should create and update matching deployments in underling clusters
I1107 08:42:08.485] Generated release_1_5 clientset should create v2alpha1 cronJobs, delete cronJobs, watch cronJobs
I1107 08:42:08.485] Kubectl alpha client Kubectl run CronJob should create a CronJob
I1107 08:42:08.485] k8s.io/kubernetes/test/e2e_node/system
I1107 08:42:08.485] 
I1107 08:42:08.486] ERROR: the test list has changed
I1107 08:42:08.489] Run ./hack/update_owners.py to fix it

These two also timed out waiting for cluster initialization:

pull-kubernetes-e2e-gke

W1107 08:51:33.535] ERROR: (gcloud.container.clusters.create) Operation [<Operation
W1107 08:51:33.535]  name: u'operation-1478536254707-2383fe35'
W1107 08:51:33.535]  operationType: OperationTypeValueValuesEnum(CREATE_CLUSTER, 1)
W1107 08:51:33.535]  selfLink: u'https://test-container.sandbox.googleapis.com/v1/projects/154064302321/zones/us-central1-f/operations/operation-1478536254707-2383fe35'
W1107 08:51:33.535]  status: StatusValueValuesEnum(DONE, 3)
W1107 08:51:33.535]  statusMessage: u'Timed out waiting for cluster initialization. Cluster API may not be available.'
W1107 08:51:33.535]  targetLink: u'https://test-container.sandbox.googleapis.com/v1/projects/154064302321/zones/us-central1-f/clusters/e2e-gke-agent-pr-20-0'
W1107 08:51:33.536]  zone: u'us-central1-f'>] finished with error: Timed out waiting for cluster initialization. Cluster API may not be available.

pull-kubernetes-e2e-gke-gci

W1107 09:59:11.300] ERROR: (gcloud.container.clusters.create) Operation [<Operation
W1107 09:59:11.301]  name: u'operation-1478540312573-574d1447'
W1107 09:59:11.301]  operationType: OperationTypeValueValuesEnum(CREATE_CLUSTER, 1)
W1107 09:59:11.301]  selfLink: u'https://test-container.sandbox.googleapis.com/v1/projects/943316867058/zones/us-central1-f/operations/operation-1478540312573-574d1447'
W1107 09:59:11.301]  status: StatusValueValuesEnum(DONE, 3)
W1107 09:59:11.301]  statusMessage: u'Timed out waiting for cluster initialization. Cluster API may not be available.'
W1107 09:59:11.302]  targetLink: u'https://test-container.sandbox.googleapis.com/v1/projects/943316867058/zones/us-central1-f/clusters/e2e-gke-agent-pr-82-0'
W1107 09:59:11.302]  zone: u'us-central1-f'>] finished with error: Timed out waiting for cluster initialization. Cluster API may not be available.

jingxu97 · 2016-11-08T02:16:14Z

cluster/gce/gci/mounter/mounter

 }

 # Garbage collect old rkt containers on exit
 trap gc EXIT

-${RKT_BINARY} run --stage1-name="coreos.com/rkt/stage1-fly:1.18.0" \
+if [[ ! $(${RKT_BINARY} image list | grep ${MOUNTER_IMAGE}) ]]; then
+        ${RKT_BINARY} fetch file://${MOUNTER_ACI}


@vishh this rkt fetch does not work because of signature issue. I found out a way to make this work by adding an option insecure-options

${RKT_BINARY} fetch --insecure-options=image file://${MOUNTER_ACI}

Oops. Fixing it now.

k8s-ci-robot · 2016-11-08T18:43:16Z

Jenkins GCI GKE smoke e2e failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T18:43:16Z

Jenkins GCI GCE e2e failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T18:43:17Z

Jenkins GCE etcd3 e2e failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot gce etcd3 e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T18:43:19Z

Jenkins Kubemark GCE e2e failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot kubemark e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T18:43:19Z

Jenkins GKE smoke e2e failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot cvm gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T18:43:49Z

Jenkins unit/integration failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot unit test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T18:44:16Z

Jenkins GCE Node e2e failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot node e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T18:44:25Z

Jenkins GCE e2e failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot cvm gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-08T19:05:49Z

Jenkins verification failed for commit 96e23163d7c90191438e53fb739503bfe3e16917. Full PR test history.

The magic incantation to run this job again is @k8s-bot verify test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

…unter-scope"" This reverts commit 402116a.

Added a make rule `make upload` to audit and automate release artifact uploads to GCS. Signed-off-by: Vishnu kannan <vishnuk@google.com>

Signed-off-by: Vishnu kannan <vishnuk@google.com>

mtaufen · 2016-11-08T19:29:26Z

WRT verification test, you probably need to:

rebase, I'm guessing type *KubeletConfiguration has no field or method ExperimentalRuntimeIntegrationType is because the name of that field changed to EnableCRIin dcce768
run hack/update-codecgen.sh

Update the gci-mounter sha1 number

…ring runtime Signed-off-by: Vishnu kannan <vishnuk@google.com>

Signed-off-by: Vishnu kannan <vishnuk@google.com>

jingxu97 · 2016-11-08T23:22:54Z

@k8s-bot node e2e test this

jingxu97 · 2016-11-08T23:23:40Z

@k8s-bot gci gce e2e test this

jingxu97 · 2016-11-08T23:23:54Z

@k8s-bot gci gke e2e test this

vishh · 2016-11-08T23:30:50Z

@jingxu97 @mtaufen @saad-ali this PR is ready to be merged.

jingxu97 · 2016-11-08T23:31:06Z

The tests passed. I rerun a few just to make sure. @mtaufen Do you have any other comments? Otherwise, LGTM

k8s-github-robot · 2016-11-09T03:11:17Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2016-11-09T03:46:32Z

Automatic merge from submit-queue

vishh · 2016-11-09T22:28:39Z

@saad-ali @jingxu97 Is this expected to be cherry-picked into v1.5?

saad-ali · 2016-11-09T23:03:31Z

@saad-ali @jingxu97 Is this expected to be cherry-picked into v1.5?

No cherry pick needed. We will fast forward the release-1.5 branch to pick up changes before the code freeze is lifted.

@mtaufen

Automatic merge from submit-queue Make GCI nodes mount non tmpfs, ext* & bind mounts using an external mounter This PR downloads the stage1 & gci-mounter ACIs as part of cluster bring up instead of downloading them dynamically from gcr.io, which was the cause for kubernetes#36206. I have also optimized the containerized mounter to pre-load the mounter image once to avoid fetch latency while using it. Original PR which got reverted: kubernetes#35821 ```release-note GCI nodes use an external mounter script to mount NFS & GlusterFS storage volumes ``` @mtaufen Node e2e is not re-enabled in this PR. cc @jingxu97

googlebot added the cla: yes label Nov 4, 2016

vishh added release-note Denotes a PR that will be considered when it comes time to generate release notes. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Nov 4, 2016

vishh added this to the v1.5 milestone Nov 4, 2016

vishh added the area/os/gci label Nov 4, 2016

vishh assigned jingxu97 Nov 4, 2016

jingxu97 reviewed Nov 4, 2016

View reviewed changes

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 4, 2016

vishh force-pushed the gci-mounter-scope branch 2 times, most recently from 835fbd0 to ee30af7 Compare November 4, 2016 23:19

k8s-github-robot added the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label Nov 4, 2016

jingxu97 reviewed Nov 8, 2016

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 8, 2016

vishh added 3 commits November 8, 2016 11:09

Revert "Revert "Merge pull request kubernetes#35821 from vishh/gci-mo…

dd8ec91

…unter-scope"" This reverts commit 402116a.

Use a local file for rkt stage1 and gci-mounter docker image.

77218d3

Added a make rule `make upload` to audit and automate release artifact uploads to GCS. Signed-off-by: Vishnu kannan <vishnuk@google.com>

re-enable node e2e for GCI mounter

0562386

Signed-off-by: Vishnu kannan <vishnuk@google.com>

Update configure.sh

d07396f

Update the gci-mounter sha1 number

vishh added 2 commits November 8, 2016 12:13

Make gci mounter pre-fetch mounter image to reduce startup latency du…

773ad9b

…ring runtime Signed-off-by: Vishnu kannan <vishnuk@google.com>

update generated files

5ad53bb

Signed-off-by: Vishnu kannan <vishnuk@google.com>

vishh force-pushed the gci-mounter-scope branch from 96e2316 to 5ad53bb Compare November 8, 2016 20:14

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 8, 2016

jingxu97 added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 9, 2016

k8s-github-robot merged commit 6983262 into kubernetes:master Nov 9, 2016

jessfraz mentioned this pull request Nov 16, 2016

Cherry picking "mounter" patches on release-1.4 #35809

Closed

chentao1596 mentioned this pull request Dec 5, 2016

WIP:kubelet: support multi-headers when getting pod from HTTP source #38089

Closed

mengjiao-liu mentioned this pull request Jul 21, 2022

[WIP]kubelet: Remove deprecated flag --experimental-mounter-path #111288

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make GCI nodes mount non tmpfs, ext* & bind mounts using an external mounter #36267

Make GCI nodes mount non tmpfs, ext* & bind mounts using an external mounter #36267

vishh commented Nov 4, 2016 •

edited

Loading

jingxu97 Nov 4, 2016

vishh Nov 4, 2016

jingxu97 commented Nov 4, 2016

jingxu97 commented Nov 7, 2016

mtaufen commented Nov 7, 2016

jingxu97 Nov 8, 2016

vishh Nov 8, 2016

vishh Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

mtaufen commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

vishh commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

k8s-github-robot commented Nov 9, 2016

k8s-github-robot commented Nov 9, 2016

vishh commented Nov 9, 2016

saad-ali commented Nov 9, 2016

Make GCI nodes mount non tmpfs, ext* & bind mounts using an external mounter #36267

Make GCI nodes mount non tmpfs, ext* & bind mounts using an external mounter #36267

Conversation

vishh commented Nov 4, 2016 • edited Loading

jingxu97 Nov 4, 2016

Choose a reason for hiding this comment

vishh Nov 4, 2016

Choose a reason for hiding this comment

jingxu97 commented Nov 4, 2016

jingxu97 commented Nov 7, 2016

mtaufen commented Nov 7, 2016

jingxu97 Nov 8, 2016

Choose a reason for hiding this comment

vishh Nov 8, 2016

Choose a reason for hiding this comment

vishh Nov 8, 2016

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

k8s-ci-robot commented Nov 8, 2016

mtaufen commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

vishh commented Nov 8, 2016

jingxu97 commented Nov 8, 2016

k8s-github-robot commented Nov 9, 2016

k8s-github-robot commented Nov 9, 2016

vishh commented Nov 9, 2016

saad-ali commented Nov 9, 2016

vishh commented Nov 4, 2016 •

edited

Loading