New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containerized Volume Client Drivers - Design Proposal #22216

Closed
wants to merge 7 commits into
base: master
from

Conversation

@childsb
Copy link
Member

childsb commented Feb 29, 2016

This document lists user stories and design for containerizing the hosts volume driver stack.

Summary :

"As a system administrator I want to use Persistent Volumes without installing any additional software on each host."


This change is Reviewable

childsb and others added some commits Feb 24, 2016

mention configMap and fuse
Signed-off-by: Huamin Chen <hchen@redhat.com>
@googlebot

This comment has been minimized.

Copy link

googlebot commented Feb 29, 2016

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

@k8s-teamcity-mesosphere

This comment has been minimized.

Copy link

k8s-teamcity-mesosphere commented on 2eb790f Feb 29, 2016

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 17784 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 229 Build time: 00:13:29

@k8s-merge-robot k8s-merge-robot added size/L and removed size/M labels Mar 1, 2016

@k8s-teamcity-mesosphere

This comment has been minimized.

Copy link

k8s-teamcity-mesosphere commented on 1535567 Mar 1, 2016

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 17820 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 229 Build time: 00:09:01

@bgrant0607

This comment has been minimized.

Copy link
Member

bgrant0607 commented Mar 1, 2016

cc @kubernetes/sig-storage @saad-ali @thockin

@k8s-teamcity-mesosphere

This comment has been minimized.

Copy link

k8s-teamcity-mesosphere commented on cbdb7d6 Mar 1, 2016

TeamCity OSS :: Kubernetes Mesos :: 4 - Smoke Tests Build 17907 outcome was SUCCESS
Summary: Tests passed: 1, ignored: 229 Build time: 00:04:32

@childsb

This comment has been minimized.

Copy link
Member

childsb commented Mar 1, 2016

Alternative Proposal from Storage SIG-

Volume "attacher" is long running daemon on each host which handles the mount operations for kubelet. Long running daemon facilitated through daemon sets: https://github.com/kubernetes/kubernetes/blob/master/docs/admin/daemons.md

I will update the design doc with the alternative approach (user stories remain the same)

**Benefits:**

1. No client installed on host.
2. No client configuration on host.

This comment has been minimized.

@lvlv

lvlv Mar 3, 2016

Contributor

Less library dependencies in kubelet, multi client version support.

@rootfs

This comment has been minimized.

Copy link
Member

rootfs commented Mar 17, 2016

Use DaemonSet to Mount Filesystems on Host

                 +- - - - - - - - - - - - - - - - - - +
                 '                   Master           '
                 '                                    '
                 '          +-----------------------+ '
                 '          |       ConfigMap       | '
                 '          +-----------------------+ '
                 '                                    '
                 +- - - - - - - - - - - - - - - - - - +
                              ^
                              | 1. get REST API path
                              |
                 +- - - - - - - - - - - - - - - - - - +
                 '                   Kubelet          '
                 '                                    '
                 '          +-----------------------+ '
                 '          |        Mounter        | ' <+
                 '          +-----------------------+ '  |
                 '                                    '  |
                 +- - - - - - - - - - - - - - - - - - +  |
                              |                          |
                              | 2. mount REST call       | 5. mount REST success
                              v                          |
+ - - - - - - - - - - - - - - - - - - - - - - - - - - +  |
'                 DaemonSet                           '  |
'                                                     '  |
'                           +-----------------------+ '  |
'   +---------------------> |        Daemon         | ' -+
'   |                       +-----------------------+ '
'   |                         |                       '
'   | 4. success              | 3. exec               '
'   |                         v                       '
'   |                       +-----------------------+ '
'   +---------------------- | /sbin/mount.glusterfs | '
'                           +-----------------------+ '
'                                                     '
+ - - - - - - - - - - - - - - - - - - - - - - - - - - +

Requirement

Needs mount namespace propagation from Docker >= 1.10

Feedbacks from Storage SIG on Mar 17th

  • One monolithic daemonset with all mount binaries or one daemon set that orchestrates multiple imagaes, each of which support certain types of filesystem.
    • Rolling upgrade of daemonset doesn't exist.
    • If new volume mount is absent in the daemonset, how to load it on the hosts without bringing down the existing mounts.
  • (pmorie) in-kernel vs in-user space mounters don't have the same resource usage profile. Long running mounts like FUSE need to be subject to quota control.
  • (saad) What is in the ConfigMap? REST API path for individual filesystems.
  • Is pod-running-inside-kubelet approach more resource efficient?
    • (sami) running a pod to mount by kubelet
    • (sami) daemonset needs a REST API redirection on all hosts, seemingly inefficient.
    • (hchen) running-pod-inside-kubelet needs to manage pod lifecycle, and ensure pod executes the mount (meaning watching apiserver)
  • (sami and pmorie) How to ensure mount namespace propagation to be set in a Pod (not Docker API)
  • (cnelluri) How to ensure daemonset reacheable only by kubelet? Do other user containers able to access the daemonset to execute mount?

A quick PoC of using DaemonSet mounter to mount glusterfs can be viewed here

@jessfraz

This comment has been minimized.

Copy link
Contributor

jessfraz commented Jul 6, 2016

This could probably be closed for the same reasons as mentioned here: #18333 (comment). Also It's missing a CLA, needs a rebase, and hasn't been updated since March 17.

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Aug 23, 2016

also relevant: #26093, #20698

I guess someone from @kubernetes/sig-storage should carry this given no cla and no activity

@k8s-bot

This comment has been minimized.

Copy link

k8s-bot commented Aug 30, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

@k8s-bot

This comment has been minimized.

Copy link

k8s-bot commented Sep 13, 2016

GCE e2e build/test passed for commit cbdb7d6.

@wattsteve

This comment has been minimized.

Copy link
Contributor

wattsteve commented Oct 2, 2016

I think we need to keep this alive? I think the reason it has been dragging is that we've been waiting for changes to propagate from OCI into the Docker CLI into the Docker golang client that Kube uses. @rootfs @childsb can you comment or at least redirect to a different PR if this is being tackled elsewhere?

@matchstick

This comment has been minimized.

Copy link
Contributor

matchstick commented Nov 1, 2016

@jingxu97 Does this work overlap with any of your containerized storage client work?
@saad-ali

@saad-ali

This comment has been minimized.

Copy link
Member

saad-ali commented Nov 2, 2016

This is very much the problem that we are trying to solve now for GCI and this captures the problem we want to solve. The details of how this will be implemented need to be worked out.

@k8s-merge-robot

This comment has been minimized.

Copy link
Contributor

k8s-merge-robot commented Dec 1, 2016

Adding label:do-not-merge because PR changes docs prohibited to auto merge
See http://kubernetes.io/editdocs/ for information about editing docs

@childsb

This comment has been minimized.

Copy link
Member

childsb commented Dec 13, 2016

@jingxu97 we would like to bring this in line with the GKE containerized mount. is there a design doc for that work somewhere?

We are scheming on containerized mount support for the other volume types and would like to begin work soon.

@luxas

This comment has been minimized.

Copy link
Member

luxas commented Dec 13, 2016

This proposal has to move to the community repo

@jingxu97

This comment has been minimized.

Copy link
Contributor

jingxu97 commented Dec 13, 2016

@childsb I will work on the doc and share with you asap. Please let me know if you have any question meanwhile. Thanks!

@k8s-merge-robot

This comment has been minimized.

Copy link
Contributor

k8s-merge-robot commented Jan 23, 2017

[APPROVALNOTIFIER] Needs approval from an approver in each of these OWNERS Files:

We suggest the following people:
cc @thockin
You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-merge-robot

This comment has been minimized.

Copy link
Contributor

k8s-merge-robot commented Feb 14, 2017

This PR hasn't been active in 62 days. It will be closed in 27 days (Mar 13, 2017).

cc @brendandburns @childsb @jingxu97 @smarterclayton @thockin

You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Feb 27, 2017

Closing for now - the new driver work will help here

@thockin thockin closed this Feb 27, 2017

@owenhaynes

This comment has been minimized.

Copy link

owenhaynes commented Feb 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment