New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Durable (non-shared) local storage management #121

Open
smarterclayton opened this Issue Oct 9, 2016 · 49 comments

Comments

Projects
@smarterclayton
Contributor

smarterclayton commented Oct 9, 2016

Feature Description

  • One-line feature description (can be used as a release note): Make locally attached (non-network attached) storage available as a persistent volume source.
  • Primary contact (assignee): @msau42 @vishh
  • Responsible SIGs: @kubernetes/sig-storage-feature-requests @kubernetes/sig-apps-feature-requests
  • Design proposal link (community repo): kubernetes/community#306
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y): 1.7/1.8
    • Beta release target (x.y): 1.10
    • Stable release target (x.y):
@davidopp

This comment has been minimized.

Member

davidopp commented Oct 9, 2016

I assume this is kubernetes/kubernetes#7562 ?

And that kubernetes/kubernetes#30044 is related?

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Oct 11, 2016

Yes and yes. Consensus is to see if we can pull together a design at most in 1.5

@bprashanth

This comment has been minimized.

Member

bprashanth commented Oct 11, 2016

It would be great if we were ready to go at this in 1.6
@kubernetes/sig-storage @jingxu97

@thockin

This comment has been minimized.

Member

thockin commented Oct 11, 2016

I think we can commit to that.

On Mon, Oct 10, 2016 at 9:35 PM, Prashanth B notifications@github.com
wrote:

It would be great if we were ready to go at this in 1.6
@kubernetes/sig-storage
https://github.com/orgs/kubernetes/teams/sig-storage @jingxu97
https://github.com/jingxu97


You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
#121 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVCoNyQWhL1_AAMD0eE1Uh79jaMWsks5qyxIggaJpZM4KR2_S
.

@idvoretskyi idvoretskyi modified the milestone: v1.5 Oct 11, 2016

@jingxu97

This comment has been minimized.

jingxu97 commented Oct 11, 2016

Yes, I will definitely spend more time on this after this short release
cycle for 1.5. Thanks!

Jing

On Mon, Oct 10, 2016 at 10:53 PM, Tim Hockin notifications@github.com
wrote:

I think we can commit to that.

On Mon, Oct 10, 2016 at 9:35 PM, Prashanth B notifications@github.com
wrote:

It would be great if we were ready to go at this in 1.6
@kubernetes/sig-storage
https://github.com/orgs/kubernetes/teams/sig-storage @jingxu97
https://github.com/jingxu97


You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/kubernetes/features/issues/
121#issuecomment-252810762>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVCoNyQWhL1_
AAMD0eE1Uh79jaMWsks5qyxIggaJpZM4KR2_S>
.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#121 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASSNxQlkIyJ98tTTTqNRkJh2JB3OtI-jks5qyyRJgaJpZM4KR2_S
.

  • Jing

@idvoretskyi idvoretskyi modified the milestones: next-milestone, v1.5 Oct 13, 2016

@davidopp davidopp referenced this issue Nov 7, 2016

Closed

Data awareness and Data/Pod affinity #139

0 of 23 tasks complete

@vishh vishh assigned vishh and msau42 and unassigned smarterclayton Jan 18, 2017

@vishh

This comment has been minimized.

Member

vishh commented Jan 18, 2017

@msau42 and myself will be tackling this problem in v1.6. The plan is as follows:

  1. Design completion by v1.6
  2. Alpha support for local storage by v1.7
  3. Beta (or may be GA) by v1.8

@rkouj and @jingxu97 will also be helping with this effort.

@msau42 and @vishh will post a high level design proposal before end of Jan `17.

@vishh vishh changed the title from Local storage for use with PetSets and bare-metal clusters to Local storage Management (ephemeral & durable) Jan 18, 2017

@davidopp

This comment has been minimized.

Member

davidopp commented Jan 24, 2017

Please loop me in on the scheduling aspects of this.

@idvoretskyi idvoretskyi modified the milestones: v1.6, next-milestone Jan 26, 2017

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Jan 26, 2017

Misread the @vishh comment, let's target 1.7 when this milestone will be open.

@wattsteve

This comment has been minimized.

wattsteve commented Jan 27, 2017

@vishh and others. Given that we're already talking about targeting releases, is there a design proposal for this? FYI @saad-ali @childsb

@vishh

This comment has been minimized.

Member

vishh commented Jan 27, 2017

@krmayankk

This comment has been minimized.

krmayankk commented Jan 29, 2017

@vishh has this anything to do with sticky empty dir proposal or local storage for statefulset ? I am getting confused which feature is doing what ? There are many things needed for local storage , e.g. 1: local storage support in statefulset 2: local storage volumes which is this one 3: pod should be scheduled in the same node where their data is already available 4: may be moving data to where the pod gets scheduled . It will be good to have one local storage featur which covers all related issues and proposals

@vishh

This comment has been minimized.

Member

vishh commented Jan 29, 2017

@krmayankk the plan is to cover all supported use cases for local storage.

@davidopp

This comment has been minimized.

Member

davidopp commented Jan 29, 2017

@vishh I assume you are not including (4) from @krmayankk's comment? (migrating data between nodes)

@idvoretskyi idvoretskyi added stage/beta and removed stage/alpha labels Jan 29, 2018

k8s-merge-robot added a commit to kubernetes/kubernetes that referenced this issue Feb 7, 2018

Merge pull request #59303 from dhirajh/localblock
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Block Volume Support: Local Volume Plugin update

**What this PR does / why we need it**:

Introduce block volume support to local volumes plugin.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #59500

**Special notes for your reviewer**:
@msau42 @mtanino @ianchakeres 

Adding support for block volumes as per kubernetes/enhancements#121

Other related PRs:
(#50457) API Change
(#53385) VolumeMode PV-PVC Binding change
(#51494) Container runtime interface change, volumemanager changes, operationexecutor changes

**Release note**:
```
Added support for Block Volume type to local-volume plugin.
```
@msau42

This comment has been minimized.

Member

msau42 commented Feb 27, 2018

1.10 Beta PR has been merged: kubernetes/kubernetes#59391

@ryanwalls

This comment has been minimized.

ryanwalls commented Apr 13, 2018

Is there documentation for how to use this feature? Can't seem to find it.

@msau42

This comment has been minimized.

Member

msau42 commented Apr 13, 2018

@ryanwalls try this: https://kubernetes.io/docs/concepts/storage/volumes/#local

It also has a link to the external provisioner page, which also has a walk through guide using the static provisioner.

@ryanwalls

This comment has been minimized.

ryanwalls commented Apr 17, 2018

@msau42 Any thoughts on if local-storage would work with StatefulSets? Examples? The biggest question I have is what to do with the nodeAffinity requirement.

@msau42

This comment has been minimized.

Member

msau42 commented Apr 17, 2018

@ryanwalls this feature is designed to work with StatefulSets. Can you describe in more detail your question about nodeAffinity?

@ryanwalls

This comment has been minimized.

ryanwalls commented Apr 17, 2018

@msau42

I'm trying to use the example here: https://kubernetes.io/docs/tutorials/stateful-application/cassandra/ to deploy a cassandra statefulset, but using local storage.

My concern is that there are multiple nodes that will have the statetfulset and in the local-storage docs it implies you have to have a 1 to 1 relationship between volume claims and nodes.

The docs say:

PersistentVolume nodeAffinity is required when using local volumes. It enables the Kubernetes scheduler to correctly schedule pods using local volumes to the correct node.

Here's the template I've come up with for the statefulset. Think it will work?

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        image: gcr.io/google-samples/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
           cpu: "500m"
           memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local-storage
      local:
          path: /mnt/disks/ssd1
      nodeAffinity:
        required:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
              - cassandra-node1
              - cassandra-node2
              - cassandra-node3
      resources:
        requests:
          storage: 1Gi
@ryanwalls

This comment has been minimized.

ryanwalls commented Apr 17, 2018

@msau42 Everything I'm reading implies that local-storage does not support dynamic provisioning yet. https://kubernetes.io/docs/concepts/storage/storage-classes/#local. So I'm assuming using a volumeClaimTemplates array will not work?

@msau42

This comment has been minimized.

Member

msau42 commented Apr 17, 2018

@ryanwalls this spec won't work. You don't actually specify the local volume directly in the statefulset.

Check out these slides I made for the cncf K8s 1.10 webinar. It walks through the whole process with a mini example: creating a storage class, creating your PVs, and how to use it in a StatefulSet. The Youtube video is also available here, skip to 12:36 mark.

@ryanwalls

This comment has been minimized.

ryanwalls commented Apr 17, 2018

@msau42 Awesome!! Super helpful. Thank you!

@msau42

This comment has been minimized.

Member

msau42 commented Apr 17, 2018

Yw, and please feel free to ping me on slack if you need any more help (I'm in pst)

@ryanwalls

This comment has been minimized.

ryanwalls commented Apr 17, 2018

@msau42 Got everything working. One more question. Does a local-storage PV have to map 1 to 1 to a node when specifying nodeAffinity?

e.g. I am using a general-purpose tag for some nodes. If multiple nodes have that tag will this PV definition be problematic?

apiVersion: v1
kind: PersistentVolume
metadata:
  name: cassandra-data-0
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: 3dsim.com/type
          operator: In
          values:
          - general-purpose
@msau42

This comment has been minimized.

Member

msau42 commented Apr 17, 2018

If you want your pod to be able to access the exact same disk/data if it restarts then yes.

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 17, 2018

@vishh @msau42
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

  • Description
  • Milestone
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

cc @idvoretskyi

@msau42

This comment has been minimized.

Member

msau42 commented Apr 17, 2018

Plan for 1.11 is to remain in beta.

LVM dynamic provisioning is being tracked in #562

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 17, 2018

Thanks @msau42!

@justaugustus justaugustus modified the milestones: v1.10, next-milestone Apr 20, 2018

@justaugustus justaugustus removed this from the next-milestone milestone Jul 2, 2018

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 30, 2018

This feature current has no milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.


Please note that Features Freeze is tomorrow, July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

P.S. This was sent via automation

@msau42

This comment has been minimized.

Member

msau42 commented Jul 30, 2018

Plan for 1.12 is to remain in beta.

justaugustus pushed a commit to justaugustus/features that referenced this issue Sep 3, 2018

Merge pull request #1054 from msau42/storage-topology-design
Automatic merge from submit-queue.

Volume topology aware scheduling design

Proposal for a smarter scheduler that influences PV binding.

Part of kubernetes#121

/sig storage
/sig scheduling
/cc @kubernetes/sig-storage-proposals @kubernetes/sig-scheduling-proposals
@kacole2

This comment has been minimized.

Contributor

kacole2 commented Oct 8, 2018

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

  • Docs (open placeholder PRs): 11/8
  • Code Slush: 11/9
  • Code Freeze Begins: 11/15
  • Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

@msau42

This comment has been minimized.

Member

msau42 commented Oct 8, 2018

Plan for 1.13 is to remain in beta. We are looking for more feedback on the feature before graduating to GA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment