Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple pods able to write to PV with accessMode of ReadWriteOnce #60903

Open
ksexton opened this issue Mar 7, 2018 · 24 comments
Open

Multiple pods able to write to PV with accessMode of ReadWriteOnce #60903

ksexton opened this issue Mar 7, 2018 · 24 comments

Comments

@ksexton
Copy link

@ksexton ksexton commented Mar 7, 2018

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
Create ReplicaSet with multiple pods mounting PV set with accessMode of ReadWriteOnce. Then verified that each pod could write successfully to the PV.

What you expected to happen:
Either the replicaset to fail because it was trying to do something invalid (mount a RWO volume to multiple pods) or to succeed but only mount to one pod.

How to reproduce it (as minimally and precisely as possible):
Here is the manifest I applied that is pointing to a local NFS server.

apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
  name: busybox
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: busybox
        version: "2"
    spec:
      containers:
        - image: busybox
          args: [/bin/sh, -c, 'sleep 9999' ]
          volumeMounts:
            - mountPath: /test
              name: busybox-volume
          name: busybox
      volumes:
      - name: busybox-volume
        persistentVolumeClaim:
          claimName: busybox-claim
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: busybox-volume
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /opt/test
    server: 192.168.1.24
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: busybox-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Demonstration of issue

Anything else we need to know?:
This may be working as intended and the accessMode is just a suggestion. If that is the case then this should probably be a ticket to update documentation at https://kubernetes.io/docs/concepts/storage/persistent-volumes/ with notes that accessControl doesn't do any enforcement.

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:54Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    Local VMware lab
  • OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
  • Kernel (e.g. uname -a):
    Linux kubernetes-alpha 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:
    kubeadm init --pod-network-cidr=10.244.0.0/16

  • Others:

@ksexton

This comment has been minimized.

Copy link
Author

@ksexton ksexton commented Mar 7, 2018

/sig storage

@k8s-ci-robot k8s-ci-robot added sig/storage and removed needs-sig labels Mar 7, 2018
@tpepper

This comment has been minimized.

Copy link
Contributor

@tpepper tpepper commented Mar 9, 2018

While not 1.10 specific, if this is deemed not just a doc's issue and is not a "working as intended" but rather a data consistency issue, @kubernetes/sig-storage-bugs please mark this asap if you deem it a release blocker for 1.10.

@justinbarrick

This comment has been minimized.

Copy link

@justinbarrick justinbarrick commented Mar 22, 2018

I'm very curious about this because the docs mention nodes, but don't mention pods at all. So it seems like this is working as intended to me, but I'd really like to know if the behavior is going to change on us.

https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes

    ReadWriteOnce – the volume can be mounted as read-write by a single node
    ReadOnlyMany – the volume can be mounted read-only by many nodes
    ReadWriteMany – the volume can be mounted as read-write by many nodes
@orainxiong

This comment has been minimized.

Copy link

@orainxiong orainxiong commented Mar 28, 2018

ReplicaSet can't ensure the at-most-one guarantee we wish.

In order to preserve the availability of ReplicaSet's pods, ReplicaSet will recreate a new pod when it found an old pod in the termination/deleting process.

But, you know, all operations are asynchronize. A pod in the termination/deleting process would continue to run for an arbitrary amount of time. As a result of that, it very likely to happen that multiple copies of one pod are running concurrently.

Deployment, building atop ReplicaSet, can't offer the at-most-one guarantee too.

@ksexton

@msau42

This comment has been minimized.

Copy link
Member

@msau42 msau42 commented Mar 29, 2018

AccessModes as defined today, only describe node attach (not pod mount) semantics, and doesn't enforce anything.

@msau42

This comment has been minimized.

Copy link
Member

@msau42 msau42 commented Mar 29, 2018

@orainxiong StatefulSet pods may be better in this regard. Because the pods are recreated with the same stable name, a replacement pod cannot be created until the old one is deleted. However, you could still do things like pod force delete, which won't wait for volumes to gracefully unmount from a node.

@orainxiong

This comment has been minimized.

Copy link

@orainxiong orainxiong commented Mar 29, 2018

@msau42 Thanks for your suggestion.

BTW, It seems very likely to create two copies of one pod during cluster partition even with StatefulSet during master-node partition happens.

There should be a fencing policy that is able to reconcile master-node partition and enforce at-most-one guarantee to protect pod against data corruption.

more details : #61832

@djschny

This comment has been minimized.

Copy link

@djschny djschny commented Jun 1, 2018

Thanks for the discussion here, I recently thought there was a bug in the documentation as well since it talks about nodes instead of pods for Access Modes.

What concerns me the most here is that from a user perspective it is all presented in an abstracted way between the Pod spec, PVC, and PV that makes it look like it’s all pod level, but it’s not.

In this secnario, if a PV has and accessMode: ReadWriteOnce and it is bound to a PVC and there is currently one Pod bound to the PVC. When a second pod tries to bind to the PVC, I would expect that pod creation to fail/error at that point, even before it may get scheduled to the same node.

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Aug 30, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Sep 29, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Oct 29, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Oct 29, 2018

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Dec 21, 2018

@kubanczyk: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@melwynjensen

This comment has been minimized.

@lojies

This comment has been minimized.

Copy link
Contributor

@lojies lojies commented Jun 26, 2019

Had the same question.I can create different pods in different nodes with the same pvc.But i thought this should be error because i had set the accessmode as ReadWriteOnce.Is this a bug or i have a missunderstand.

[root@controller213:~/]$ cat mongodb-pv-nfs.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs01
spec:
  capacity:
    storage: 200Mi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 10.46.178.117
    path: "/home/nfs/nfs01"
[root@controller213:~/]$ cat mongodb-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongodb-pvc
spec:
  resources:
    requests:
      storage: 200Mi
  accessModes:
  - ReadWriteOnce
  storageClassName: ""
[root@controller213:~/]$ kubectl get pvc
NAME          STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongodb-pvc   Bound     nfs01     200Mi      RWO                           1h
[root@controller213:~/]$ cat busybox-rc.yml
apiVersion: v1
kind: ReplicationController
metadata:
  name: busybox
spec:
  replicas: 6
  selector:
    app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: 10.46.177.91:5000/busybox:latest
        volumeMounts:
        - name: mongodb-data
          mountPath: /data/busybox
        command:
         - sleep
         - "3600"
      volumes:
      - name: mongodb-data
        persistentVolumeClaim:
          claimName: mongodb-pvc

[root@controller213:~/]$ kubectl get pods -o wide
NAME            READY     STATUS    RESTARTS   AGE       IP               NODE
busybox-bzlkj   1/1       Running   0          59m       172.22.22.216    10.46.179.213
busybox-hqnpn   1/1       Running   1          1h        172.22.55.217    10.46.179.218
busybox-n78tm   1/1       Running   1          1h        172.22.22.193    10.46.179.213
busybox-ncx2z   1/1       Running   0          59m       172.22.154.134   10.46.179.214
busybox-v6tck   1/1       Running   1          1h        172.22.154.133   10.46.179.214
busybox-wzrtf   1/1       Running   0          59m       172.22.55.194    10.46.179.218
@zouyee

This comment has been minimized.

Copy link
Member

@zouyee zouyee commented Jul 15, 2019

/reopen

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Jul 15, 2019

@zouyee: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Jul 15, 2019
@TsubasaEX

This comment has been minimized.

Copy link

@TsubasaEX TsubasaEX commented Jul 19, 2019

So the question, is it about two nodes can't access a same PV or two pods can't access a same PV with RWO? Or both? Came into the same question.

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Oct 17, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@jkugler

This comment has been minimized.

Copy link

@jkugler jkugler commented Oct 17, 2019

/remove-lifecycle stale

@vvbogdanov87

This comment has been minimized.

Copy link

@vvbogdanov87 vvbogdanov87 commented Dec 6, 2019

Is there a way to prevent two pods mount the same PVC even if they are scheduled to be run on the same node?

@vvbogdanov87

This comment has been minimized.

Copy link

@vvbogdanov87 vvbogdanov87 commented Dec 9, 2019

The answer to my own question is pod anti-affinity. It is not the same as not to mount one volume to 2 pods scheduled on the same node. But anti-affinity can be used to ask scheduler not to run 2 pods on the same node. Therefore it prevents mounting one volume into 2 pods.

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Mar 8, 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@jkugler

This comment has been minimized.

Copy link

@jkugler jkugler commented Mar 8, 2020

/remove-lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.