-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transport endpoint is not connected when csi-s3 pod is restarted #153
Comments
Facing the same issue |
Rolling out daemonsets and dataset-operator in dlf namespace altogether fixed this issue for me |
Actually restarting the operator did not fix the issue for me, the only thing that fixes the issue is to restart the pod that uses the pvc created by dataset operator, but would be better if when the daemonset or operator restarts reconciles the mount, otherwise each time we update the csi provider to a new version connectivity will be lost on all pods PS: the issue is happening for goofys and s3fs mounters Scenario to reproduce: attacher logs Can someone please take a look on this ? |
Verified that this problem exists. To solve this, the CSI-S3 driver would need to be extended to support This will be a sizeable development, so not sure about the timelines yet. |
Tried adding an extra argument |
RPC_LIST_VOLUMES_PUBLISHED_NODES is officially not a solution :-) kubernetes-csi/external-attacher#374 (comment) |
@vitalif Thanks for researching this issue, though the answer is disappointing :-) CSI-S3 (atleast Datashim's fork) uses If you do have a workaround, I'll be happy to look into it. |
Any update on when this will be resolved. We are also facing this issue. We are getting this very frequently, we are mounting s3 to 5-6 pods. Whenever we do some read or write, we are getting this error, And we have to restart frequently |
@rrehman-hbk Could I ask under what conditions are you getting errors for read/write from S3 buckets ? This is a different problem from the one above. If you can create an issue and post the logs from your |
@srikumar003 #324 |
Also of note in this case for me if I have a livenessProbe kill the container it cannot just pickup from where it left off the whole pod must be destroyed. The CSI-S3 deamon reports the following
If the pod is subsequently restarted the mount the succeeds and all is fine again. |
+1 |
If for any reason the csi-s3 pod is restarted, the Pod that uses s3 volumes looses connectivity to the mount target and we get
Transport endpoint is not connected
errorThe error is solved if we restart the pod that uses the volume this forces csi-s3 pod to remount the volume.
I think when csi-s3 restarts it should check for existing volumes and remount the volume.
To reproduce this behaviour just rollout restart the deamonset
Could you please take a look ?
The text was updated successfully, but these errors were encountered: