Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Glusterfs wrong server list in IPv6 case #78900

Open
aga20 opened this issue Jun 11, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@aga20
Copy link

commented Jun 11, 2019

What happened:
We are using kubernetes with IPv6 host network. The glusterfs cluster what we are using is running on this network too (and all endpoint is created with this IPv6 address). I have these endpoints:

[root@node-1 ~]# kubectl get ep gluster-service
NAME              ENDPOINTS                                            AGE
gluster-service   [fd00:16::104]:1,[fd00:16::106]:1,[fd00:16::107]:1   84m

And the mount command from a live env ps command:

[root@node-1 ~]# ps axuf | grep gluster
/usr/sbin/glusterfs --log-level=ERROR --log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/oam/troublebot-hv7qc-glusterfs.log --fuse-mountopts=auto_unmount --xlator-option=transport.address-family=inet6 --process-name fuse --volfile-server=fd00:16::104 --volfile-server=fd00 --volfile-server=16 --volfile-server=104 --volfile-server=fd00 --volfile-server=16 --volfile-server=106 --volfile-server=fd00 --volfile-server=16 --volfile-server=107 --volfile-id=oam --fuse-mountopts=auto_unmount /var/lib/kubelet/pods/47758365-8c22-11e9-bc5f-fa163e29254e/volumes/kubernetes.io~glusterfs/oam

So as you can see the glusterfs mount helper script split the ipv6 addresses and keep only one (this is why the mount work but when the server is not reachable the podstart can fail).

What you expected to happen:
The mount command must use correct IPv6 addresses.

How to reproduce it (as minimally and precisely as possible):
Run kubernetes/glusterfs combo on IPv6 network .

Anything else we need to know?:
Detailed problem description and possible solution (localy we are using this solution so this can work).
Problem: as I see the glusterfs plugin create a mount -t glusterfs... command and the mount helper script split this parameter:
Source: https://www.systutorials.com/docs/linux/man/8-mount.glusterfs/

backup-volfile-servers=SERVERLIST
   Provide list of backup volfile servers in the following format [default: None]
   $ mount -t glusterfs -obackup-volfile-servers=<server2>: <server3>:...:<serverN> <server1>:/<volname> <mount_point>

So this list is using : as the separator what is not exactly good in IPv6 case.

Possible solution:
Don't use backup-volfile-servers parameter instead of this based on the manual you can use the following format:

mount -t glusterfs [-o <options>] <server1>,<server2>,<server3>,..<serverN>:/<volname> <mount_point> 

This can work but maybe some of the glusterfs version based on the doc support this format but can't handle because of a wrong regex in the mount helper script. (I will raise a glusterfs ticket too and link it under this thread).

Environment:

  • Kubernetes version (use kubectl version): v1.12.6
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release): CentOS Linux 7 (Core)
  • Kernel (e.g. uname -a): 4.4.179-1.el7.elrepo.x86_64 #1 SMP Sat Apr 27 08:29:04 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: -
  • Network plugin and version (if this is a network-related bug): Host network, not relevant
  • Others:

@aga20 aga20 added the kind/bug label Jun 11, 2019

@aga20

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

/sig storage

@k8s-ci-robot k8s-ci-robot added sig/storage and removed needs-sig labels Jun 11, 2019

@aga20

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

@aga20

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

If you agree with this I can fix it.

@aga20

This comment has been minimized.

Copy link
Author

commented Jun 11, 2019

Relevant glusterfs bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1719290
Please note this is only affect IPv6 case!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.