Automatically configure the backup volfile servers in clients

[poornimag]

Currently, both FUSE and gfapi provides commandline options and api to set backup volfile servers. WRT to gfapi, every application that integrates with gfapi has to provide a way for user to input these volfile server hostnames, and call glfs_set_volfile_servers with these hostnames. Instead, we can have an RPC implemented in client and glusterd, which fetches the list of volfile severs available and gfapi automatically configures these as backup volfile servers.

One thing is, if clients use a different network(IP) to communicate to glusterd, than the IPs used by glusterd to communicate with each other, we may end up configuring wrong volfile server IPs?. Any thoughts? @nixpanic @soumyakoduri @anoopcs9

[prashanthpai]

FYI: This is has been considered and is planned in glusterd2. We may not need the CLI option at all in future. On connection to first/one glusterd, glusterd can provide the list of peers which the client can store. Glusterd shall also notify clients when there are are new peers.

[gluster-ant]

A patch https://review.gluster.org/19504 has been posted that references this issue.
Commit message: Fetch backup volfile servers from glusterd2

[ShyamsundarR]

Dropping a thought here, In the current form the CLI functionality is the same as connecting to the first glusterd(1/2) instance and fetching the server list.

The older CLI, as well, did not have the functionality (to my knowledge) that when backup volfile servers are present, and connection to the primary fails, it would/could use one of the backup volfile servers. The backup volfile servers were used after the connection to the primary succeeded (I could be wrong).

In the above case there is still a SPOF, if the primary server provided is down. Similarly, fetching this list from the first server will not help, if that is down.

What this possibly means is that we need to enhance the CLI option to consider fetching from backup servers in case primary is down. What this possibly also means, is that even gfapi based applications to ensure SPOF failures may need to have an option to provide a backup list from an availability perspective.

[prashanthpai]

To summarize, to avoid SPOF, glusterfs clients need:

1. ability to specify multiple glusterd servers during mount (fuse, gfapi)
2. a way to automatically and periodically update this list of glusterd servers after initial connection establishment

1 is satisfied with:

• availability of backupvolfile-server CLI option (FUSE)
• ability to call glfs_set_volfile_server multiple times (gfapi apps)

2 is implemented in:

• https://review.gluster.org/19504 (glusterfs)
• HA: Return list of gd2 servers in GETSPEC response glusterd2#540 (glusterd2)

On a related note, I assume users can get some form of crude HA if volfile server configured with round-robin DNS.

[ShyamsundarR]

To summarize, to avoid SPOF, glusterfs clients need:

ability to specify multiple glusterd servers during mount (fuse, gfapi)

Yes, but also to go through this list instead of only one during the mount,

For example will this mount work, if the server1 is down, when the mount command is invoked (syntax maybe a bit off)? If it will, then we are good, else we need to possibly use the backup volfile server list to make the mount happen first, and the part (2) comes into play.
mount -t glusterfs -o backupvolfile-server=server2 server1:myvol /mnt/myvol

a way to automatically and periodically update this list of glusterd servers after initial connection establishment

1 is satisfied with:

iff the above is satisfied, which is my confusion.

availability of backupvolfile-server CLI option (FUSE)
ability to call glfs_set_volfile_server multiple times (gfapi apps)

2 is implemented in:

https://review.gluster.org/19504 (glusterfs)
gluster/glusterd2#540 (glusterd2)

On a related note, I assume users can get some form of crude HA if volfile server configured with round-robin DNS.

Yes, did think about this, in which case we do not need either 1/2, but then it is always better to have in-built tolerance in such cases, especially in container farm environments.

[nixpanic]

One thing is, if clients use a different network(IP) to communicate to glusterd, than the IPs used by
glusterd to communicate with each other, we may end up configuring wrong volfile server IPs?. Any
thoughts?

Indeed, this might be a problem for clients that are outside the local (protected) storage network. There are users that have a split-horizon DNS which resolves hostnames different depending on the client(-network).

As long as the implementation returns hostnames instead of IP-addresses, things should work correctly. It it possible that IP-addresses of the internal network connections are not routed outside the storage internal network, clients should not try to use those. I think this also is assumed for secured connections with TLS, which is something that should become the default at one point.

[prashanthpai]

glusterd2 uses different ports for

1. client to glusterd2 communication (defaults to 24007)
2. glusterd2 to glusterd2 communication

Both the interfaces used and ports are configurable.

This document describes the current state of things in glusterd2:
https://github.com/gluster/glusterd2/blob/master/doc/network.md

As long as the implementation returns hostnames instead of IP-addresses, things should work correctly

Good point. This is something that glusterd2 has ignored so far and needs to be fixed