Issue with glusterfs replicated volume mount

[Almeida18]

The issue is a little bit complicated to explain straightforward so i'll try to split it.

I have a 3 gluster nodes hosting replicated volumes. The gluster daemons are up and running and the nodes are in peers as shown:

gluster peer status
Number of Peers: 2

Hostname: glustervm2
Uuid: 85ebe7a3-7d14-4fae-9e0f-5963ade08122
State: Peer in Cluster (Connected)

Hostname: glustervm3
Uuid: 5ff5a23b-38c2-4c7e-8f37-e5f485e496db
State: Peer in Cluster (Connected)
root@glustervm1:~#

Output of targetted volume:

Status of volume: applications
Gluster process TCP Port RDMA Port Online Pid

Brick glustervm1:/data/glusterfs/applications/brick1/brick 59678 0 Y 452022
Brick staging-glustervm2:/data/glusterfs/applications/brick1/brick 53239 0 Y 523282
Brick staging-glustervm3:/data/glusterfs/applications/brick1/brick 51821 0 Y 702749
Self-heal Daemon on localhost N/A N/A Y 1836412
Self-heal Daemon on glustervm2 N/A N/A Y 2126964
Self-heal Daemon on glustervm3 N/A N/A Y 2136930

Task Status of Volume applications

There are no active volume tasks

I wish to mount a volume on a 3 clients. When i issue the command to mount on the first client (client 1) as follows (i perform a manual mount but mounting automatically still gives the same result):
mount -t glusterfs glustervm1:/applications /mnt/glusterfs/applications

When issuing the exact same command on client 2 , i have the error "Mounting glusterfs on /mnt/glusterfs/applications". But when mounting via another gluster node (glustervm2 to be precise), it works. The same scenario on client 3 (client 3 only succeeds its mount on on node )glustervm3.

So to resume, client 1 accepts mount from glustervm1, client 2 accepts mount from glustervm1, client 3 accepts mount from glustervm3.

When mounts succeeds through this method however, accessing or copying files into the mount point that corresponds to the volume issues "Transport endpoint not connected" (note that ping to each of the gluster node is successfull, both via IP and hostname)

The output of the logs on gluster nodes are as follows:

[2025-04-03 10:44:08.398159 +0000] I [MSGID: 106496] [glusterd-handshake.c:954:__server_getspec] 0-management: Received mount request for volume /applications
[2025-04-03 10:45:13.934250 +0000] I [MSGID: 106496] [glusterd-handshake.c:954:__server_getspec] 0-management: Received mount request for volume /applications
[2025-04-03 10:46:19.471343 +0000] I [MSGID: 106496] [glusterd-handshake.c:954:__server_getspec] 0-management: Received mount request for volume /applications

Note that this is an isolated case, rarely happens but it has been months since i carried this issue
I'm hinting a network issue in the environment in which these nodes are but i am not sure. Nodes are running in debian 12
Any ideas?

[pranithk]

@Almeida18 Could you attach the mount logs when the issue happens again? But does look like network issue.