Critical: Possible data loss during rebalance if there is any linkfile on the system (gluster-11 version)

[rafikc30]

Description of problem:
The cause of the data loss was the Rebalance daemon mistakenly deleting a data file instead of a linkfile, resulting in the deletion of the entire file from the gluster namespace and causing complete data loss. In addition to the file being deleted, a modification to the data file can also cause data loss.

This data loss can occur in the following situations:

Deletion of the entire file:
This situation arises when there is a linkfile present on the hashed subvol. In this case, the rebalance daemon adds both the linkfile and the data file to the queue. Suppose the data file entry is picked first for migration and successfully migrated. When the linkfile is subsequently picked from the queue, the rebalance daemon checks whether the added linkfile has both hashed and cached to the same subvol. Mistakenly assuming it is a linkfile, the daemon deletes it. The check for the linkfile is based on the stbuf populated during the rebalance readdir. However, by the time the check is performed, the file has already been migrated and has become a data file.

Incorrect permissions and size:
Similar to the previous case, this occurs when a linkfile and a data file are selected for migration simultaneously. In this situation, a failure can occur at various stages of the rebalance process. Due to this failure, a cleanup operation might be performed on the destination where the rebalance process is incomplete. However, by the time the first file is completed and the cleanup is executed, it is performed on an actual file, resulting in incorrect permissions (e.g., "") or an incomplete data file with incorrect information.

The exact command to reproduce the issue:

The full output of the command that failed:

Expected results:

Mandatory info:
- The output of the gluster volume info command:

- The output of the gluster volume status command:

- The output of the gluster volume heal command:

**- Provide logs present on following locations of client and server nodes -
/var/log/glusterfs/

**- Is there any crash ? Provide the backtrace and coredump

Additional info:
A possible scenrio to have link file include rename operation, The used space on one of the replica set is above min-free-disk(90% by default)

- The operating system / glusterfs version:
gluster11

Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration