[BUG] Failed replicas not being cleared from node

[dbpolito]

Describe the bug

I got a instability on one node that made some replicas to corrupt and got rebuilt, but it seems that the disk usage wasn't freed by these replicas that failed:

Environment:

• Longhorn version: 1.1.0
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: EKS

Not sure that's really the problem but the only thing i noticed is this node disk increase after all these replica rebuilds...

[khushboo-rancher]

@dbpolito Can you check the staleReplicaTimeout for you storage class, the failed replica remains on the disk for sometime based on the staleReplicaTimeout.

[dbpolito]

I don't remember customizing that, so i guess it's the default: staleReplicaTimeout: "30"

That's not even customizable on helm chart: https://github.com/longhorn/charts/blob/master/charts/longhorn/templates/storageclass.yaml#L21

What is this? minutes? hours? days?

[cclhsu]

staleReplicaTimeout: "30" is in minutes as in example of https://longhorn.io/docs/1.1.0/volumes-and-nodes/create-volumes/

[dbpolito]

I see, well, after a few days i still have this huge difference between nodes:

Which i can't tell what it is... i'm guessing it's related to these failed replicas because this node had that and the size increased during that period of rebuild and never freed.

[jenting]

Could you help to check if there any orphan replicas directory under the host path /var/lib/longhorn/replicas/ to see if there any mismatch volume replicas?
If yes, you could manually delete the orphan replicas on the host.

Right now, if the node even went down and back, Longhorn has no information anymore which is the orphan replicas (i.e., Longhorn did not scan the /var/lib/longhorn/replicas/ now). We'll put it in our backlog to see how we can enhance it.

[joshimoo]

I left some quick thoughts on how replica deletion could be improved to ensure cleanup of replica data here:
#685 (comment)