NDM using extremely high memory

[tathougies]

What steps did you take and what happened:

I have a 64 GB server as part of my kubernetes cluster. I'm running openebs on the cluster. NDM is running on all nodes, but on this node, it's eating up 50% of the total memory. Examining the logs shows that it seems to be stuck in some kind of loop

This just repeats over and over.

What did you expect to happen:

On my other servers, ndm is reported as taking up <0.01% of memory, which is what I expect here

The output of the following commands will help us better understand what's going on: https://gist.github.com/tathougies/70f8cb54929ea890c07896e0b48616ff

• kubectl get pods -n openebs
• kubectl get blockdevices -n openebs -o yaml
• kubectl get blockdeviceclaims -n openebs -o yaml
• kubectl logs <ndm daemon pod name> -n openebs
• lsblk from nodes where ndm daemonset is running

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• OpenEBS version: 2.12.1
• Kubernetes version (use kubectl version): version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"archive", BuildDate:"1980-01-01T00:00:00Z", GoVersion:"go1.16.13", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version: Using stock kubernetes NixOS
• Cloud provider or hardware configuration: Custom
• Type of disks connected to the nodes (eg: Virtual Disks, GCE/EBS Volumes, Physical drives etc): 2.5" SAS SSDs connected to RAID controller in pass-thru mode
• OS (e.g. from /etc/os-release):

NAME=NixOS
ID=nixos
VERSION="21.11 (Porcupine)"
VERSION_CODENAME=porcupine
VERSION_ID="21.11"
BUILD_ID="21.11.336386.bacbfd713b4"
PRETTY_NAME="NixOS 21.11 (Porcupine)"
LOGO="nix-snowflake"
HOME_URL="https://nixos.org/"
DOCUMENTATION_URL="https://nixos.org/learn.html"
SUPPORT_URL="https://nixos.org/community.html"
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"

[Paramoshka]

Found a solution?

[Paramoshka]

openebs/openebs#3564 (comment)