enlarge NPD memory limit to 150Mi to avoid OOM

[pacoxu]

What type of PR is this?

/kind flake

What this PR does / why we need it:

I0225 17:35:11.161300 20814 ooms_tracker.go:284] OOM detected: &Event{ObjectMeta:{gce-scale-cluster-minion-heapster.17b72a4a62c4a7d8 default d60b06ff-11d1-4cb8-be1f-7264e084769b 85768 0 2024-02-25 17:11:22 +0000 UTC map[] map[] [] [] [{node-problem-detector Update v1 2024-02-25 17:11:22 +0000 UTC FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},"f:host":{}},"f:type":{}} }]},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:gce-scale-cluster-minion-heapster,UID:gce-scale-cluster-minion-heapster,APIVersion:,ResourceVersion:,FieldPath:,},Reason:OOMKilling,Message:Memory cgroup out of memory: Killed process 4550 (metrics-server) total-vm:834776kB, anon-rss:100100kB, file-rss:33900kB, shmem-rss:0kB, UID:65534 pgtables:400kB oom_score_adj:999,Source:EventSource{Component:kernel-monitor,Host:gce-scale-cluster-minion-heapster,},FirstTimestamp:2024-02-25 17:11:22 +0000 UTC,LastTimestamp:2024-02-25 17:11:22 +0000 UTC,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}

Does this indict the oom of metrics server or the node problem detector?

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1761798551869853696/build-log.txt

I0225 17:35:11.161639   20814 ooms_tracker.go:284] OOM detected: &Event{
    ObjectMeta:{
        gce-scale-cluster-minion-heapster.17b72a5024e5e2b7  default  f318a878-4b13-4e0b-a19d-a58bdbaaa3a8 155636 0 2024-02-25 17:11:47 +0000 UTC <nil> <nil> map[] map[] [] []  
        [
            {
                node-problem-detector Update v1 2024-02-25 17:11:47 +0000 UTC 
                FieldsV1 {"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:source":{"f:component":{},  "f:host":{}},"f:type":{}} 
            }
        ]},
        InvolvedObject:ObjectReference{
                Kind:Node,
                Namespace:,Name:gce-scale-cluster-minion-heapster,
                UID:gce-scale-cluster-minion-heapster,
                APIVersion:,
                ResourceVersion:,FieldPath:,
            },
        Reason:OOMKilling,
        Message: Memory cgroup out of memory: Killed process 5072 (metrics-server) total-vm:829304kB, anon-rss:103180kB, file-rss:33136kB, shmem-rss:0kB, UID:65534 pgtables:380kB oom_score_adj:999,Source:EventSource{Component:kernel-monitor,Host:gce-scale-cluster-minion-heapster,},FirstTimestamp:2024-02-25 17:11:47 +0000 UTC,LastTimestamp:2024-02-25 17:11:47 +0000 UTC,
        Count:1,
        Type:Warning,
        EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,
    }

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1761798551869853696/artifacts/SystemPodMetrics_access-tokens_2024-02-25T19:27:23Z.json

    {
      "name": "node-problem-detector-qrt5w",
      "containers": [
        {
          "name": "node-problem-detector",
          "restartCount": 1,
          "lastRestartReason": "\u0026ContainerStateTerminated{ExitCode:255,Signal:0,Reason:Unknown,Message:,StartedAt:2024-02-25 17:24:13 +0000 UTC,FinishedAt:2024-02-25 18:19:59 +0000 UTC,ContainerID:containerd://685c8897aa5538550976c05ee7e23fc8a6d8325d9d6e0fcf183556d621cac0df,}"
        }
      ]
    },

Which issue(s) this PR fixes:

Fixes #123328

Special notes for your reviewer:

Does this PR introduce a user-facing change?

None

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pacoxu
Once this PR has been reviewed and has the lgtm label, please assign random-liu for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cluster/addons/node-problem-detector/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pacoxu]

#123328
It flakes 3 times in recent runs for npd with OOM-caused restart.

/cc @SergeyKanzhelev @bart0sh

[pacoxu]

https://perf-dash.k8s.io/#/?jobname=gce-5000Nodes&metriccategoryname=SystemPodMetrics&metricname=Load_SystemPodMetrics&RestartCount=RestartCount

[pacoxu]

/hold

so the log shows a OOM of the metric-server, we cannot know the restart reason of NPD here.