Tainted nodes are bidding

[88plug]

Describe the bug
Some providers have nodes that are tainted in their cluster and the Akash provider is still using them to bid when a cluster is at capacity.

[Warning] [FailedScheduling] [Pod] 0/6 nodes are available: 1 Insufficient memory, 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 4 Insufficient cpu. preemption: 0/6 nodes are available: 1 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod..

[FailedScheduling] [Pod] 0/4 nodes are available: 1 node(s) had untolerated taint {CriticalAddonsOnly: true}, 3 Insufficient cpu. preemption: 0/4 nodes are available: 1 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.

I have found at least two styles of taint that are not being respected :

1. {node-role.kubernetes.io/control-plane: },
2. {CriticalAddonsOnly: true}

This bug causes causes the deployment never to deploy - and within 5 minute it's closed automatically by the provider.

To Reproduce
Deploy to fill each node on a provider and you will get a bid from the tainted node.

Observered in the wild on validatornode.com and various other providers.

Expected behavior
Providers who have nodes that are tainted for no-deploy/noscheduling should not bid on workloads.

Additional context
Discussed on Sep 18th support call for more detail.

[88plug]

Users are reporting this in Discord as well

[neyy91]

Traced this: node discovery (operator/inventory/node-discovery.go) excludes nodes that are not-ready, cordoned, or in the exclude list, but never checks Spec.Taints - so a Ready node with a NoSchedule/NoExecute taint (e.g. node-role.kubernetes.io/control-plane, CriticalAddonsOnly) is still counted as available capacity and bid on. Since Akash pods carry no custom tolerations, the pod can never schedule there and the deployment is closed minutes later.

Opened a fix that keeps NoSchedule/NoExecute-tainted nodes out of the inventory (PreferNoSchedule left alone), with a unit test: akash-network/provider#406