feature: add inflight checks to detect some configuration issues

[tzneal]

Report as events and logs the reasons why a node is stuck terminating or failing to initialize.

Fixes aws/karpenter-provider-aws#2829

Description

How was this change tested?
Unit testing & deployed.

Sample log:

karpenter-74bdb86d9c-kc77c controller 2022-11-10T15:56:42.861Z	INFO	controller.inflightchecks	Inflight check failed for node ip-192-168-85-164.us-west-2.compute.internal, Can't drain node, pod default/my-shell is not owned	{"commit": "f691533-dirty", "node": "ip-192-168-85-164.us-west-2.compute.internal"}

Sample event:

51s         Warning   FailedInflightCheck             node/ip-192-168-85-164.us-west-2.compute.internal    Can't drain node, pod default/my-shell is not owned

There is some duplication here as we also report failures to evict. I haven't come to an opinion on if that's ok. I'm leaning towards it's better to have the duplication as this is rate limited to once per 10 minutes per node and also logs at the info level in addition to creating the event.

  Warning  FailedInflightCheck  2m34s (x2 over 12m)  karpenter  Can't drain node, pod default/my-shell is not owned
  Warning  FailedDraining       32s (x7 over 12m)    karpenter  Failed to drain node, pod default/my-shell does not have any owner references

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[coveralls]

Pull Request Test Coverage Report for Build 3481295853

• 174 of 230 (75.65%) changed or added relevant lines in 10 files are covered.
• 23 unchanged lines in 3 files lost coverage.
• Overall coverage increased (+0.2%) to 74.647%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/controllers.go	0	1	0.0%
pkg/controllers/deprovisioning/consolidation.go	12	14	85.71%
pkg/controllers/inflightchecks/termination.go	21	28	75.0%
pkg/controllers/inflightchecks/failedinit.go	36	46	78.26%
pkg/controllers/inflightchecks/nodeshape.go	30	42	71.43%
pkg/controllers/inflightchecks/controller.go	39	63	61.9%

Files with Coverage Reduction	New Missed Lines	%
pkg/test/environment.go	6	73.91%
pkg/controllers/provisioning/scheduling/preferences.go	7	86.81%
pkg/controllers/provisioning/scheduling/topology.go	10	84.71%

Totals
Change from base Build 3480855207:	0.2%
Covered Lines:	5073
Relevant Lines:	6796

---

💛 - Coveralls

[jonathan-innis]

Overall, love this feature for observability! Really cool work! 🎉

[ellistarn]

karpenter-74bdb86d9c-kc77c controller 2022-11-10T15:56:42.861Z INFO controller.inflightchecks Inflight check failed for node ip-192-168-85-164.us-west-2.compute.internal, Can't drain node, pod default/my-shell is not owned {"commit": "f691533-dirty", "node": "ip-192-168-85-164.us-west-2.compute.internal"}

The latest from @dewjam is that we shouldn't block on unowned pods. Worth sequencing these PRs? I'm happy either way.

[tzneal]

karpenter-74bdb86d9c-kc77c controller 2022-11-10T15:56:42.861Z INFO controller.inflightchecks Inflight check failed for node ip-192-168-85-164.us-west-2.compute.internal, Can't drain node, pod default/my-shell is not owned {"commit": "f691533-dirty", "node": "ip-192-168-85-164.us-west-2.compute.internal"}

The latest from @dewjam is that we shouldn't block on unowned pods. Worth sequencing these PRs? I'm happy either way.

There's no PR for that is there? No issues removing the controllerless checks if that code goes in, but I think this check is worthwhile till it does.

[tzneal]

karpenter-74bdb86d9c-kc77c controller 2022-11-10T15:56:42.861Z INFO controller.inflightchecks Inflight check failed for node ip-192-168-85-164.us-west-2.compute.internal, Can't drain node, pod default/my-shell is not owned {"commit": "f691533-dirty", "node": "ip-192-168-85-164.us-west-2.compute.internal"}

The latest from @dewjam is that we shouldn't block on unowned pods. Worth sequencing these PRs? I'm happy either way.

There's no PR for that is there? No issues removing the controllerless checks if that code goes in, but I think this check is worthwhile till it does.

Removed the controller-less pod check since @dewjam 's change went in.

[tzneal]

Merge conflict, rebased.