Add informers, listers, full ScyllaCluster reconciliation, typed clients and much more

[tnozicka]

Description of your changes:

• Full ScyllaCluster CR reconciliation
• Pruning old resources (like services on scale down)
• Abides The Three Laws of Controllers
• Using Informers and caches as much as possible
• Using generated typed client for ScyllaCluster
• Independent, cache based probes
• Isolates flags from internal abstractions
• All controllers can be run from outside of the cluster (for dev)
• Includes partial seedless node support to avoid an old race (tests and restriction lift will go in Implement seedless mode #597)
• Fixes scylla graceful termination
• Splits Oprhaned PV controller and wires handlers for node deletion
• Separates helper functions and validation out of the API package
• Dedicated webhook server for separation of concerns and lifetime (better availability)
• Graceful termination for webhook server, probes and operators
• Fixes upgrade handlers that could be unintentionally skipped if a pod was evicted during the rolling upgrade.

Which issue is resolved by this Pull Request:
Fixes #469
Fixes #470
Fixes #429
Fixes #475
Fixes #469
Fixes #513
Fixes #514
Fixes #459
Fixes #607
Fixes #558
Fixes #526
Fixes #520

Followups

• Add unit test to cover running hooks on upgrades - alternatively migrate hooks to jobs (should be done anyways at some point Run upgrade hooks as Jobs instead inside the operator #587) which could be covered in e2e test

Audit results

• Before https://github.com/scylladb/scylla-operator/suites/3100807483/artifacts/70904312
  Over the run of an e2e suite (~48m)

  {
    "user": "system:serviceaccount:scylla-manager:scylla-manager-controller",
    "total": 17230,
    "verbs": {
      "create": 8,
      "get": 12174,
      "list": 6,
      "update": 5026,
      "watch": 16
    }
  },
  {
    "user": "system:serviceaccount:scylla-operator:scylla-operator",
    "total": 19855,
    "verbs": {
      "create": 578,
      "delete": 24,
      "get": 13100,
      "list": 18,
      "patch": 208,
      "update": 5798,
      "watch": 129
    }
  },

37085/48 = ~772 API calls per minute (doesn't include the side car and health checks)

  {
    "user": "system:serviceaccount:scylla-manager:scylla-manager-cluster-member",
    "total": 2466,
    "verbs": {
      "get": 2446,
      "list": 4,
      "watch": 16
    }
  },

the longest lived scylla cluster (health checks + side car) made 2466 API calls

• With this PR https://github.com/scylladb/scylla-operator/suites/3102089817/artifacts/70933413
  e2e run took about ~46m

  {
    "user": "system:serviceaccount:scylla-manager:scylla-manager-controller",
    "total": 1640,
    "verbs": {
      "create": 2,
      "get": 964,
      "list": 4,
      "update": 632,
      "watch": 38
    }
  },
  {
    "user": "system:serviceaccount:scylla-operator:scylla-operator",
    "total": 4929,
    "verbs": {
      "create": 1204,
      "delete": 530,
      "get": 1038,
      "list": 286,
      "patch": 440,
      "update": 1260,
      "watch": 171
    }
  },

6569/46= ~143 API calls per minute (doesn't include the side car and health checks)

  {
    "user": "system:serviceaccount:e2e-test-scyllacluster-92jtt-q6brr:e2e-user",
    "total": 155,
    "verbs": {
      "create": 6,
      "get": 124,
      "list": 14,
      "patch": 2,
      "watch": 9
    }
  },

the longest lived scylla cluster (health checks + side car) made 155 API calls

Final results

6569/37085=~18% of API calls per controllers
155/2466=~6% of API calls per scyllacluster

[tnozicka]

@zimnx thanks for the first round, comments addressed

[tnozicka]

fixed the new scale down test

[zimnx]

2nd batch

[zimnx]

last batch for this iteration. I like changes very much, code looks very mature. Awesome work!

[tnozicka]

Thanks for the review. I'v addressed most of it but there are still some comments I need to address tomorrow.

[zimnx]

LGTM, good job!