Convert to kubebuilder

[scothis]

The controller currently uses knative/pkg as a foundation. The Kubernetes ecosystem is consolidating around Kubebuilder as the model for reconcilers.

Kubebuilder has a few key advantages:

• maintains boiler plate (no copy pasta)
• broad support in the community
• runs with more modern k8s api versions

[scothis]

As is, Kubebuilder has lukewarm support for projects with multiple api groups. See https://book.kubebuilder.io/migration/multi-group.html

In light of our desire to also decompose the system into individually installable components (#59), we have a couple options:

1. keep everything in a single project, create individual install units
   • pro: a single unified codebase
   • con: may prove tricky as kubebuilder isn't setup for multiple api groups never mind decomposed installs
2. create a separate repo for each system component with a dedicated kubebuilder footprint
   • pro: more flexibility for dependencies and release lifecycles
   • con: the dependency tree all comes together in the CLI, and releases come together in the charts, so we're not gaining much at the cost of more moving pieces
3. "mono"-repo, with kubebuilder footprints in directories instead of repos
   • pro: a single codebase
   • con: we'd have a lot of duplication in a single repo, it may be possible to share some things, but that may also add complexity.

[sbawaska]

FWIW, I was playing around with the 3rd approach last week: #61

[scothis]

Done in #62 with follow on work in #67 #64 #65 #66