consider exposing CAPI resources to end users

[bcle]

From Chris Jones: (slack post: https://platform9.slack.com/archives/C02SEAQ0NB1/p1643832931650859)

The idea in my mind was to leverage Argo to manage CAPI objects as Argo Applications. This is essentially the demo that Spectro Cloud did during ArgoCon.
When i first thought about the end to end profile flow it was something like
Create CAPI YAML in Github -  Platform9 would enable this to be created automatically from an existing cluster.
Create Apps & K8s Resources in github - Platform9 would enable this to be created automatically from an existing cluster (we do this today for RBAC)
Create Profile from 1 &2
Arlon creates cluster from profile
Create Argo Apps from Profile (with any particular overrides, AWS region for example)
CAPI App for the cluster
Register the cluster into Argo (For the Apps and objects in the profile to deploy something needs to Automatically register the cluster into CAPI )
Once registered the Apps and K8s objects deploy,
App for all other objects
From your demo it would appear we have the following
Write cluster spec as plain text/YAML
Add cluster spec to Arlon so its stored as a CRD
Create Apps & K8s Resources in github OR local YAML/Helm files
Map Bundles to Apps and K8s Objects and store as CRD
Map Bundles to Profile as PRD
Create cluster by passing ClusterSpec and Profile
CAPI creates cluster
Registration controller registers the Cluster into Argo
Arlon then creates the argo Apps
You raised some valid points around Lifecycle that we need to think about.
One way around this is to mark the ClusterAPI Apps in Argo as Manual (disable sync) OR clone the YAML for each cluster in Github.
The issue with cloning is that there is no Change Once - Update Many Cluster ability.
Honestly we should look to support both methods, so that a user can choose,
"Deploy as baseline" where the CAPI YAML is cloned and sync is manual.
In this scenario we still need to enable a user to ask "which ClusterSpec and which version was cloned?" so that they can do a comparison between clusters
They should also be able to compare the clusters current state to that for which it is running, Assuming that they're using CAPI to augment the cluster.
"Deploy as cloned with sync" -  this could clone the YAML but keep sync on, such that changing the code in github would reflect a change in the Argo App which in turn is picked up by Cluster API and applied to the clusters
With this, items such as upgrades become a question, and i would need Anmol to help as i have not read up on how CAPI does upgrades.
"Deploy as Linked Cluster without Sync" - this would deploy directly from the CAPI Yaml in Github and work as a  "one-off" type cluster
"Deploy as Linked Cluster with Sync" - this would deploy directly from the CAPI Yaml in Github where any change to the ClusterSpec would reflect as a change in any cluster running off of the same ClusterSpec
The end goal is to have 100% of the cluster and apps defined in gitub,

Aha! Link: https://pf9.aha.io/features/ARLON-113

[bcle]

duplicate of all features tracking "next-gen" clusters provisioned from base clusters.