Setup operational concept with SRE

[tobiscr]

Description

The reconcile requires an operational concept which is aligned with SREs. The concept has to cover:

• How the mothership- and component-reconciler(s) will be integrated into existing deployment pipelines (e.g. adding it pipelines which are also used to deploy KEB, provisioner etc.)
• What the operational requirements for mothership- and component-reconcilers are (expected deliverables like trouble-shooting guide, further documentation, integration into SRE's logging-/monitoring-system)
• What are the requirements / expectations from SRE side to the reconciler to analyse incidents (e.g. access to reconciler-logs, mandatory CLI features etc.): see Administrate reconciler remotely via CLI using REST API #115

AC:

• Written concept (could also be a checklist or similar stored inside of this Github issue) which covers the agreed action items / decision and was reviewed by SREs
• If tickets are created for some of these items, the tickets have to be referenced in the concept

Reasons

Ensure the reconciler is addressing operational constraints from SREs properly.

Attachments

[varbanv]

After having a discussion with SREs on 02.09.2021 we came up with the following requirements:

• Reconciler should be deployed using the same approach and as part of the KCP.
• Reconciler database will be deployed and configured by SRE.
• Reconciler will init and upgrade database using admin DB creadentials provided by SRE.
• We will provide high level design documentation.
• Metrics Exported will be used by SRE for monitoring.
• Mothership and component reconciler logs will be scraped using Fluent Bit so CLI will not provide them.
• KCP CLI will be augmented to include reconciler management functions: Administrate reconciler remotely via CLI using REST API #115
• Reconciler will sync state with KEB to avoid data loss during down times: Reconciler has to synchronise cluster inventory with KEB #127

[tobiscr]

Agreed with @ebensom that all parts from reconciler side are considered