Administrate reconciler remotely via CLI using REST API

[tobiscr]

Description

The reconciler has to support remote administration via the CLI. The communication between the CLI and the mothership-reconciler has to be handled via an REST API.

The REST API has to be specified using OpenAPI specification (e.g. Swagger, see #116), support a secure and trusted communication (HTTPS) and be integrated into the SAP SSO solution (ORY?). Any user-action triggered by a client has to be recorded in an audit log.

AC:

• CLI communicates remotely via REST API with the mothership reconciler
• API is described in a OpenAPI specification (Swagger spec) - see Create OpenAPI specification (Swagger) for REST API of mothership reconciler #116
• SSO integration is available to allow users to login using their SAP account
• Any action triggered via the REST API is recoreded in an audit log
• Following features have to be supported by the REST API and can be used via the CLI (acting as client of the REST API):

1. Show reconciliation runs of a cluster
2. Show details of a reconciliation run (start time, end-time, reported progress of the component-reconcilers)
3. Show details of reconcilation output created for a particular component (requires integration with logging-system from SRE?)
4. Disable the reconciliation of a cluster (either for a particular time range or endless) / Enable the reconciliation of a cluster if ti was disabled - Allow SRE to administer cluster reconciliation status via CLI #188

Reasons

Establish a standardised tooling to control and administrate the mothership reconciler which fulfils security requirements and is integrated with the SAP SSO system.

Attachments

[kwiatekus]

After an interview with SRE representatives we see that the following should be part of the MVP:

• List of runtimes incl. reconciliation status with filtering by:
  • reconciliation status
  • shoot name
  • service instance ID
• force start ( enable --force ) of reconciliation ( MVP - whole runtime, later per component )
• stop / pause (disable --time) reconciliation for a given runtime for a given time

What we also realised is that SRE engineers prefer to get the full picture from monitoring and logging tools that are already part of control plane. They would like to build their own dashboards and configure alerting rules on their own provided that metrics are exported ( incl reconciliation status per runtime per comonent ) and logs are exportable to the logging provider via their fluentbit.

CLI tooling should only provide a subset of the full capabilities of the observability tooling ( i.e general overview of reconciliation status. Reconciliation logs do not need to be accessible via CLI ) + CLI should provide commands that are useful in daily ops ( pause, force start of reconciliation )

[clebs]

HI team,
the delete feature (#200) added the following states to the external API for the reconciler:

reconciler/openapi/external_api.yaml

Lines 412 to 415 in cd630f8

	- delete_pending
	- delete_error
	- deleting
	- deleted