RFC: Enable gitops usage on management clusters #2

MarcelMue · 2021-02-10T12:30:03Z

Please discuss the comment publicly here. We want to find some alignment on our approach.

MarcelMue · 2021-02-10T12:33:00Z

Comment by @webwurst in the internal issue:

I like the idea and think it is also the right time to explore that area. People are becoming more aware of GitOps and eventually customers will ask for it. The development teams behind Flux and ArgoCD gathered to exchange experiences from production usage. Finally they decided to follow separated paths, but hopefully for good reasons. As the result there is Flux2 based on GitOps Toolkit which can also be used in your own golang tools or operators.

Gerald put out a first preview of a possible flux-app in our playground-catalog. Challenge at the moment is that it did not reach v1.0 yet and there are multiple releases per week to catch up with ;) Halo provides this "as is" for the current cycle since we are focusing on Loki. But I guess we are happy for PRs and discussions.

Trying out Flux1 earlier didn't make much sense since there were discussions about a re-write and breaking changes and finally the deprecation note last autumn.

ArgoCD and Fleet might be interesting for comparison.

cokiengchiara · 2021-02-10T12:52:26Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+We currently give customers access to the management clusters but do not support them in utilizing this access effectively in terms of git ops related management (e.g. for apps).
+
+Context:
+Some customers might be interested in utilizing mainstream git ops tooling in order to manage their apps across clusters and installations.


Main question for me:

What does it help our customers achieve? Is it, "Manage stacks of apps on fleets of clusters" and challenges related to that?

(If it is, then this is the thing Batman wants to help customers achieve. And I'd like to explore this opportunity to take this next step in that direction.)

I would say this is one approach which could achieve the goal of "Manage stacks of apps on fleets of clusters".

I would appreciate Batman input & opinions here :)

@cokiengchiara Yes this tooling would help customers to manage the app CRs and user values configmaps and secrets for their apps. With stacks of apps on fleets of clusters that becomes more complex.

There is a lot of content out there on the benefits the GitOps approach provides. I don't want to repeat that but maybe https://www.gitops.tech/ helps as an overview / starting point.

I agree with "Manage stacks (or fleets) of apps on fleets of clusters", and to be clear that means it is not just app CRs that get managed, but all kinds of YAML that a customer can and should create on the MCs. So I would generalize this a bit in two directions:

it's not about apps, it's about the Management API (MAPI)

it's not about gitops per se, it is about enabling tooling against the MAPI that needs "an agent" in the MC and cannot just externally work against the MAPI.

rossf7 · 2021-02-10T13:37:50Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+## Open Questions
+
+### 1. General direction
+1.1 We would like to allow our customers to choose their management tooling freely. Do we want to support some tools with e.g. a managed app first though?


We want to let customers choose their tooling freely but I think we need some validation so only a subset of apps can be installed in management clusters.

Starting with a managed app first makes sense to me. Especially if its a tool we choose to use ourselves.

I agree that managed apps makes sense. It's also a question of how much freedom we give customers / how much we allow them to "fuck up".

It could be that we limit this for the beginning to there's this selection of apps (at the get go 1) that you can install on your MC using our interface. very controlled.

Later, it could be extended to having nicely isolated namespaces in the MC where we can give a certain amount of more freedom, but even there because of the limits of isolation we need to be very careful and we might never get there completely.

because it's not only about "fuckign up" it might also be that there's cases of actual multi-tenancy within an MC, where there's an untrusted org

rossf7 · 2021-02-10T13:42:15Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+1.2 Do we own the gitops tooling or is it purely owned by the customer?
+
+### 2. Technical issues
+2.1 How does the setup for an in-cluster agent look like? Currently customers will struggle setting up an in-cluster agent with appropriate permissions.


In the app CR the customer just needs to set .spec.kubeConfig.inCluster to true.

We should also make our kubeconfigs work with GitOps tooling. They don't currently work with Flux because of how we name the secrets we generate in cluster-operator.

Sorry this was bad phrasing on my part:
This more refers to the initial agent set-up (at least as long as we don't have a managed app). We will currently struggle to not give the agent full permissions to all namespaces, right?

This more refers to the initial agent set-up (at least as long as we don't have a managed app). We will currently struggle to not give the agent full permissions to all namespaces, right?

Ah got it. Yes I think we'll need to restrict the permissions. Taking Flux as an example AIUI there are multi tenancy patterns with a system Flux for platform admins and then multiple Flux's for teams.

My proposal would be we follow that approach and offer a managed app with the locked down setup. This is another argument for offering managed apps for this IMO.

rossf7 · 2021-02-10T13:43:50Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+### 2. Technical issues
+2.1 How does the setup for an in-cluster agent look like? Currently customers will struggle setting up an in-cluster agent with appropriate permissions.
+
+2.2 How do we ensure security requirements when customer interaction increases?


There is a multi tenancy problem with chart CRs and their related configmaps and secrets.

These are currently stored in the giantswarm namespace. Before enabling customers to create apps in management clusters I think we should move this to their organization or cluster namespace.

rossf7 · 2021-02-10T13:58:25Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+We currently give customers access to the management clusters but do not support them in utilizing this access effectively in terms of git ops related management (e.g. for apps).
+
+Context:
+Some customers might be interested in utilizing mainstream git ops tooling in order to manage their apps across clusters and installations.


@cokiengchiara Yes this tooling would help customers to manage the app CRs and user values configmaps and secrets for their apps. With stacks of apps on fleets of clusters that becomes more complex.

There is a lot of content out there on the benefits the GitOps approach provides. I don't want to repeat that but maybe https://www.gitops.tech/ helps as an overview / starting point.

rossf7 · 2021-02-10T14:02:32Z

I think the primary interface should remain the Management API and App CRD to not force an option on customers. But I like the idea and I don't see this as blocking that.

Allowing customers to choose their own GitOps tooling could make using app platform easier and unblock more complex use cases.

I liked the demo Halo did with Flux. Especially using SOPS to encrypt the secrets which is a complex area. Argo could also be interesting especially its workflow engine. This could also enable customers to use these tools directly rather than App Platform if they work better for their use case.

piontec · 2021-02-12T16:39:29Z

There are also some notes (since quite some time) in our brain storm doc: https://docs.google.com/document/d/1-zgREXVmnu2Rd7QLnrKO15MJWSkWS9_yBWkzFZsSRNw/edit#heading=h.c80optp1pse7

piontec · 2021-02-12T16:45:29Z

Few more comments from me:

to get started, I think it totally makes sense to do managed Apps only
I'd like to keep (as Ross already wrote) the API intact: so we're looking for a gitops tool that manages/syncs App/CM/Secret (C)Rs
To get started, I would avoid allowing customers to choose and install stuff on MCs. This seems like a huge security and stability risk that will need a lot of effort to implement. I like it in general, just I don't think it's necessary to get customers started with gitops. By recommendation would be "opt-in request": if customer wants to use gitops tools, we deploy it for them to selected MCs. Customers then continue by providing configuration as necessary CRs. Limiting RBAC access to necessary namespaces and CRs on the MC should be much faster and easier than allowing them to install anything from a set of apps (this feature might come later)
TO get started, I would offer just a single gitops tool. @rossf7 and me already agreed to do a quick comparison project: we want to compare flux/argo/fleet/jenkinsx (OK, the last one is already out after what I learned today). Out of them , we pick one, we support this one and we offer it as on opt-in way for customers on MCs. We will evaluate what works best with our app management API, so they don't have to. And since we need to learn it, we can make some training for customers, so it's easier for them to get started.

All in all, as MVP, 1 opinionated tool, deployed on-demand by us and only configured by customers makes a lot of sense to me.

puja108 · 2021-02-15T11:44:46Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+### 1. General direction
+1.1 We would like to allow our customers to choose their management tooling freely. Do we want to support some tools with e.g. a managed app first though?
+
+1.2 Do we own the gitops tooling or is it purely owned by the customer?


I would say this could be similar to other managed apps, they might start in a low support level and move up if we feel confident, some might never be mananged

puja108 · 2021-02-15T11:46:19Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+
+2.2 How do we ensure security requirements when customer interaction increases?
+
+2.3 Do we foresee issues when introducing gitops tooling to already existing resources?


depends how invasive the tooling is, like does it delete and recreate resources or does it "adopt" and apply/update

Yes! This needs to be checked for each GitOps tool separately if resources changed by happa or gsctl will be changed back to the state the GitOps tool knows.

Random idea: At least flux applies labels to ressourced managed by it, so we could display (for example) clusters installed through the GitOps tool as read-only

puja108 · 2021-02-15T11:47:22Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+2.3 Do we foresee issues when introducing gitops tooling to already existing resources?
+
+### 3. Guidance
+3.1 How do we support customers in making sensible decision in terms of tooling choice with gitops?


@piontec was proposing a SPIKE within balo looking at the alternatives and forming a first opnion, as currently GS has no or only very low informed opinions it feels

puja108 · 2021-02-15T11:48:29Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+### 3. Guidance
+3.1 How do we support customers in making sensible decision in terms of tooling choice with gitops?
+
+3.2 Do we aid customers with repository structure for their gitops approach?


if this is sth we believe should be used, we could or rather should accompany the offering with docs as well as tutorials or workshops that educate the users. This is really nice in the context of moving Room 1 customer to Room 2 for example.

puja108 · 2021-02-15T11:50:46Z

0002-gitops-management-clusters/0002-gitops-management-clusters.md

+
+3.2 Do we aid customers with repository structure for their gitops approach?
+
+3.3 Do we offer customers to give us shared access to their repository for additional review?


Interesting Q, for me especially interesting if we could find a mutual interest there, as we already often have this between us and customers, where we also hate up waking up because of misconfiguration so we have an incentive to help out and avoid bad config. Also, interesting how that would play into our new Solutions Engineering approaches. cc @giantswarm/se

If we publish some guidelines (previous point) I do think it is fine checking current existing or future repo structure and whole design as part of SE consultancy work

puja108

I think this could be a really nice opportunity, especially thinking of moving Room 1 customers towards Room 2.

I could also see sense in making our approach staged from "very restrictive/controlled" MVP towards more "freedom", but I'd leave that open to future evaluation as there could be benefits to a more free approach that save initial efforts on our side and do not open up the pandoras box of security.

* main: remove duplicate rfc description file (#18) Use README.md in the folders (#14) Remove rfc numbers (#13)

…structure for RFCs.

teemow · 2021-04-22T10:24:32Z

Please make sure to pull the changes before making new modifications. I've renamed the folder and markdown as we agreed on a slightly different structure for RFCs.

JosephSalisbury · 2021-05-14T12:21:59Z

@snizhana-dynnyk do we have roadmap issues towards implementing the ideas here now?

MarcelMue · 2021-07-12T16:21:29Z

There is a roadmap issue now. Merging here.

RFC: Enable gitops usage on management clusters

c7ba85e

MarcelMue self-assigned this Feb 10, 2021

MarcelMue requested review from a team February 10, 2021 12:35

Fix markdown

bcefdc2

cokiengchiara reviewed Feb 10, 2021

View reviewed changes

rossf7 reviewed Feb 10, 2021

View reviewed changes

puja108 reviewed Feb 15, 2021

View reviewed changes

teemow added 2 commits April 22, 2021 12:23

Merge branch 'main' into gitops-management-clusters

5dc0c24

* main: remove duplicate rfc description file (#18) Use README.md in the folders (#14) Remove rfc numbers (#13)

Renamed the folder and markdown as we agreed on a slightly different …

370787d

…structure for RFCs.

cokiengchiara mentioned this pull request May 18, 2021

Enable customers to use gitops tooling for apps (with Flux CD) giantswarm/roadmap#279

Closed

MarcelMue merged commit 8216169 into main Jul 12, 2021

MarcelMue deleted the gitops-management-clusters branch July 12, 2021 16:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Enable gitops usage on management clusters #2

RFC: Enable gitops usage on management clusters #2

MarcelMue commented Feb 10, 2021

MarcelMue commented Feb 10, 2021

cokiengchiara Feb 10, 2021

MarcelMue Feb 10, 2021

rossf7 Feb 10, 2021

puja108 Feb 15, 2021

rossf7 Feb 10, 2021

MarcelMue Feb 11, 2021

puja108 Feb 15, 2021

puja108 Feb 15, 2021

rossf7 Feb 10, 2021

MarcelMue Feb 11, 2021

rossf7 Feb 11, 2021

rossf7 Feb 10, 2021

rossf7 Feb 10, 2021

rossf7 commented Feb 10, 2021

piontec commented Feb 12, 2021

piontec commented Feb 12, 2021

puja108 Feb 15, 2021

puja108 Feb 15, 2021

ubergesundheit Feb 15, 2021

puja108 Feb 15, 2021

puja108 Feb 15, 2021

puja108 Feb 15, 2021

pipo02mix Feb 17, 2021

puja108 left a comment

teemow commented Apr 22, 2021

JosephSalisbury commented May 14, 2021

MarcelMue commented Jul 12, 2021


		2.2 How do we ensure security requirements when customer interaction increases?

		2.3 Do we foresee issues when introducing gitops tooling to already existing resources?


		3.2 Do we aid customers with repository structure for their gitops approach?

		3.3 Do we offer customers to give us shared access to their repository for additional review?

RFC: Enable gitops usage on management clusters #2

RFC: Enable gitops usage on management clusters #2

Conversation

MarcelMue commented Feb 10, 2021

MarcelMue commented Feb 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossf7 commented Feb 10, 2021

piontec commented Feb 12, 2021

piontec commented Feb 12, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

puja108 left a comment

Choose a reason for hiding this comment

teemow commented Apr 22, 2021

JosephSalisbury commented May 14, 2021

MarcelMue commented Jul 12, 2021