Support syncing and updating more than one git repository

[squaremo]

From time to time people say that it would be handy if flux supported syncing with more than one repo.

These are use cases:

• We keep our configs alongside our app code, and that's in one repo per app
• I have a infra layer of config that I only want to sync, and then other app config that I want to sync and update

These are not use cases:

• We have different teams, and they each have a repo with config (use multiple fluxd daemons, so you can narrow the RBAC permissions for each)

Also see #1139.

[rade]

Would git subtrees be of help here? After all, they do allow you to in effect aggregate multiple repos into one.

[squaremo]

Would git subtrees be of help here? After all, they do allow you to in effect aggregate multiple repos into one.

Maybe, but I'm not sure it would be the simplest way to accomplish this. (Genuinely not sure, rather than British not sure)

[Smana]

We're exactly in the first use case and we have multiple repositories.

We keep our configs alongside our app code, and that's in one repo per app

Keeping the "config" with the code allows the devs to work in a single place on the project and on how it is deployed.

Let's suppose you deploy 50 projects in a given namespace and you want to keep the helm charts (that's what we're using) along with the code. Does it mean you'll have 50 operators in the same namespace ? and how about the helm-operator ? I guess there's some work to do if you want to keep a single tiller.

I'm looking for advices on the best setup in my case. I'm going to explore 2 options:

• The first one as explained above, one flux-operator per git repository
• A new repository with a directory tree per namespace and a flux operator per namespace

Help is welcome :)

[Smana]

I managed to have something that fits my needs.

I've create a very simple operator that deploy fluxes :)
In a namespace flux I have:

• 1 tiller
• a flux per github repository
• a helm operator per repository

Todo:

• use a single memcache instance for all fluxes if possible
• limit the scope of the fluxes (RBAC)

[mazzy89]

@Smana what it scares me with that approach is that you need to have a flux for each service that you want to deploy. Basically, you need to double the resources in your cluster just to accommodate flux 😱. If you have 50 microservices, you need to have 50 different flux daemons running. So you will eat your cluster resources faster.

I think at the moment the only feasible approach is to have mono repos.

[Smana]

@mazzy89 I understand your concerns, but I can't change what's has been build for years and the current status of the github repositories in my current.
That said i'm pretty happy with my setup.
Here's an updated version:

• 1 flux operator in charge of creating fluxes
• 1 memcached cluster for all the instances
• 1 sealed-secrets operator

Todo:

• The sealed secrets option is temporary, we're thinking about dynamic secrets injection at pod startup with vault or even security vendor solutions (POCs starting soon)
• Limit the fluxes instances to a single namespace
• We want to keep the decision of changing the image tag in our CI workflow. I'm going to do some tests.

But yes, I agree, we'll have number of couples flux/helm-operator :

flux-operator-5d957f64f7-flsfj                                    1/1     Running   0          2d20h
project1-namespace1-4wpi7jr0p61vonmvidlatf8k-flux-7f444bd5c9-859w9       1/1     Running   0          2d19h
project1-namespace1-4wpi7jr0p61vonmvidlatf8k-flux-helm-operator-65nnh5   1/1     Running   0          2d19h
project2-namespace2-qtldck8q7sj0kgdyo8ixjyw4-flux-545d8c6d48-g627s     1/1     Running   0          2d
project2-namespace2-qtldck8q7sj0kgdyo8ixjyw4-flux-helm-operatorcgspj   1/1     Running   0          2d
project3-namespace3-a59qvo33udy3gu7l5klwjpmck-flux-748657bcf7gwl97   1/1     Running   0          47h
project3-namespace3-a59qvo33udy3gu7l5klwjpmck-flux-helm-operar595g   1/1     Running   0          47h
project4-namespace4-9z7xaxmhodgomfqoxnls85kro-flux-96697b455-hsb5x   1/1     Running   0          41h
project4-namespace4-9z7xaxmhodgomfqoxnls85kro-flux-helm-operag4xpg   1/1     Running   0          41h
project5-namespace5-7k9qgulhls6zfmsevylvex1lw-flux-6d9f76f7lqkkz   1/1     Running   0          43h
project5-namespace5-7k9qgulhls6zfmsevylvex1lw-flux-helm-opehfvm6   1/1     Running   0          43h
memcached-0                                                       1/1     Running   0          3d13h
memcached-1                                                       1/1     Running   0          3d13h
memcached-2                                                       1/1     Running   0          3d13h
tiller-deploy-67b4548b8c-b579k                                    1/1     Running   0          4d12h

[stillinbeta]

My use case is relatively simple: I'd like to have most of my configuration live in an open source repository, and then just have secrets live in a private repository. Multiple repositories seems like the easiest way to accomplish this, but I may be missing something obvious

[cruisehall]

Has anyone seen a pattern for compiling k8s manifests from multiple repositories using something like jsonnet (or the late ksonnet)? https://jsonnet.org/

With features like “flux release”, it certainly seems like flux intends to treat application source code as a first class citizen, but by doing so flux is biting off more than it can (or should?) chew.

If flux is just a tool for managing cluster state, it’s fair to say that flux only needs to be concerned with one repo - the “cluster repo”. Leave it to users to figure out how to populate that repo with their “fully compiled” k8s manifests.

But if flux is a tool for managing cluster state AND releasing applications (among other things),, then it must be able to handle resolving resources from multiple repositories.

[toran414]

We keep our configs side by side the code, each workload in its own repo. We also would like to use GitFlow with different branches in each of those repos applying to different namespaces in our cluster.

[tschonnie]

I am confused. Does https://github.com/fluxcd/multi-tenancy solve this issue or not?

[hiddeco]

I am confused. Does fluxcd/multi-tenancy solve this issue or not?

It does kind of solve this by spawning multiple Flux daemons, this issue is however about supporting multiple git repositories with one Flux daemon, which the mentioned repository does not solve.

[ostigley]

Would git subtrees be of help here? After all, they do allow you to in effect aggregate multiple repos into one.

We would really like to see this. This would allow our users to be restricted to making changes to only the submodule code, not the entire cluster.

[mheck136]

Would it be possible to use a CRD to define flux-requirements per namespace? A team which is restricted to only change things in a certain namespace could add a CRD (e.g. "flux") to that namespace, containing information about the git repository used as source for kubernetes resources in that namespace and a service account with the privileges to execute deployments.
The flux operator, running in a dedicated namespace and possibly managed by an operations team, would watch those CRDs and use them to determine the Git repositories to watch. If a change in a repo is detected, it does not role out the changes by itself but rather starts a POD in the namespace, in which the changes have to be executed, using the service account configured in the CRD. That way, the role-out is executed using exactly the roles which were intended by the team.

This approach has multiple advantages. It would allow to use multiple repositories with only one flux operator running and watching repositories. (Less resource usage and less maintenance)
Teams with access restricted to a certain namespace would be able to configure flux within that namespace without risks for other namespaces.

[ryedin]

@mheck136 that design seems like the optimal architecture to me. 1 operator -> 1-many CRs -> ephemeral pods (via Jobs most likely) to execute the actual work when needed -> when sync complete, pod goes away, job deleted

EDIT: Another benefit is that the Flux CR files can then be managed as code/Helm/etc.

[smveloso]

I've been trying to adopt flux in my organization (one kubernetes cluster hosting several apps by multiple teams) and the only thing that's preventing me is that I always find myself in need of multiple flux instances, which seems very inneficient in terms of resource usage.

Since all my deployments today use kustomize it took me a while to think about taking a deeper look at the helm operator. I did it now and it seems to me that could be a way forward. Could the same approach be used for kustomize-based deployments ? Something like a "generalized" helm operator ?

Not sure whether @mheck136 was thinking the same thing, i.e., having flux (daemon) and a flux (operator).

To be precise, the helm operator depends on flux to keep the CRDs in sync so it is still limited to a single git repo for that. The CRDs, however, can refer to other repos.

[alllexandrz]

Hello Team. Do you have a plan to implement this?

[stefanprodan]

Implemented in Flux v2 https://github.com/fluxcd/flux2

Docs here https://toolkit.fluxcd.io/components/source/gitrepositories/#spec-examples