source-controller: define source dependencies

[seaneagan]

Sometimes kustomizations need to combine manifests from multiple git repositories. For example one may keep most of their manifests for a given cluster type in one repo, while each cluster instance of this type contains parameters to patch into the cluster type manifests in its own repo. When using kustomize as a CLI this can be done by cloning the dependency repositories to predictable paths within the dependent repository checkout.

Note: while kustomizations can reference remote git refs in resources, this is highly repetitive and last time i checked i don't think caching was fully implemented. It also doesn't allow for automation of updates when the dependencies change.

This could be supported by allowing sources to specify dependencies including where to store them within the dependent source artifact. For example:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: type
# ...
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: instance
spec:
  dependsOn:
    - name: type
      targetPath: dependencies/type
  # ...
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: deps
spec:
  sourceRef:
    kind: GitRepository
    name: instance
  path: . # < kustomization references `dependencies/type`
  # ...

This could also be used to potentially solve the following limitation (mentioned here with sourcing Helm charts from Git:

Chart dependencies must be committed to Git, as the source-controller does not attempt to download them. This limitation may be removed in a future release.

Alternatives

• Use KPT subpackages. KPT in general requires one to commit changes (such as dependency updates) back to git in order to consume them from something like the GitOps Toolkit, unless the GitOps Toolkit added a kpt-controller which ran KPT commands on a source Kpt package and applied them. Automation of dependency updates could perhaps be implemented through something similar to the image update automation except targeted at updating Kptfile dependencies.
• Use git subtree (see above KPT link for limitations) or git submodules.

[stefanprodan]

When using remote bases, kustomize-controller does not support private repos, your proposal would solve this along with the caching problem.

I don't think dependsOn is the right term to describe this features, my proposal is:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: app
# ...
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: infra
spec:
  include:
    - name: app
      namespace: gotk-system
      from: "./deploy"
      to: "./bases/apps/app"
# ...
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: infra
spec:
  sourceRef:
    kind: GitRepository
    name: infra
  path: ./clusters/prod # < kustomization.yaml references `../bases/apps/app`
  # ...

[raffis]

@stefanprodan Will this proposal by any chance be implemented? Or something which would be accepted as contribution?

[phillebaba]

We had a discussion about this at the community meeting two weeks ago and there seemed to be some differing opinions on this matter. @squaremo thought that this may be a complex solution for the problem it solved. I am a bit on the fence regarding this question. On the one hand it introduces a flux custom solution which will not work locally out of the box, on the other hand I currently do not have a better solution.

[stefanprodan]

On the one hand it introduces a flux custom solution which will not work locally out of the box, on the other hand I currently do not have a better solution.

This solution works locally by using git submodules and in-cluster by replacing the submodules with include. I see the include feature as a drop-in replacement for submodules. You would still use submodules to test your kustomization locally and in CI, but in your production cluster you would use Flux's include to pull the manifests from different repos and avoid the Git SaaS limitation where a deploy key can't be used for more than one repo.

[nlamot]

Hi @stefanprodan , is there an update on this? This is a very important for our adoption of flux v2 and I would like to work on this feature if it would be (if implemented correctly of course) accepted as a contribution.

We currently use flux v1 and do use kustomize references to external private repositories that are very difficult to remove.

[stefanprodan]

I think @phillebaba is working on this? In the meantime you can use git submodules, see https://toolkit.fluxcd.io/components/source/gitrepositories/#git-submodules

[vladimirfx]

I think 2 options are needed:

1. Git submodules (Support Git submodules source-controller#169)
2. Authentication support for remote resources in kustomization.

We use both but submodules more often.

In our practice we compose fleet repository from cluster wide manifests and application/team manifests tracked in external repositories. In such scenario fleet repository contains main application kustomization in which external manifests is used as base or referenced somehow else.
As so we need flat file structure of fleet repository with all submodules inited and updated.
Submodules makes ops team work easier and more transparent because what is applied is visible and controlled in flat file structure.
Remote kustomize resources is useful for rarely changed external manifests (such case is often covered by Helm charts).

[stefanprodan]

How would you set up authentication? GitHub doesn't allow the same deploy key to be reused, source-controller will use the main repo credentials to pull the submodules and those will fail.

[vladimirfx]

How would you set up authentication? GitHub doesn't allow the same deploy key to be reused, source-controller will use the main repo credentials to pull the submodules and those will fail.

We use HTTPS/Basic - so simplest implementation can reuse credentials from root repo.
For deployment keys lookup-table-like secret can be used (repo-to-key) which can be conveniently resumed in many source controllers (something similar to SOPS secrets).

[vladimirfx]

Additionally we implement another workaround - attach SSH keys to kustomize-controller that manage distinct namespace. So that controller can use ssh:git@somhost resources with SSH-based authorization.
But it is workaround... with all its drawbacks... So very waiting for solution.

[raffis]

How would you set up authentication? GitHub doesn't allow the same deploy key to be reused, source-controller will use the main repo credentials to pull the submodules and those will fail.

If the credentials are from a user (for example dedicated gitops github user), not deploy keys it should work.

[bob-rohan]

we have no plans on adding Git auth to kustomize-controller as the reconciler shouldn't deal with Git or any other type of source, that's source-controller responsibility.

Kustomize-controller does currently deal with Git for kustomize remote bases on public repos.

@stefanprodan Would it be acceptable to supply a credentials as a secret ref to the Kustomization CRD, which would be propagated to the file system pre buildKustomization along with supporting git/SSH config??

[vladimirfx]

We build workaround that uses kustomize (go-get) ability to fetch external resources. Of course it has drawbacks:

• auth token to access external resources is stored in plain text
• content of customized resources not transparent when working with fleet repo

[freefood89]

Hello-

I was wondering what the status of this discussion is.

I'm really looking forward to adopt flux v2 for multitenancy, but I depend on git submodules to manage shared kustomize bases for various teams. So far with flux v1 I've been using flux.yaml file to pull those submodules using ssh keys that are registered with my ci bot user (instead of repo deploy key).

Is there any way I can help?

[stefanprodan]

Git submodules

Starting with flux2 v0.12 you can bootstrap with --recurse-submodules and create GitRepositories by enabling Git submodules:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: repo-with-submodules
  namespace: default
spec:
  interval: 1m
  url: https://github.com/<organization>/<repository>
  secretRef:
    name: https-credentials
  ref:
    branch: main
  recurseSubmodules: true
  gitImplementation: go-git
---
apiVersion: v1
kind: Secret
metadata:
  name: https-credentials
  namespace: default
type: Opaque
data:
  username: <Username>
  password: <Token or Password>

Note that deploy keys can't be used to pull submodules from private repositories as GitHub and GitLab doesn't allow a deploy key to be reused across repositories. You have to use either HTTPS token-based authentication, or an SSH key belonging to a user that has access to the main repository and all its submodules.

More info here.

[siegenthalerroger]

Sorry to resurrect this from the past, but is there any intention of supporting a similar flow for OCI sources? Kustomize's support for remote OCI bases is stalled but the same security and performance concerns as for git remote bases exist, so it'd be preferable if there was some way to do this with Flux.