Cannot combine Git files and Git directories generators with matrix

[romuduck]

Hello,

Current behavior

When using matrix generator, we cannot combine:

• a Git directory generator which would browse through services for example
• a Git file generator which would look for some cross-service configuration files

Probably obvious, but as both generators output a path key, but are browsing different sources, it results in following error:

msg="error generating params" error="found duplicate key path.basename with different value, a: xxx ,b: yyy" ...

Expectation

I guess we should either update the documentation and/or provide a way to give precedence.

Thanks all

[crenshaw-dev]

@romuduck wouldn't a file generator with something like ../**/config.yaml be the same thing as a file x directory matrix generator?

[w4rgrum]

Allowing to prefix the variables in output of the generators would be nice and help using twice the same generator in a matrix generator. This could be done by adding an optional "name" field to generator for example, if present all outputs would be prefixed by <name>.

[crenshaw-dev]

@w4rgrum true, that would work. Do you have a use case that requires a git file * git directory matrix generator?

[dh-apporto]

@crenshaw-dev - I was actually wishing I could do this. My use-case is we have an appstore with multiple apps that will be deployed to every tenant. I've got my tenants defined as git files (provides basic parameters), and I would like to use git dirs generator to generate the list of helm templates to install. Right now, I'm having to use git files for tenant definition and a list generator to specify the applications to be deployed. It would be nice to have it automatically read the git directory structure to generate the list of apps to deploy.

When I did try the git files * git paths - it seemed like it would work except that the variable names were not allowed to collide (eg. path.basename). Maybe an easy way to let this type of configuration happen is allow an override of the variables names or add a prefix to the variables that are created by the git generator. Then people could use them as needed as long as they picked non-conflicting prefixes/names.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: appstore-tenants-appset
spec:
  generators:
  # Matrix multiple all customers by list of apps
  - matrix:
      generators:
      # Find all Customers
      - git:
          repoURL: git@github.com:somecorp/k8s-gitops.git
          revision: HEAD
          files:
          - path: appset-configs/appstore/tenants/REGION_NAME/*.yaml
      # Matrix multiple list of repo parameters with list of apps to deploy
      - matrix:
          generators:
          # List All Apps to be deployed for tenant
          - list:
              elements:
              - appName: tenant-app
                groupName: appstore
              - appName: tenant-api
                groupName: appstore
              - appName: tenant-redis
                groupName: appstore
              - appName: secrets
                groupName: common
          # List all gitops repo level parameters.
          # (should be a single element to work as platform parameters)
          - list:
              elements:
              - helmBase: helm
                helmValuesPath: ../../../../helm_values
  template:
    metadata:
      name: '{{tenant.name}}-{{appName}}'
    spec:
      project: tenant-appstore
      source:
        repoURL: git@github.com:somecorp/k8s-gitops.git
        targetRevision: '{{app.release_version}}'
        path: '{{helmBase}}/{{app.release_tree}}/{{groupName}}/{{appName}}'
        helm:
          parameters:
          - name: tenant.name
            value: '{{tenant.name}}'
          - name: tenant.id
            value: '{{tenant.id}}'
          - name: tenant.region
            value: '{{path.basename}}'
          - name: app.release_tree
            value: '{{app.release_tree}}'
          - name: app.release_version
            value: '{{app.release_version}}'
          # NOTE: ignoreMissingValueFiles coming in v2.3 of argocd
          #ignoreMissingValueFiles: true
          valueFiles:
          - 'values.yaml'
          - 'current_release_values.yaml'
          - '{{helmValuesPath}}/values.yaml'
          - '{{helmValuesPath}}/base/{{path.basename}}/values.yaml'
          - '{{helmValuesPath}}/base/{{path.basename}}/{{groupName}}/values.yaml'
          - '{{helmValuesPath}}/base/{{path.basename}}/{{groupName}}/{{appName}}/values.yaml'
          - '{{helmValuesPath}}/tenants/values.yaml'
          - '{{helmValuesPath}}/tenants/{{path.basename}}/{{tenant.name}}/values.yaml'
          - '{{helmValuesPath}}/tenants/{{path.basename}}/{{tenant.name}}/{{groupName}}/values.yaml'
          - '{{helmValuesPath}}/tenants/{{path.basename}}/{{tenant.name}}/{{groupName}}/{{appName}}/values.yaml'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'tenant-{{tenant.name}}'

[crenshaw-dev]

@dh-apporto thanks for the example! That makes a lot of sense. Makes me think that something like a ConfigMap generator might make sense at some point, for folks who prefer one big JSON structure over directories and files.

I think adding a prefix option is the simplest way to tackle this. If anyone's interesting in contributing that, it would be appreciated. If not, I'll see if I can carve out some time.

[adamjohnson01]

@crenshaw-dev, I am happy to have a go at implementing this.

[elebioda]

I have a use case where our git file generator will grab all our cluster configs that contain certain information: gcpProject, gcpRegion and then use a directory generator to bring up our apps across all clusters.

[yoav-klein]

+1, I could use this feature also, any new about that?

[gaeljw]

I have a use case where our git file generator will grab all our cluster configs that contain certain information: gcpProject, gcpRegion and then use a directory generator to bring up our apps across all clusters.

I have a very similar use case. Being able to name/prefix generators would be helpful for this.

[crenshaw-dev]

Being implemented here: argoproj/argo-cd#10523