feat: Multiple sources for applications

[ishitasequeira]

Note on DCO:

If the DCO action in the integration test fails, one or more of your commits are not signed off. Please click on the Details link next to the DCO action for instructions on how to resolve this.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).

Closes #677
Closes #2789

Description

This change enables users to provide multiple resources for the Application. The change aims to be fully backwards compatible.

This PR implements proposal #8322.

Change details

Adding Multiple Sources

The new field sources would allow users to input list of ApplicationSources. We would ignore the source field we can find even a single entry under sources field.

Below example shows how the yaml would look like for source and sources field. We would ignore the source field and apply the resources mentioned under sources field.




spec:
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    path: guestbook
    targetRevision: HEAD
  sources:
    - chart: elasticsearch
      helm:
        valueFiles:
          - values.yaml
      repoURL: https://helm.elastic.co
      targetRevision: 7.6.0
    - repoURL: https://github.com/argoproj/argocd-example-apps.git
      path: guestbook
      targetRevision: HEAD

Multiple Value files for Helm Source

For making files accessible to other sources, add a new ref field to the source. For example, in below ApplicationSpec, we have added ref: myRepo to the my-repo repository and have used reference $myRepo to the elasticSearch repository.

spec:
  sources:
    - repoURL: https://github.com/my-org/my-repo  # path is missing so no manifests are generated
      targetRevision: master
      ref: myRepo                                 # repo is available via symlink "myRepo"
    - repoURL: https://github.com/helm/charts
      targetRevision: master
      path: incubator/elasticsearch               # path "incubator/elasticsearch" is used to generate manifests
      helm:
        valueFiles:
          - $myRepo/values.yaml                   # values.yaml is located in source with reference name $myRepo

How to test manually

• run locally, use make start or make start-local
• Create an application with sources field. Below is one example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
  labels:
    argocd.argoproj.io/refresh: hard
spec:
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  sources:
    - chart: elasticsearch
      helm:
        valueFiles:
          - values.yaml
      repoURL: https://helm.elastic.co
      targetRevision: 7.6.0
    - repoURL: https://github.com/argoproj/argocd-example-apps.git
      path: guestbook
      targetRevision: HEAD
    - chart: elasticsearch
      helm:
        valueFiles:
          - values.yaml
      repoURL: https://helm.elastic.co
      targetRevision: 7.7.0

• Try different things - app update, rollback, remove sources and add a single source under source field, value files for helm sources, etc.

[codecov]

Codecov Report

Base: 47.01% // Head: 47.19% // Increases project coverage by +0.17% 🎉

Coverage data is based on head (bbb9240) compared to base (a2d756e).
Patch coverage: 60.24% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10432      +/-   ##
==========================================
+ Coverage   47.01%   47.19%   +0.17%     
==========================================
  Files         243      243              
  Lines       40587    41088     +501     
==========================================
+ Hits        19084    19393     +309     
- Misses      19583    19734     +151     
- Partials     1920     1961      +41     

Impacted Files	Coverage Δ
cmd/argocd/commands/applicationset.go	8.78% <0.00%> (-0.03%)	⬇️
cmd/util/app.go	55.57% <9.80%> (-0.01%)	⬇️
cmd/argocd/commands/app.go	20.35% <19.56%> (+0.01%)	⬆️
server/application/application.go	28.10% <24.56%> (+0.19%)	⬆️
util/webhook/webhook.go	68.28% <48.57%> (+2.72%)	⬆️
util/helm/client.go	47.51% <50.00%> (ø)
controller/appcontroller.go	51.98% <62.76%> (-0.36%)	⬇️
controller/state.go	67.76% <63.22%> (-4.12%)	⬇️
controller/sync.go	56.69% <63.63%> (-0.98%)	⬇️
reposerver/repository/repository.go	60.59% <70.22%> (+0.15%)	⬆️
... and 14 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ArshamTeymouri]

Thanks for the PR! I'm looking forward to v2.5.0.

Just a question about plugin, how will these remote sources work in case of using plugins? In our case, we use the helm-secrets plugin to fetch secrets directly from Vault by an argocd custom plugin. I wondered how I can integrate that with remote values files from git without doing so much manual scripting.

[amohamedhey]

Thanks for the PR!

[ishitasequeira]

Thanks for the PR! I'm looking forward to v2.5.0.

Just a question about plugin, how will these remote sources work in case of using plugins? In our case, we use the helm-secrets plugin to fetch secrets directly from Vault by an argocd custom plugin. I wondered how I can integrate that with remote values files from git without doing so much manual scripting.

Hi @amohamedhey, we are not going to support CMP with initial release of multiple sources in v2.5, but the feature would follow up as an extension to this feature in future releases.

[jannfis]

First round of superficial style check. PTAL.

I'm gonna test the PR locally.

[tback]

It looks as if this addresses #677, yet I didn't find the issue mentioned here.

[crenshaw-dev]

@tback updated the PR description, thanks!

[KrasimirAntonov]

Seeing this PR move forward, I think this proposal could be interesting.
Clone the different sources into the same {uuid} directory to access them with the repository name
Example:
If I have two git sources Test A (https://github.com/test/test-a.git) and Test B(https://github.com/test/test-b.git), I'd like the repo-server to clone them into /tmp/_argocd-repo/{uuid} like this:

/tmp/_argocd-repo/{uuid}/test-a
/tmp/_argocd-repo/{uuid}/test-b

Also, it would be interesting if repo-server first clone the repositories in the order they are defined in the sources field of the application.yaml file and then performs the apply or build tasks.

[crenshaw-dev]

@KrasimirAntonov I think it could definitely be done that day. It would also reduce lock contention, at the cost of disk space/time. What additional use case(es) would this solve?

[KrasimirAntonov]

Thanks for your quick response Michael!
In multi-repository approach, it resolves the deployment dependencies of the environments. Imagine I have two deployment environments (dev and pro) and these environments have common settings like license. With this function I can create a common environment/repository (let's call it: base) and inside it I can put the common configuration for all environments and change only once the license for all the environments. Significantly reduces time

[zenitraM]

I've got another related question (happy to create another issue if desired for this!) - are there any plans or ideas on how to expose multiple repos to custom CMPs?

The first obvious case I have in mind would be to use this with argocd-vault-plugin but also possibly with other custom CMPs. It would be nice if we were able to pass as a parameter the path to a file in another repo by reference somehow (to be able to replicate what the built-in Helm support does)

[crenshaw-dev]

@KrasimirAntonov sure, makes sense... I wonder if that's wandering too far into the territory of Argo CD becoming its own Config Management tool. But I'm curious. Would you be up for opening an issue for further discussion?

[crenshaw-dev]

@zenitraM good thought! That definitely deserves a new issue, since a lot of folks use Helm in their CMPs. Would you mind opening an issue?

[ShirMonDT]

Hey guys,
Been waiting for this PR!
Came across a problem with my use-case, I want to reference the same source with different versions, but I'm getting a comparison error;

Is there a reason for not being able to use the same repo with different refs?

[fad3t]

Hi,

That's a cool feature, thanks for this!
I'm trying to use multiple sources but I keep getting error messages such as:

• missing required field "source" in io.argoproj.v1alpha1.Application.spec]
• unknown field "sources" in io.argoproj.v1alpha1.Application.spec

I'm using ArgoCD argocd:v2.6.0-rc1.

Am I missing something?
Thx,
Fred

[jannfis]

Am I missing something?

@fad3t I think you may want to update your installation manifests, especially the CRDs, as well.

[fad3t]

Oh right, indeed :)

[danhawkins]

Love this feature, thanks so much

[omri-shilton]

any update on the UI side of this feature?

[crenshaw-dev]

@omri-shilton afaik this is the only ongoing frontend-related work: https://github.com/argoproj/argo-cd/pull/14124/files

Full support is "planned," but just in the sense that it obviously needs to be done - afaik there's no ongoing, concerted effort to write the necessary code.

[xgt001]

Thank you for this wonderful feature. Works dandy!