Refresh Failed: can not refresh repository which contains submodules cause argocd modified some files

[tanyaofei]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to initialize repository resources: rpc error: code = Internal desc = Failed to checkout FETCH_HEAD: `git submodule update --init --recursive` failed exit status 1: error: Your local changes to the following files would be overwritten by checkout: kustomization.yaml Please commit your changes or stash them before you switch branches. Aborting fatal: Unable to checkout '8315e94e5dbebe05e52301252cd7ac76b3298539' in submodule path 'acerp/overlays/prod'

Here is my Repository:

argocd.git

├── acerp
│   ├── base
│   └── overlays
│       ├── gray
│       ├── localdev
│       └── prod (This is a git submodule)

We have a Git pipeline: when prod.git repository was update, run git submodule update --remote for argocd.git repository. So the argocd.git always keeping the newest prod.git

To Reproduce

Argocd refresh will modify kustomization.yaml, such as adding new line at end of file, or adding namespace field in kustomization.yml if I config namespace in the Web UI.

git submodule update failed cause can not checkout for the modified submodule.

Expected behavior

Screenshots

Version

argocd: v2.9.3+6eba5be
BuildDate: 2023-12-01T23:05:50Z  GitCommit: 6eba5be864b7e031871ed7698f5233336dfe75c7
GitTreeState: clean  GoVersion: go1.21.3
Compiler: gc
Platform: linux/amd64

Logs

Paste any relevant application logs here.

[agaudreault]

Hey @tanyaofei, do you have any other logs on the repo-server components when it tries to generate your application? And can you share your Application manifest to help reproduce the issue.

Also, can you clarify

Argocd refresh will modify kustomization.yaml, such as adding new line at end of file, or adding namespace field in kustomization.yml if I config namespace in the Web UI.

What is done in the Web UI exactly?

And last question, is that a new error after an upgrade, did anything change recently in your deployment or it has always been like this?

[tanyaofei]

@agaudreault
Thanks for your kind response. Recently, we've been experimenting with using git submodules to deploy our production environment, aiming to prevent developers from accessing sensitive production configuration files. And our kustomization.yaml did not specify a namespace because our pre-production and production environments are on the same Kubernetes cluster, so we configure namespace in Web UI. This bug seems quite clear: argocd modified the kustomization.yaml, causing the git submodule checkout to fail and displaying Unable to checkout '8315e94e5dbebe05e52301252cd7ac76b3298539 at Last Sync on the page. Now we solved this problem by these following steps:

1. Explicitly specifying the namespace in the kustomization.yaml file;
2. Adding a newline at the end of each file.

I think argocd should rollback modifitions before updating submodule.