Argocd rejects app creation with "Unable to resolve to a commit SHA" even though branch exists

[KarstenSiemer]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

We are creating apps dynamically using the api and just updated from v1.8.4+28aea3d to v2.1.2+7af9dfb.
When a pull request is opened for a argocd managed repo a jenkins pipeline will do some validity tests of the PR and will then try to deploy the App of the PR to create commit status checks if the app gets healthy and stuff.
For that we dynamically create an app yaml and insert the PR branch as the source branch of the app.

Somewhere in between v1.8.4 and v2.1.0 argocd added or changed the way it is validity checking on apps before they get created but I failed to find the exact cause.
It takes at least 80 seconds after pushing a branch to GitHub until argocd will allow app creation.
Sometimes even longer, but around 2 minutes in general.

I failed to find what is exactly missing on the GitHub side that prevents argocd for so long to allow app creation.
Jenkins itself is already happily checking out the branch locally after pulling from GitHub, showing the commit sha's and is doing tests on the kustomize manifest.

The error is thrown by lsRemote as I always just the branch name e.g test9:

$argocd app create argocd-pipeline-pr-23 --file c.yaml
FATA[0000] rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Unknown desc = Unable to resolve 'test9' to a commit SHA

... and we were unable to resolve it to a commit SHA

But I don't get why that takes 2 minutes after push.
Doing something like this instantly shows the branch and it's corresponding commit sha on the remote site:

$git push --set-upstream origin test9 && git ls-remote
...
956a2...	refs/heads/test9

I added server and repoServer logs below.

What I am now missing is why that is actually checked? In prior versions ( and still if you directly create an app via CRD ) an app gets created that then runs into "Unkown" status, because it cannot find the branch in question. Which is a totally fine process imo. Plus, there is no way to deactivate that check via cli with --validate false for example.

Is anyone aware what argocd is missing to be able to create the app faster and can we speed up that process?

Anyways, big thanks for all your work on the argo projects!

To Reproduce

Try to create a new app with a source branch that was just created using the branch name ( only tested on GitHub private repos ).
I need to wait around 2 minutes until I am able to post the app.
I used this loop to test around:

git push --set-upstream origin test9 && for i in {1..150}; do date; echo $i; argocd app create argocd-pipeline-pr-23 --file c.yaml;sleep 1;done

test9 is a branch ( not default branch of the repo ) that includes a tiny kustomize manifest and nothing else.
App looks like so:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  name: argocd-pipeline-pr-23
  namespace: argocd
spec:
  destination:
    namespace: argocd-pipeline-pr-23
    server: https://kubernetes.default.svc
  project: argocd-pipeline
  source:
    kustomize:
      nameSuffix: -pr-23
    path: kustomizations/overlays/dev/
    repoURL: https://github.com/someOrga/gitops-argocd-pipeline.git
    targetRevision: test9
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true

The branch gets pushed and argocd cli failes around 50ish times until it eventually succeeds.

After testing around a bit, I found that if I set the targetRevision to e.g refs/heads/test9 it absolutely instantly works and finds the sha. Which is great and fixes my usecase as I can just prepend the branch with refs/heads/, but I wonder why it takes to long to find just test9?

Expected behavior

I expect to create argocd apps either directly putting the new app into "unkown" status or to be able to create an app once the branch is pullable using git cli.
Or using the branch name should be as fast as using refs/heads/branch as the targetRevision.

Version

argocd-server: v2.1.2+7af9dfb
  BuildDate: 2021-09-02T18:05:23Z
  GitCommit: 7af9dfb3524c13e941ab604e36e49a617fe47d2e
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.6.0+g7f2df64
  Kubectl Version: v0.21.0
  Jsonnet Version: v0.17.0

Logs

argocd-server-787d86dfd7-p5np7 server time="2021-09-22T11:50:23Z" level=info msg="received unary call /application.ApplicationService/Get" grpc.method=Get grpc.request.claims="{\"exp\":1632397823,\"iat\":1632311423,\"iss\":\"argocd\",\"jti\":\"6169f69f-ba1d-4106-94e1-7fe0e2971447\",\"nbf\":1632311423,\"sub\":\"admin\"}" grpc.request.content="name:\"argocd-pipeline-pr-19\" resourceVersion:\"\" selector:\"\" repo:\"\" " grpc.service=application.ApplicationService grpc.start_time="2021-09-22T11:50:23Z" span.kind=server system=grpc
argocd-server-787d86dfd7-p5np7 server time="2021-09-22T11:50:23Z" level=info msg="finished unary call with code NotFound" error="rpc error: code = NotFound desc = applications.argoproj.io \"argocd-pipeline-pr-19\" not found" grpc.code=NotFound grpc.method=Get grpc.service=application.ApplicationService grpc.start_time="2021-09-22T11:50:23Z" grpc.time_ms=6.218 span.kind=server system=grpc
argocd-server-787d86dfd7-p5np7 server time="2021-09-22T11:50:24Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"exp\":1632397823,\"iat\":1632311423,\"iss\":\"argocd\",\"jti\":\"6169f69f-ba1d-4106-94e1-7fe0e2971447\",\"nbf\":1632311423,\"sub\":\"admin\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2021-09-22T11:50:24Z" span.kind=server system=grpc
argocd-server-787d86dfd7-p5np7 server time="2021-09-22T11:50:24Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Unknown desc = Unable to resolve 'test5' to a commit SHA" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-09-22T11:50:24Z" grpc.time_ms=240.441 span.kind=server system=grpc
argocd-repo-server-7cfd679fdd-l64kk repo-server time="2021-09-22T11:50:24Z" level=error msg="finished unary call with code Unknown" error="Unable to resolve 'test5' to a commit SHA" grpc.code=Unknown grpc.method=GetAppDetails grpc.request.deadline="2021-09-22T11:51:24Z" grpc.service=repository.RepoServerService grpc.start_time="2021-09-22T11:50:24Z" grpc.time_ms=1.389 span.kind=server system=grpc

[alexmt]

Issue is fixed by #7244 . Fix is cherry-picked into 2.1 and will be released shortly as a patch release.

Thank you for reporting it!

[mdreizin]

It looks like we still have the same issue on 2.1.3 :(

Version

argocd: v2.1.3+d855831
  BuildDate: 2021-09-29T21:51:21Z
  GitCommit: d855831540e51d8a90b1006d2eb9f49ab1b088af
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64

Logs

time="2021-10-18T23:21:53Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="<redacted>" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:21:53Z" span.kind=server system=grpc
time="2021-10-18T23:21:58Z" level=info msg="autouser created application" application=<redacted> dest-namespace=<redacted> dest-server= reason=ResourceCreated type=Normal
time="2021-10-18T23:21:58Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:21:53Z" grpc.time_ms=5180.792 span.kind=server system=grpc
time="2021-10-18T23:22:03Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="<redacted>" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:22:03Z" span.kind=server system=grpc
time="2021-10-18T23:22:08Z" level=info msg="autouser created application" application=<redacted> dest-namespace=<redacted> dest-server= reason=ResourceCreated type=Normal
time="2021-10-18T23:22:08Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:22:03Z" grpc.time_ms=5038.199 span.kind=server system=grpc
time="2021-10-18T23:23:19Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="<redacted>" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:23:19Z" span.kind=server system=grpc
time="2021-10-18T23:23:20Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="<redacted>" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:23:20Z" span.kind=server system=grpc
time="2021-10-18T23:23:22Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in <redacted>: rpc error: code = Unknown desc = Unable to resolve '<redacted>' to a commit SHA" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:23:19Z" grpc.time_ms=2860.757 span.kind=server system=grpc
time="2021-10-18T23:23:24Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to generate manifests in <redacted>: rpc error: code = Unknown desc = Unable to resolve '<redacted>' to a commit SHA" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-10-18T23:23:20Z" grpc.time_ms=4118.85 span.kind=server system=grpc

[mdreizin]

@alexmt Unfortunately, it looks that issue is not fixed. I am happy to provide any details to help you to fix it completely :)

[alexmt]

Thanks you for update. Reopening. @mdreizin can you please check repo-server logs? Is error coming from GetAppDetails method ?

[alexmt]

Nevermind. How could I missed it 🤦 . API server makes two API calls to repo server and should not use cache revision in both: #7508

[mdreizin]

@alexmt I am sorry that I could not share the logs, because I was away from computer.

That is amazing that you figured out what was wrong and have already fixed the issue.

Is it possible to release a new version with this fix?

[mike-serchenia]

@alexmt seems to be the problem is still there (

time="2021-11-01T22:49:54Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1628558643,\"iss\":\"argocd\",\"jti\":\"fcf179c2-e3c7-439a-b0c9-20a0e0f8470e\",\"nbf\":1628558643,\"sub\":\"autouser\"}" grpc.request.content="%!v(PANIC=String method: reflect.Value.Interface: cannot return value obtained from unexported field or method)" grpc.service=application.ApplicationService grpc.start_time="2021-11-01T22:49:54Z" span.kind=server system=grpc
time="2021-11-01T22:49:54Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: Unable to get app details: rpc error: code = Unknown desc = Unable to resolve 'fbranch/test-run-08cb580' to a commit SHA" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2021-11-01T22:49:54Z" grpc.time_ms=436.257 span.kind=server system=grpc

argocd-server: v2.1.6+a346cf9
  BuildDate: 2021-10-28T19:59:40Z
  GitCommit: a346cf933e10d872eae26bff8e58c5e7ac40db25
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.6.0+g7f2df64
  Kubectl Version: v0.21.0
  Jsonnet Version: v0.17.0

also it looks like the problem appears less often then before 1 of 10 applies

[manabusakai]

I’m facing the same issue.

$ argocd version
(snip)
argocd-server: v2.1.6+a346cf9
  BuildDate: 2021-10-28T19:59:40Z
  GitCommit: a346cf933e10d872eae26bff8e58c5e7ac40db25
  GitTreeState: clean
  GoVersion: go1.16.5
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.21.0
  Jsonnet Version: v0.17.0

[mdreizin]

@alexmt could you please re-open it? :)

[vorozhko]

It seems issue is in {{branch_slug}}(targetRevision) - it transformed to lower case, but git expecting case to match.
Fixing branch for application manually resolved similar issue for us.

The {{branch_slug}} must keep original case of letters.

[dwgillies-bluescape]

I just had the problem myself. You know it would not be hard to generate a browser URL that could be clicked on or cut & paste into a browser to show the person that the target directory is empty. I would suggest doing that to help your users it could save a lot of time. Usually the config is pointed to a compiled directory of yamls on a release branch and sometimes the yamls for that particular instance are just not compiled, and it's not obvious. In my case I compiled it on a local branch in the shell but couldn't see the 2-lines in "git status" indicating they were yet to pushed to github because I had 25 other locally edited files in my branch. Giving the user a github link to clink (on the right branch, in the right path) would demonstrate why it failed, and would help your users, please, and it's probably a 5-mins change ...

[jannfis]

You know it would not be hard to generate a browser URL that could be clicked on [...]

Sure could be done for a couple of known providers (github.com, gitlab.com, maybe bitbucket.com). Then there are some other environments (GitHub enterprise, self hosted GitLab, Gitea, pure Git repos accessed via SSH, the list goes on).

Giving the user a github link to clink (on the right branch, in the right path) would demonstrate why it failed, and would help your users, please, and it's probably a 5-mins change ...

If it's a 5-mins change, you're welcome to contribute it :)

As an alternative, feel free to file an enhancement request.

[fernandino143]

Ok I'm also having this issue now running v 2.6.7 trying to create an app to connect to github. Is there a fix or a workaround I might be missing? I see people asking for reopening this

[dwgillies-bluescape]

It works - its just that the user interface is terribly bad and it waste's everybody lives. Here is a correctly configured github project page - when setting up the prject. You can put together the 3 fields plus the text "/tree/" to create a github url. Then paste it into your browser to make certain it is correct. You will get an error because your resulting URL is not correct. In that case, visit the correct page in your browser (starting from the top of your repo + your selected branch), and determine the correct 3 fields you need. See the before/after config picture below (url is correct for our project).

During Setup

After Successful Setup