Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v2.5.0-rc1 HA argocd-repo-server CrashLoopBackOff #10882

Open
3 tasks done
ohauer opened this issue Oct 10, 2022 · 14 comments
Open
3 tasks done

v2.5.0-rc1 HA argocd-repo-server CrashLoopBackOff #10882

ohauer opened this issue Oct 10, 2022 · 14 comments
Labels
bug Something isn't working regression Bug is a regression, should be handled with high priority

Comments

@ohauer
Copy link

ohauer commented Oct 10, 2022 •
edited

Checklist:

  • I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  • I've included steps to reproduce the bug.
  • I've pasted the output of argocd version.

Describe the bug
Update argocd from 2.4.14 to 2.5.0-rc1 via kustomize
old url: https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.14/manifests/ha/install.yaml
new url: https://raw.githubusercontent.com/argoproj/argo-cd/v2.5.0-rc1/manifests/ha/install.yaml

Everything is comming up except argocd-repo-server, log can be found below

Expected behavior
No issues with gpg inside the repo server

Version

argocd: v2.5.0-rc1+2bf51f4
  BuildDate: 2022-10-05T17:41:04Z
  GitCommit: 2bf51f401d6700f8e8b9565d9fc3f66dcf60a0b6
  GitTreeState: clean
  GoVersion: go1.18.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.5.0-rc1+2bf51f4
  BuildDate: 2022-10-05T17:14:51Z
  GitCommit: 2bf51f401d6700f8e8b9565d9fc3f66dcf60a0b6
  GitTreeState: clean
  GoVersion: go1.18.7
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.0+gce66412
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

Logs

k -n argocd logs -f argocd-repo-server-7b4d9b979-cf8dg
time="2022-10-10T14:15:51Z" level=info msg="ArgoCD Repository Server is starting" built="2022-10-05T17:14:51Z" commit=2bf51f401d6700f8e8b9565d9fc3f66dcf60a0b6 port=8081 version=v2.5.0-rc1+2bf51f4
time="2022-10-10T14:15:51Z" level=info msg="Generating self-signed TLS certificate for this session"
time="2022-10-10T14:15:51Z" level=info msg="Initializing GnuPG keyring at /app/config/gpg/keys"
time="2022-10-10T14:15:51Z" level=info msg="gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe735869143" dir= execID=30201
time="2022-10-10T14:15:57Z" level=error msg="`gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe735869143` failed exit status 2" execID=30201
time="2022-10-10T14:15:57Z" level=info msg=Trace args="[gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe735869143]" dir= operation_name="exec gpg" time_ms=6010.954976
@ohauer ohauer added the bug Something isn't working label Oct 10, 2022
@crenshaw-dev crenshaw-dev added the regression Bug is a regression, should be handled with high priority label Oct 10, 2022
@crenshaw-dev crenshaw-dev added this to the v2.5 milestone Oct 10, 2022
@crenshaw-dev
Copy link
Collaborator

This might not be specific to 2.5.0-rc1: #9809

@crenshaw-dev
Copy link
Collaborator

I'm curious if this workaround works for you: #9809 (comment)

@crenshaw-dev crenshaw-dev removed this from the v2.5 milestone Oct 17, 2022
@benjamin-bergia
Copy link

Hi, I just upgraded from 2.4 to 2.5 this morning I am also hitting this issue. The workaround you mentioned @crenshaw-dev still works.

@aprams
Copy link

aprams commented Oct 27, 2022 •
edited

Just as @benjamin-bergia I had the same issue upgrading v2.4.14 to v2.5.0 just now.
The workaround that you mentioned @crenshaw-dev did not work for me.

@yonahd
Copy link

yonahd commented Oct 27, 2022

Having this issue as well

@benjamin-bergia
Copy link

@aprams In my case I couldn't remove the field like mentioned in the workaround. Instead I replaced the value of /spec/template/spec/containers/0/securityContext/seccompProfile/type by Unconfined on the argocd-repo-server Deployment.

@crenshaw-dev
Copy link
Collaborator

If anyone who has encountered this has time to run a custom build of Argo CD, I'd appreciate the help debugging. I think we could change this command to add --debug-level guru:

argo-cd/util/gpg/gpg.go

Line 283 in 309654c

cmd := exec.Command("gpg", "--no-permission-warning", "--logger-fd", "1", "--batch", "--gen-key", f.Name())

Alternatively, we could check logs to find out what syscall seccomp is blocking. I haven't yet found the docs for how/where to find those logs, but I recall reading that they exist.

@aprams
Copy link

aprams commented Oct 27, 2022

@benjamin-bergia thanks a lot, that worked nicely.
@crenshaw-dev I tried to hack my way around having to do a custom build (changing the entrypoint to do that/opening a shell and running the command with debug options), but I did not succeed in a reasonable timeframe, sorry! Might look into it again

@an-tex
Copy link

an-tex commented Nov 2, 2022 •
edited

this worked for me #9809 (comment) (upgrading from 2.4.11 to 2.5.1)

@henrik-koren
Copy link

I am hitting this issues on version 2.5.5 and on v2.6.0-rc1
I try to apply #9809 without any success

@yonahd
Copy link

yonahd commented Jan 1, 2023 •
edited

@crenshaw-dev when running with debug guru I get

time="2023-01-01T15:27:27Z" level=info msg="ArgoCD Repository Server is starting" built="2023-01-01T15:24:37Z" commit=7c8dd73c04d072bbe897b008348d36d3dee29198 port=8081 version=v2.6.0+7c8dd73.dirty
time="2023-01-01T15:27:27Z" level=info msg="Generating self-signed TLS certificate for this session"
time="2023-01-01T15:27:27Z" level=info msg="Initializing GnuPG keyring at /app/config/gpg/keys"
time="2023-01-01T15:27:27Z" level=info msg="gpg --debug-level guru --no-permission-warning --batch --gen-key /tmp/gpg-key-recipe1943744750" dir= execID=bee42
time="2023-01-01T15:27:33Z" level=error msg="gpg --debug-level guru --no-permission-warning --batch --gen-key /tmp/gpg-key-recipe1943744750 failed exit status 2: gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust hashing ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: fd_cache_invalidate (/app/config/gpg/keys/pubring.kbx)
gpg: DBG: iobuf-1.0: open '/app/config/gpg/keys/pubring.kbx' desc=file_filter(fd) fd=3
gpg: DBG: iobuf-1.0: close 'file_filter(fd)'
gpg: DBG: /app/config/gpg/keys/pubring.kbx: close fd/handle 3
gpg: DBG: fd_cache_close (/app/config/gpg/keys/pubring.kbx) new slot created
gpg: DBG: iobuf-.: ioctl '/app/config/gpg/keys/pubring.kbx' invalidate
gpg: DBG: fd_cache_invalidate (/app/config/gpg/keys/pubring.kbx)
gpg: DBG: did (/app/config/gpg/keys/pubring.kbx)
gpg: keybox '/app/config/gpg/keys/pubring.kbx' created
gpg: DBG: fd_cache_open (/tmp/gpg-key-recipe1943744750) not cached
gpg: DBG: iobuf-2.0: open '/tmp/gpg-key-recipe1943744750' desc=file_filter(fd) fd=3
gpg: DBG: iobuf-2.0: ioctl 'file_filter(fd)' no_cache=1
gpg: DBG: iobuf-2.0: underflow: buffer size: 8192; still buffered: 0 => space for 8192 bytes
gpg: DBG: iobuf-2.0: underflow: A->FILTER (8192 bytes)
gpg: DBG: iobuf-2.0: A->FILTER() returned rc=0 (ok), read 196 bytes
gpg: DBG: chan_4 <- [eof]
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: DBG: chan_4 -> BYE
gpg: can't connect to the agent: End of file
gpg: agent_genkey failed: No agent running
gpg: key generation failed: No agent running
gpg: DBG: iobuf-2.0: underflow: buffer size: 8192; still buffered: 0 => space for 8192 bytes
gpg: DBG: iobuf-2.0: underflow: A->FILTER (8192 bytes)
gpg: DBG: iobuf-2.0: A->FILTER() returned rc=-1 (EOF), read 0 bytes
gpg: DBG: /tmp/gpg-key-recipe1943744750: close fd/handle 3
gpg: DBG: fd_cache_close (3) real
gpg: DBG: iobuf-2.0: close '?'
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=0 locks=0 parse=0 get=0
gpg: build=0 update=0 insert=0 delete=0
gpg: reset=0 found=0 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=0 cached=0 good=0 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/65536 bytes in 0 blocks" execID=bee4

@JoseRIvera07
Copy link

same issue in argocd v2.5.2 I tried to use the Unconfined option but argocd-repo-server is not starting cause:
Error creating: pods "argo-cd-argocd-server-7457cc7f76-kpphl" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/server: Forbidden: unconfined is not a valid seccomp profile.

@yonahd
Copy link

yonahd commented Jan 5, 2023 •
edited

@JoseRIvera07 What version is your kubernetes?

@yonahd
Copy link

yonahd commented Mar 6, 2023

Using strace I can see

time="2023-03-06T07:52:57Z" level=info msg="gpg --debug-level guru --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe3101903093" dir= execID=16375
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=9, si_uid=999} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24, si_uid=999, si_status=2, si_utime=0, si_stime=0} ---

But it won't tell me what syscall is blocked

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working regression Bug is a regression, should be handled with high priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@crenshaw-dev @ohauer @an-tex @benjamin-bergia @aprams @henrik-koren @JoseRIvera07 @yonahd