Repo-server has silent unmarshalling errors leading to empty applications #4423
Assignees
Labels
Milestone
Comments
jessesuen
added
bug
@alexmt and I agree we should introduce a new field in the cached manifest datastructure which contains a checksum of the rendered JSON manifests. This checksum would be verified whenever we retrieve manifests it from cache. /cc @jgwest this might be a good one for you |
|
@jessesuen +1, feel free to assign to me. |
7 tasks
7 tasks
jgwest
added a commit
to jgwest/argo-cd
that referenced
this issue
Oct 29, 2020
jgwest
added a commit
to jgwest/argo-cd
that referenced
this issue
Oct 29, 2020
terrycorley
pushed a commit
to terrycorley/argo-cd
that referenced
this issue
Nov 3, 2020
…zation-generators * 'master' of github.com:argoproj/argo-cd: fix: RevisionFormField component crashes in 'refs' API returns no tags (argoproj#4735) docs: add Opensurvey to USERS.md (argoproj#4727) docs: correct parameters usage in CLI (argoproj#4725) fix: Repo-server has silent unmarshalling errors leading to empty applications (argoproj#4423) (argoproj#4708) fix: inject artificial delay between sync waves to better support health assessments (argoproj#4715) fix: exclude files listed under exclusions (argoproj#4686) feat: support resource actions on CRDs that use status subresources (argoproj#4690) feat: Add autocomplete for repo Revisions (argoproj#4645) (argoproj#4713) fix: webhook don't refresh apps pointing to HEAD (argoproj#4717) feat: Add support for ExecProvider cluster auth (argoproj#4600) (argoproj#4710) fix: adding helm values file in New App (argoproj#4635) docs: Instructions on `make verify-kube-connect` step when using k3d (argoproj#4687) feat: Annotation based app paths detection in webhooks (argoproj#4699) fix: adding commonAnnotations for Kustomize (argoproj#4613)
|
Implemented by #4708 |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
We noticed a situation where when retrieving cached manifests from redis, the cache silently returned an empty datastructure of cached manifests, despite there being content in the redis key. This made it appear that the manifests at the commit SHA had zero manifests, and Argo CD wanted to prune everything.
When investigating the issue, the problem was traced to the
github.com/vmihailenco/msgpacklibrary. When connecting directly to redis, it was seen that there was partial text, so there was indeed content in the database for the key. However, when msgpack deserialize it, it returned an empty struct without returning an error, which was interpreted by the repo server as having no manifests.To Reproduce
This is not easy to reproduce.
Expected behavior
I think we may need to add some corruption verification, or at least a basic struct check that validates what we retrieve from redis is valid. For example, it could be a checksum against the manifests.
Version
v1.7
The text was updated successfully, but these errors were encountered: