Deployment is still progressing since Ingress resource is not considered as synced #1704

augabet · 2019-06-06T09:44:35Z

Deployment is still progressing since Ingress resource is not considered as synced

Describe the bug
We deploy an application in an "On Premise" Kubernetes using nginx Ingress. When deploying an application through Argo, the ingress object is still syncing although the application is available behind the Ingress.
Consequently, our deployment is still progressing in Argo.

We think that this is related to that PR https://github.com/argoproj/argo-cd/pull/1053/files#diff-d5a0105b0157a44898f0cd002d7d827dR157 and that issue #997 .

Our ingress controller is nginx and each ingress has the following status (which is a healthy status according to us):

...
status:
  loadBalancer: {}

To Reproduce

Deploy an application with an ingress on a Kubernetes cluster using an internal ingress controller nginx (an on premise cluster for example)
yaml of the ingress:

  kind: Ingress
  metadata:
    name: my-ingress
    namespace: myns
  spec:
    rules:
    - host: whatever.example.com
      http:
        paths:
        - backend:
            serviceName: my-service
            servicePort: 80
    tls:
    - hosts:
      - whatever.example.com

See error

Expected behavior
The ingress should be deployed in the Kubernetes cluster, and still be syncing in Argo. The Application deployment should still be progressing.

Screenshots
If applicable, add screenshots to help explain your problem.

Version

argocd: v1.0.1+5fe1447.dirty
  BuildDate: 2019-05-28T17:26:35Z
  GitCommit: 5fe1447b722716649143c63f9fc054886d5b111c
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.0.1+5fe1447.dirty
  BuildDate: 2019-05-28T17:27:38Z
  GitCommit: 5fe1447b722716649143c63f9fc054886d5b111c
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

Logs

/tmp/argocd-linux-amd64 app list
NAME               CLUSTER                         NAMESPACE             PROJECT  STATUS  HEALTH       SYNCPOLICY  CONDITIONS
white-application  https://kubernetes.default.svc  whiteapp-development  default  Synced  Progressing  <none>      <none>


/tmp/argocd-linux-amd64 app get white-application
Name:               white-application
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          whiteapp-development
URL:                https://cd.devops.caas.cagip.group.gca/applications/white-application
Repo:               https://scm.saas.cagip.group.gca/cagip/devops/tools-dashboard
Target:             HEAD
Path:               deploy
Sync Policy:        <none>
Sync Status:        Synced to HEAD (2b1892f)
Health Status:      Progressing

GROUP       KIND        NAMESPACE             NAME             STATUS  HEALTH
apps        Deployment  whiteapp-development  tools-dashboard  Synced  Healthy
extensions  Ingress     whiteapp-development  tools-dashboard  Synced  Progressing
            Service     whiteapp-development  tools-dashboard  Synced  Healthy

Have you thought about contributing a fix yourself?

Yes :)

Open Source software thrives with your contribution. It not only gives skills you might not be able to get in your day job, it also looks amazing on your resume.

If you want to get involved, check out the
contributing guide, then reach out to us on Slack so we can see how to get you started.

The text was updated successfully, but these errors were encountered:

alexec · 2019-06-06T15:06:09Z

Please see. https://argoproj.github.io/argo-cd/faq/

You can use this for the Ingress (thank you @stevesea )

 resource.customizations: |
    extensions/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs

augabet · 2019-06-06T15:17:24Z

This is perfect, many thanks :) @stevesea !

zetsub0u · 2020-04-15T20:41:17Z

I'm facing the same issue, can you explain where you set this block @alexec ? is this in the argo configmap?

stevesea · 2020-04-15T20:56:48Z

yes, it's in the argocd-cm configmap. Here's the relevant section of the documentation: https://argoproj.github.io/argo-cd/operator-manual/health/#way-1-define-a-custom-health-check-in-argocd-cm-configmap

zetsub0u · 2020-04-16T13:47:16Z

Huh, i don't seem to be able to get this to work, i see the snippet i added in the argocd-cm configmap correctly, i rebooted the argocd-server pods and im still seeing my ingress as progressing. What am i missing? This is with 1.5.1.

al26p · 2020-04-30T10:00:38Z

Me either, I don't know how to troubleshoot this. I added the resource customization, reloaded all pods related to argoCD.

I'm using HA-Proxy's ingress controller and Kubernetes' Ingress (not some fancy custom resource).

Kubernetes v1.17.0
ArgoCD v1.5.0+bdda410
argocd-cm :

data:
  resource.customizations: |
    Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs

I tried unsuccessfully with Ingress and extensions/Ingress.

jannfis · 2020-04-30T10:08:09Z

Hi. You need to make sure to apply the resource customization to the correct resource type.

I think recent versions of Kubernetes moved the Ingress resource type from the extensions API to the networking.k8s.io API, so what was

 resource.customizations: |
    extensions/Ingress:

before should now be

 resource.customizations: |
    networking.k8s.io/Ingress:

al26p · 2020-04-30T10:12:55Z

Hi,
You're right ! It now works :)
Thanks a lot for your quick answer !

bbqtk · 2020-05-15T16:41:04Z

Encountered the same problem, the above method can solve this problem, but there is a new problem. can not delete the application version 1.5.4

argoproj/argo-cd#1704 (comment)

enys · 2021-08-24T15:25:44Z

For Traefik users : https://doc.traefik.io/traefik/providers/kubernetes-ingress/#publishedservice
And this a boolean option in the helm chart.

fernferret · 2021-11-27T09:35:02Z

@enys Thanks for the great tip, I've ended up at this issue several times, but for those of us traefik users yours is the correct answer (as it actually fixes the issue rather than ignoring it). The documentation you linked is absolutely correct but when I read it I was still a bit confused so here are a few details for others that might stumble upon this issue.

The traefik publishedService config is applied to the kubernetesingress "controller". This flag tells traefik what IP address it is using for its ingress so that it can correctly update standard kubernetes Ingress objects with the IP address used for ingress. This can be seen in the source here

I was thinking to myself when first reading the documentation that I'd need to set that flag to the service that each downstream ingress object targeted, but the reality is that it just needs to be set to the namespace/service of traefik itself. As you noted it's very easy via the helm chart:

providers:
  kubernetesIngress:
    publishedService:
      enabled: true

This ends up just adding the following flag (assuming traefik is installed in the ingress-ns namespace and the traefik service is named traefik-svc) to traefik.

--providers.kubernetesingress.ingressendpoint.publishedservice=ingress-ns/traefik-svc

enys · 2021-11-27T12:24:26Z

@fernferret thanks and yes. Your write up here could be added to the docs !

benr · 2021-12-28T22:06:54Z

For K3S Users that should come across this that are confused about how to actually implement the fix above, edit /var/lib/rancher/k3s/server/manifests/traefik.yaml and add the above-provided lines, so that it looks roughly like this:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
  valuesContent: |-
    rbac:
      enabled: true
.....
    kubernetes:
      ingressEndpoint:
        useDefaultPublishedService: true
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
.....

Then restart K3S. You should then look at the config map ('kubectl -n kube-system get cm/traefik -o yaml') and you'll see these lines added:

    [kubernetes]
      [kubernetes.ingressEndpoint]
      publishedService = "kube-system/traefik"

Thanks @enys and @fernferret for the solution!

dalei2019 · 2022-05-19T15:58:08Z

For helm user, add this into traefik override.yaml:

additionalArguments:
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"

And upgrade traefik use helm.

dinesh0314 · 2022-05-24T12:13:18Z

For helm user, add this into traefik override.yaml:
additionalArguments:
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"
And upgrade traefik use helm.

for non helm charts.. Where we can add this in deploy file of traefik ? and let how to use this inside deploy file

dalei2019 · 2022-05-25T11:01:09Z

For helm user, add this into traefik override.yaml:
additionalArguments:
  - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"
And upgrade traefik use helm.
for non helm charts.. Where we can add this in deploy file of traefik ? and let how to use this inside deploy file

It's easy to find .

# helm upgrade -f override.yaml traefik traefik-10.9.1.tgz -n traefik --dry-run  |grep -B 100 publishedservice
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik
  labels:
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-10.9.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: traefik
  annotations:
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: traefik
      app.kubernetes.io/instance: traefik
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 0
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9100"
      labels:
        app.kubernetes.io/name: traefik
        helm.sh/chart: traefik-10.9.1
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/instance: traefik
    spec:
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      containers:
      - image: "traefik:2.5.6"
      ...
        args:
        ...
          - "--serverstransport.insecureskipverify=true"
          - "--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik"

See: argoproj/argo-cd#1704

source: argoproj/argo-cd#1704 (comment)

Joseph94m · 2023-04-20T13:42:47Z

For those who could still find this thread in 2023:
In my values.yaml file, i added:

configs:
  cm:
    resource.customizations: |
      networking.k8s.io/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs

It works now!

jdiemke · 2023-05-16T18:31:47Z

Shouldn't this be the default behavior? I dont think it makes sense that everyone has to adjust config maps.

emuchogu · 2023-06-24T13:44:20Z

resource.customizations:

configs:
  cm:
    resource.customizations: |
      networking.k8s.io/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs

It's 2023 and I still have this issue.

Is this put in the argocd values.yaml?
Where should I put this entry in the values.yaml file?

Can you provide a step by step guide for all future sufferers...

sxyandapp · 2023-06-25T03:55:20Z

resource.customizations:
configs:
  cm:
    resource.customizations: |
      networking.k8s.io/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs
It's 2023 and I still have this issue.

Is this put in the argocd values.yaml? Where should I put this entry in the values.yaml file?

Can you provide a step by step guide for all future sufferers...

直接修改这个configmap：argocd-cm

apiVersion: v1
data:
  resource.customizations: |
    networking.k8s.io/Ingress:
      health.lua: |
        hs = {}
        hs.status = "Healthy"
        return hs
 
kind: ConfigMap
metadata:
  name: argocd-cm

改完后argocd会自动生效

emuchogu · 2023-06-25T05:26:45Z

resource.customizations:
configs:
  cm:
    resource.customizations: |
      networking.k8s.io/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs
It's 2023 and I still have this issue.
Is this put in the argocd values.yaml? Where should I put this entry in the values.yaml file?
Can you provide a step by step guide for all future sufferers...
直接修改这个configmap：argocd-cm
apiVersion: v1
data:
  resource.customizations: |
    networking.k8s.io/Ingress:
      health.lua: |
        hs = {}
        hs.status = "Healthy"
        return hs
 
kind: ConfigMap
metadata:
  name: argocd-cm
改完后argocd会自动生效

This worked great.
Translation:

Modify this configmap directly: argocd-cm

apiVersion: v1
data:
  resource.customizations: |
    networking.k8s.io/Ingress:
      health.lua: |
        hs = {}
        hs.status = "Healthy"
        return hs
 
kind: ConfigMap
metadata:
  name: argocd-cm

argocd will automatically take effect after the modification

adv4000 · 2023-09-05T22:16:24Z

I using Helm Chart to deploy ArgoCD and I have nginx Ingress contoroller, I used this value.yaml file to fix this issue:

# Fixing issue with Stuck Processing for Ingress resource
server:
  config:
    resource.customizations: |
      networking.k8s.io/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs

After update, just Resync your applications Ingress resource.

fatsolko · 2024-04-01T14:25:36Z

I using Helm Chart to deploy ArgoCD and I have nginx Ingress contoroller, I used this value.yaml file to fix this issue:
# Fixing issue with Stuck Processing for Ingress resource
server:
  config:
    resource.customizations: |
      networking.k8s.io/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs        
After update, just Resync your applications Ingress resource.

I added that in ConfigMap argocd-cm and reload all pods with argoCD

data:
  resource.customizations: |
    networking.k8s.io/Ingress:
      health.lua: |
        hs = {}
        hs.status = "Healthy"
        return hs

but it doesn't help, ingress-nginx-controller still processing

adv4000 · 2024-04-01T16:40:11Z

See official info for future fix: https://argo-cd.readthedocs.io/en/stable/faq/#why-is-my-application-stuck-in-progressing-state

dlsniper · 2024-05-15T21:00:25Z

I'm sorry to revive this "obvious" thread, but I spent more than 2 hours trying to understand what I'm doing wrong and failed.

I'm using the latest stable version of Argo CD, v2.11.0, with the latest stable version of Argo Rollouts, v1.6.6, and trying this example: https://argoproj.github.io/argo-rollouts/getting-started/nginx/ with the NGINX Ingress Controller v1.10.1 installed via Helm.

I manually created the application in Argo CD and then bumped into the same issue as others in this thread.

I've read the documentation at https://argo-cd.readthedocs.io/en/stable/faq/#why-is-my-application-stuck-in-progressing-state

I also tried to apply the suggested fixes above but the issue persists.

Since this appears to be a recurring problem, can we please get a clear documentation on how to fix this and update the examples in the documentation with how to fix this too?

I'd be happy to send PRs to the documentation with the fix, but I don't know how to fix this in the first place :/

adv4000 · 2024-05-15T21:30:20Z

Hi @dlsniper

See how I deployed ArgoCD using Terraform, working fine for me: https://github.com/adv4000/argocd-terraform/tree/main/terraform_argocd_eks

dlsniper · 2024-05-15T22:46:01Z

@adv4000 thank you!

When using the latest version of the Helm chart, 6.9.2, the values.yaml file for the Helm chart should look something like this:

[...]
configs:
  cm:
    resource.customizations: |
      networking.k8s.io/Ingress:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          return hs
[...]

Thanks to argoproj/argo-helm#1872 (comment) I can now report that Argo CD works as expected.

zyfyy · 2024-06-05T15:22:20Z

@enys Thanks for the great tip, I've ended up at this issue several times, but for those of us traefik users yours is the correct answer (as it actually fixes the issue rather than ignoring it). The documentation you linked is absolutely correct but when I read it I was still a bit confused so here are a few details for others that might stumble upon this issue.

The traefik publishedService config is applied to the kubernetesingress "controller". This flag tells traefik what IP address it is using for its ingress so that it can correctly update standard kubernetes Ingress objects with the IP address used for ingress. This can be seen in the source here

I was thinking to myself when first reading the documentation that I'd need to set that flag to the service that each downstream ingress object targeted, but the reality is that it just needs to be set to the namespace/service of traefik itself. As you noted it's very easy via the helm chart:
providers:
  kubernetesIngress:
    publishedService:
      enabled: true
This ends up just adding the following flag (assuming traefik is installed in the ingress-ns namespace and the traefik service is named traefik-svc) to traefik.
--providers.kubernetesingress.ingressendpoint.publishedservice=ingress-ns/traefik-svc

great job! Just very helpful for traefik users!

thiagowfx · 2024-06-14T09:43:35Z

The

providers:
  kubernetesIngress:
    publishedService:
      enabled: true

solution did not work for me. Despite verifying that it is applied in the traefik deployment, k get ingress -n argocd -o yaml still has:

  status:
    loadBalancer: {}

...therefore the health check remains stuck at "Processing".

The custom health check solution ("resource.customizations") worked.

Let me know if you have any other ideas on how to debug the publishedService solution.

augabet added the bug Something isn't working label Jun 6, 2019

augabet closed this as completed Jun 6, 2019

W4lspirit added a commit to W4lspirit/kubernetes-training that referenced this issue Apr 11, 2021

Customize argocd configmap

0771b6d

argoproj/argo-cd#1704 (comment)

jonathankao97 mentioned this issue Apr 4, 2022

feat(argo-cd): add terraform pennlabs/infrastructure#146

Open

larsks added a commit to larsks/k8s-argocd that referenced this issue Sep 28, 2022

Avoid reporting ingress objects as always progressing

625a6ab

See: argoproj/argo-cd#1704

vehagn added a commit to vehagn/homelab that referenced this issue Jan 24, 2023

fix(traefik-argocd): Ingress stuck in 'Progressing' status

7576311

source: argoproj/argo-cd#1704 (comment)

vehagn added a commit to vehagn/homelab that referenced this issue Jan 24, 2023

fix(traefik-argocd): Ingress stuck in 'Progressing' status

c24329c

source: argoproj/argo-cd#1704 (comment)

vehagn added a commit to vehagn/homelab that referenced this issue Jan 24, 2023

fix(traefik-argocd): Ingress stuck in 'Progressing' status

48dea25

source: argoproj/argo-cd#1704 (comment)

vehagn added a commit to vehagn/homelab that referenced this issue Jan 24, 2023

fix(traefik-argocd): Ingress stuck in 'Progressing' status

1f0136a

source: argoproj/argo-cd#1704 (comment)

spantaleev mentioned this issue Jan 29, 2024

ArgoCD losing some of its settings after re-running cluster.yml kubernetes-sigs/kubespray#10846

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deployment is still progressing since Ingress resource is not considered as synced #1704

Deployment is still progressing since Ingress resource is not considered as synced #1704

augabet commented Jun 6, 2019

alexec commented Jun 6, 2019

augabet commented Jun 6, 2019

zetsub0u commented Apr 15, 2020

stevesea commented Apr 15, 2020

zetsub0u commented Apr 16, 2020

al26p commented Apr 30, 2020 •

edited

Loading

jannfis commented Apr 30, 2020

al26p commented Apr 30, 2020

bbqtk commented May 15, 2020

enys commented Aug 24, 2021

fernferret commented Nov 27, 2021 •

edited

Loading

enys commented Nov 27, 2021

benr commented Dec 28, 2021

dalei2019 commented May 19, 2022

dinesh0314 commented May 24, 2022

dalei2019 commented May 25, 2022

Joseph94m commented Apr 20, 2023

jdiemke commented May 16, 2023

emuchogu commented Jun 24, 2023 •

edited

Loading

sxyandapp commented Jun 25, 2023

emuchogu commented Jun 25, 2023 •

edited

Loading

adv4000 commented Sep 5, 2023

fatsolko commented Apr 1, 2024 •

edited

Loading

adv4000 commented Apr 1, 2024

dlsniper commented May 15, 2024

adv4000 commented May 15, 2024

dlsniper commented May 15, 2024 •

edited

Loading

zyfyy commented Jun 5, 2024

thiagowfx commented Jun 14, 2024

Deployment is still progressing since Ingress resource is not considered as synced #1704

Deployment is still progressing since Ingress resource is not considered as synced #1704

Comments

augabet commented Jun 6, 2019

alexec commented Jun 6, 2019

augabet commented Jun 6, 2019

zetsub0u commented Apr 15, 2020

stevesea commented Apr 15, 2020

zetsub0u commented Apr 16, 2020

al26p commented Apr 30, 2020 • edited Loading

jannfis commented Apr 30, 2020

al26p commented Apr 30, 2020

bbqtk commented May 15, 2020

enys commented Aug 24, 2021

fernferret commented Nov 27, 2021 • edited Loading

enys commented Nov 27, 2021

benr commented Dec 28, 2021

dalei2019 commented May 19, 2022

dinesh0314 commented May 24, 2022

dalei2019 commented May 25, 2022

Joseph94m commented Apr 20, 2023

jdiemke commented May 16, 2023

emuchogu commented Jun 24, 2023 • edited Loading

sxyandapp commented Jun 25, 2023

emuchogu commented Jun 25, 2023 • edited Loading

adv4000 commented Sep 5, 2023

fatsolko commented Apr 1, 2024 • edited Loading

adv4000 commented Apr 1, 2024

dlsniper commented May 15, 2024

adv4000 commented May 15, 2024

dlsniper commented May 15, 2024 • edited Loading

zyfyy commented Jun 5, 2024

thiagowfx commented Jun 14, 2024

al26p commented Apr 30, 2020 •

edited

Loading

fernferret commented Nov 27, 2021 •

edited

Loading

emuchogu commented Jun 24, 2023 •

edited

Loading

emuchogu commented Jun 25, 2023 •

edited

Loading

fatsolko commented Apr 1, 2024 •

edited

Loading

dlsniper commented May 15, 2024 •

edited

Loading