Skip to content

fix: avoid metric increments on no-op delete reconcile paths#8480

Merged
jukie merged 5 commits intoenvoyproxy:mainfrom
felipesabadini:fix/delete-metrics-noop-reconcile
Mar 31, 2026
Merged

fix: avoid metric increments on no-op delete reconcile paths#8480
jukie merged 5 commits intoenvoyproxy:mainfrom
felipesabadini:fix/delete-metrics-noop-reconcile

Conversation

@felipesabadini
Copy link
Copy Markdown
Contributor

What type of PR is this?

fix: avoid metric increments on no-op delete reconcile paths

What this PR does / why we need it

This PR fixes metric inflation during no-op reconciles for optional resources.

Previously, deleteDeployment, deleteDaemonSet, deleteHPA, and deletePDB always called DeleteAllOf and recorded resource_delete_* success metrics, even when no matching resources existed. In default Deployment mode (without DaemonSet/HPA/PDB), this caused delete metrics to grow continuously despite no real deletions.

Additionally, createOrUpdateHPA and createOrUpdatePodDisruptionBudget registered apply metric defers before nil checks, causing resource_apply_total to be incremented on the nil->delete path.

Changes

  • Add pre-delete existence checks (List with the same namespace/label selector) in:
    • deleteDeployment
    • deleteDaemonSet
    • deleteHPA
    • deletePDB
  • If no matching resources exist, return early and skip delete metric recording.
  • Move apply-metric defer blocks in:
    • createOrUpdateHPA
    • createOrUpdatePodDisruptionBudget
      to run only in actual apply paths (after nil checks), matching Deployment/DaemonSet behavior.

Which issue(s) this PR fixes

Fixes #8438

Release Notes

No

@felipesabadini felipesabadini requested a review from a team as a code owner March 10, 2026 23:24
@netlify
Copy link
Copy Markdown

netlify bot commented Mar 10, 2026

Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit c528d5b
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/69cbf9126b753700080fa964
😎 Deploy Preview https://deploy-preview-8480--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
@felipesabadini felipesabadini force-pushed the fix/delete-metrics-noop-reconcile branch from 34ab03c to 3d91d12 Compare March 10, 2026 23:31
@felipesabadini
Copy link
Copy Markdown
Contributor Author

Validated in a local Kind test cluster (default Deployment mode, no HPA/PDB/DaemonSet configured).

  • Deployed Envoy Gateway with this fix.
  • Checked /metrics after startup and again after ~60s of reconcile loops.
  • Re-checked across multiple reconcile cycles.

Observed:

  • resource_delete_total: no increments for DaemonSet, HPA, or PDB (previously these grew continuously on no-op reconciles).
  • resource_apply_total: no increments for HPA/PDB on nil -> delete paths (previously incremented incorrectly).
  • Counters remained stable over time.

zirain
zirain previously approved these changes Mar 11, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 11, 2026

Codecov Report

❌ Patch coverage is 70.96774% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.34%. Comparing base (c86eb78) to head (c528d5b).
⚠️ Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
...ternal/infrastructure/kubernetes/infra_resource.go 70.96% 10 Missing and 8 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8480      +/-   ##
==========================================
- Coverage   74.38%   74.34%   -0.05%     
==========================================
  Files         243      243              
  Lines       38227    38262      +35     
==========================================
+ Hits        28436    28444       +8     
- Misses       7802     7823      +21     
- Partials     1989     1995       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@arkodg arkodg added this to the v1.8.0-rc.1 Release milestone Mar 15, 2026
@arkodg arkodg requested review from a team March 15, 2026 02:08
@kkk777-7
Copy link
Copy Markdown
Member

Hi @felipesabadini, thanks for working this!
Overall LGTM, could you fix lint?

Copy link
Copy Markdown
Contributor

@jukie jukie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will fix linting issues:

Comment thread internal/infrastructure/kubernetes/infra_resource_test.go Outdated
Comment thread internal/infrastructure/kubernetes/infra_resource_test.go Outdated
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
@jukie jukie requested review from jukie and zirain March 23, 2026 13:19
@jukie
Copy link
Copy Markdown
Contributor

jukie commented Mar 28, 2026

/retest

@jukie jukie merged commit 7a2a4ec into envoyproxy:main Mar 31, 2026
57 of 59 checks passed
@arkodg
Copy link
Copy Markdown
Contributor

arkodg commented Apr 2, 2026

hey @felipesabadini can you help with these follow ups

  • createOrUpdateHPA() now returns directly on r.HorizontalPodAutoscaler()
    errors before the defer is installed, so HPA render/patch failures no longer
    increment resource_apply_total{result="failure"}.
  • deleteDeployment(), deleteDaemonSet(), deleteHPA(), and deletePDB()
    now return on the pre-delete List() call before the delete-metrics defer is
    installed, so list/RBAC/API failures no longer increment
    resource_delete_total{result="failure"}.

#8438 looks fixed for the no-op delete case, but failure-path observability
regressed

@arkodg arkodg mentioned this pull request Apr 2, 2026
rudrakhp pushed a commit to rudrakhp/gateway that referenced this pull request Apr 15, 2026
…oxy#8480)

* fix: avoid metric increments on no-op delete reconcile paths

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
rudrakhp added a commit that referenced this pull request Apr 16, 2026
* fix: avoid metric increments on no-op delete reconcile paths (#8480)

* fix: avoid metric increments on no-op delete reconcile paths

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: restore failure-path metric recording for delete and HPA reconcile (#8656)

Fixes #8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: helm secrets rbac for gateway namespace with watch list of namespaces (#8706)

* fix: helm secrets rbac for gateway namespace with watch list of namespaces

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* add release notes

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* review update

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

---------

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Co-authored-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: handle network errors in rate limit e2e tests (#8446)

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: propagate the HTTPFilter translation errors to the outer layer (#7556)

* progate the HTTPFilter validation errors to the outer layer

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: return 500 error for invalid filters (#7605)

return 500 error for invalid filters

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: prevent configuring requestMirror filter and directResponse/RequestRedirect filter together (#7474)

* fix: prevent configuring RequestMirror and DirectResponse filters together

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

check redirect respose filter

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* address comments

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix bug with grpcroute mirror filter (#8541)

* fix bug with grpcroute mirror filter

Signed-off-by: Adam Buran <aburan@roblox.com>

* add indexers test

Signed-off-by: Adam Buran <aburan@roblox.com>

* add release note

Signed-off-by: Adam Buran <aburan@roblox.com>

---------

Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: normalize CRLF line endings in htpasswd basic auth secrets (#8557)

Fixes #8554

Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: status for mirror backend (#8675)

Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: set HTTPRoute Accepted condition as true with mixed invalid and valid rules (#7625)

* set HTTPRoute Accepted condition as true with mixed invalid and valid rules

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: basic auth validation (#8053)

* fix basic auth validation

Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* [release/v1.6] fix gen check

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Co-authored-by: Felipe Sabadini <fsabadini@hotmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Karol Szwaj <karol.szwaj@gmail.com>
Co-authored-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Co-authored-by: aburanrbx <aburan@roblox.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: stekole <30674956+stekole@users.noreply.github.com>
Co-authored-by: Kota Kimura <86363983+kkk777-7@users.noreply.github.com>
cnvergence pushed a commit to cnvergence/gateway that referenced this pull request Apr 16, 2026
…oxy#8480)

* fix: avoid metric increments on no-op delete reconcile paths

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
(cherry picked from commit 7a2a4ec)
cnvergence pushed a commit to cnvergence/gateway that referenced this pull request Apr 16, 2026
…oxy#8480)

* fix: avoid metric increments on no-op delete reconcile paths

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
(cherry picked from commit 7a2a4ec)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
cnvergence added a commit that referenced this pull request Apr 16, 2026
* fix: handle network errors in rate limit e2e tests (#8446)

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
(cherry picked from commit b0638d5)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* refactor/perf: use LuaPerRoute instead of FilterConfig (#8355)

perf: use LuaPerRoute instead of FilterConfig

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
(cherry picked from commit f31ac4e)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: per-endpoint hostname override blocked by auto-generated wildcad host (#8565)

* fix: per-endpoint hostname override blocked by auto-generated wildcard host

Signed-off-by: zirain <zirain2009@gmail.com>

* add UT

Signed-off-by: zirain <zirain2009@gmail.com>

---------

Signed-off-by: zirain <zirain2009@gmail.com>
(cherry picked from commit 595010a)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix bug with grpcroute mirror filter (#8541)

* fix bug with grpcroute mirror filter

Signed-off-by: Adam Buran <aburan@roblox.com>

* add indexers test

Signed-off-by: Adam Buran <aburan@roblox.com>

* add release note

Signed-off-by: Adam Buran <aburan@roblox.com>

---------

Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
(cherry picked from commit e633c08)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: normalize CRLF line endings in htpasswd basic auth secrets (#8557)

Fixes #8554

Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
(cherry picked from commit 9cac348)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: avoid metric increments on no-op delete reconcile paths (#8480)

* fix: avoid metric increments on no-op delete reconcile paths

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
(cherry picked from commit 7a2a4ec)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix(telemetry): support BackendTLSPolicy for telemetry backends (#8545)

* fix(telemetry): support BackendTLSPolicy for telemetry backends

processBackendRefs does not look up BackendTLSPolicy for telemetry
backends (access logs, tracing, metrics), so TLS can only be configured
via Backend.spec.tls. Replace inline processServerValidationTLSSettings
with applyBackendTLSSetting so telemetry backends get the full Backend +
BackendTLSPolicy + EnvoyProxy TLS merge.

Workaround: envoyproxy/ai-gateway#1964

Signed-off-by: Adrian Cole <adrian@tetrate.io>
(cherry picked from commit ac18feb)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: restore failure-path metric recording for delete and HPA reconcile (#8656)

Fixes #8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
(cherry picked from commit 2a5bfd0)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: status for mirror backend (#8675)

Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
(cherry picked from commit fa81778)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: client certificate secret never delivered when it is exclusively referenced by a SecurityPolicy extAuth Backend (#8654)

* fix: client certificate secret never delivered when it is exclusively referenced by a SecurityPolicy extAuth Backend

Signed-off-by: zirain <zirain2009@gmail.com>

* fix lint

Signed-off-by: zirain <zirain2009@gmail.com>

---------

Signed-off-by: zirain <zirain2009@gmail.com>
(cherry picked from commit c7e21fa)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: client certificate secret never delivered when it is exclusively referenced by a SecurityPolicy jwt/oidc Backend (#8711)

Signed-off-by: zirain <zirain2009@gmail.com>
(cherry picked from commit 95c3a79)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: helm secrets rbac for gateway namespace with watch list of namespaces (#8706)

* fix: helm secrets rbac for gateway namespace with watch list of namespaces

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* add release notes

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* review update

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

---------

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Co-authored-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
(cherry picked from commit c48a346)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* add release notes

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix lint

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix lint

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

---------

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Signed-off-by: zirain <zirain2009@gmail.com>
Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
Co-authored-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
Co-authored-by: aburanrbx <aburan@roblox.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: stekole <30674956+stekole@users.noreply.github.com>
Co-authored-by: Felipe Sabadini <fsabadini@hotmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Adrian Cole <64215+codefromthecrypt@users.noreply.github.com>
Co-authored-by: Kota Kimura <86363983+kkk777-7@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Delete metrics increment on no-op reconcile when DaemonSet/HPA/PDB are not configured

5 participants