Skip to content

fix: restore failure-path metric recording for delete and HPA reconcile#8656

Merged
cnvergence merged 1 commit intoenvoyproxy:mainfrom
felipesabadini:fix/delete-metrics-failure-observability
Apr 2, 2026
Merged

fix: restore failure-path metric recording for delete and HPA reconcile#8656
cnvergence merged 1 commit intoenvoyproxy:mainfrom
felipesabadini:fix/delete-metrics-failure-observability

Conversation

@felipesabadini
Copy link
Copy Markdown
Contributor

What type of PR is this?

fix

What this PR does / why we need it

Follow-up to #8480 — restores failure-path observability that regressed when
the no-op delete metric fix was introduced.

Problem: Moving the metric-recording defer blocks to after existence checks
fixed no-op metric inflation, but failures on the pre-defer paths stopped being
recorded:

  • createOrUpdateHPA() returned on r.HorizontalPodAutoscaler() errors before
    the defer, so HPA render failures no longer incremented
    resource_apply_total{result="failure"}.
  • deleteDeployment(), deleteDaemonSet(), deleteHPA(), and deletePDB()
    returned on the pre-delete List() call before the defer, so list/RBAC/API
    failures no longer incremented resource_delete_total{result="failure"}.

Fix: Add explicit failure metric recording on the early-return error paths,
matching the pattern already used by createOrUpdateDeployment,
createOrUpdateDaemonSet, and createOrUpdatePodDisruptionBudget.

Changes

  • Add resourceApplyTotal.WithFailure(...) on the r.HorizontalPodAutoscaler()
    error path in createOrUpdateHPA()
  • Add resourceDeleteTotal.WithFailure(...) on the List() error paths in
    deleteDeployment(), deleteDaemonSet(), deleteHPA(), deletePDB()
  • Add TestDeleteListFailureReturnsError to verify error propagation when
    pre-delete List() calls fail

Which issue(s) this PR fixes

Fixes #8651

Release Notes

No

Fixes envoyproxy#8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
@netlify
Copy link
Copy Markdown

netlify bot commented Apr 2, 2026

Deploy Preview for cerulean-figolla-1f9435 canceled.

Name Link
🔨 Latest commit 1569c12
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/69ce4d0fe79c640008b0b693

@felipesabadini felipesabadini requested a review from a team as a code owner April 2, 2026 11:03
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 2, 2026

Codecov Report

❌ Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.44%. Comparing base (dd79a83) to head (1569c12).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...ternal/infrastructure/kubernetes/infra_resource.go 80.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8656   +/-   ##
=======================================
  Coverage   74.44%   74.44%           
=======================================
  Files         243      243           
  Lines       38350    38355    +5     
=======================================
+ Hits        28549    28553    +4     
- Misses       7804     7811    +7     
+ Partials     1997     1991    -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cnvergence cnvergence merged commit 2a5bfd0 into envoyproxy:main Apr 2, 2026
36 checks passed
rudrakhp pushed a commit to rudrakhp/gateway that referenced this pull request Apr 15, 2026
…le (envoyproxy#8656)

Fixes envoyproxy#8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
rudrakhp added a commit that referenced this pull request Apr 16, 2026
* fix: avoid metric increments on no-op delete reconcile paths (#8480)

* fix: avoid metric increments on no-op delete reconcile paths

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: restore failure-path metric recording for delete and HPA reconcile (#8656)

Fixes #8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: helm secrets rbac for gateway namespace with watch list of namespaces (#8706)

* fix: helm secrets rbac for gateway namespace with watch list of namespaces

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* add release notes

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* review update

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

---------

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Co-authored-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: handle network errors in rate limit e2e tests (#8446)

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: propagate the HTTPFilter translation errors to the outer layer (#7556)

* progate the HTTPFilter validation errors to the outer layer

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: return 500 error for invalid filters (#7605)

return 500 error for invalid filters

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: prevent configuring requestMirror filter and directResponse/RequestRedirect filter together (#7474)

* fix: prevent configuring RequestMirror and DirectResponse filters together

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

check redirect respose filter

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* address comments

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

* fix lint

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>

---------

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix bug with grpcroute mirror filter (#8541)

* fix bug with grpcroute mirror filter

Signed-off-by: Adam Buran <aburan@roblox.com>

* add indexers test

Signed-off-by: Adam Buran <aburan@roblox.com>

* add release note

Signed-off-by: Adam Buran <aburan@roblox.com>

---------

Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: normalize CRLF line endings in htpasswd basic auth secrets (#8557)

Fixes #8554

Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: status for mirror backend (#8675)

Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: set HTTPRoute Accepted condition as true with mixed invalid and valid rules (#7625)

* set HTTPRoute Accepted condition as true with mixed invalid and valid rules

Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* fix: basic auth validation (#8053)

* fix basic auth validation

Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

* [release/v1.6] fix gen check

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Signed-off-by: Huabing Zhao <zhaohuabing@gmail.com>
Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
Signed-off-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Co-authored-by: Felipe Sabadini <fsabadini@hotmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Karol Szwaj <karol.szwaj@gmail.com>
Co-authored-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com>
Co-authored-by: aburanrbx <aburan@roblox.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: stekole <30674956+stekole@users.noreply.github.com>
Co-authored-by: Kota Kimura <86363983+kkk777-7@users.noreply.github.com>
cnvergence pushed a commit to cnvergence/gateway that referenced this pull request Apr 16, 2026
…le (envoyproxy#8656)

Fixes envoyproxy#8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
(cherry picked from commit 2a5bfd0)
cnvergence pushed a commit to cnvergence/gateway that referenced this pull request Apr 16, 2026
…le (envoyproxy#8656)

Fixes envoyproxy#8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
(cherry picked from commit 2a5bfd0)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
cnvergence added a commit that referenced this pull request Apr 16, 2026
* fix: handle network errors in rate limit e2e tests (#8446)

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
(cherry picked from commit b0638d5)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* refactor/perf: use LuaPerRoute instead of FilterConfig (#8355)

perf: use LuaPerRoute instead of FilterConfig

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
(cherry picked from commit f31ac4e)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: per-endpoint hostname override blocked by auto-generated wildcad host (#8565)

* fix: per-endpoint hostname override blocked by auto-generated wildcard host

Signed-off-by: zirain <zirain2009@gmail.com>

* add UT

Signed-off-by: zirain <zirain2009@gmail.com>

---------

Signed-off-by: zirain <zirain2009@gmail.com>
(cherry picked from commit 595010a)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix bug with grpcroute mirror filter (#8541)

* fix bug with grpcroute mirror filter

Signed-off-by: Adam Buran <aburan@roblox.com>

* add indexers test

Signed-off-by: Adam Buran <aburan@roblox.com>

* add release note

Signed-off-by: Adam Buran <aburan@roblox.com>

---------

Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
(cherry picked from commit e633c08)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: normalize CRLF line endings in htpasswd basic auth secrets (#8557)

Fixes #8554

Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
(cherry picked from commit 9cac348)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: avoid metric increments on no-op delete reconcile paths (#8480)

* fix: avoid metric increments on no-op delete reconcile paths

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

* Update internal/infrastructure/kubernetes/infra_resource_test.go

Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>

---------

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
(cherry picked from commit 7a2a4ec)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix(telemetry): support BackendTLSPolicy for telemetry backends (#8545)

* fix(telemetry): support BackendTLSPolicy for telemetry backends

processBackendRefs does not look up BackendTLSPolicy for telemetry
backends (access logs, tracing, metrics), so TLS can only be configured
via Backend.spec.tls. Replace inline processServerValidationTLSSettings
with applyBackendTLSSetting so telemetry backends get the full Backend +
BackendTLSPolicy + EnvoyProxy TLS merge.

Workaround: envoyproxy/ai-gateway#1964

Signed-off-by: Adrian Cole <adrian@tetrate.io>
(cherry picked from commit ac18feb)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: restore failure-path metric recording for delete and HPA reconcile (#8656)

Fixes #8651

Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
(cherry picked from commit 2a5bfd0)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: status for mirror backend (#8675)

Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
(cherry picked from commit fa81778)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: client certificate secret never delivered when it is exclusively referenced by a SecurityPolicy extAuth Backend (#8654)

* fix: client certificate secret never delivered when it is exclusively referenced by a SecurityPolicy extAuth Backend

Signed-off-by: zirain <zirain2009@gmail.com>

* fix lint

Signed-off-by: zirain <zirain2009@gmail.com>

---------

Signed-off-by: zirain <zirain2009@gmail.com>
(cherry picked from commit c7e21fa)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: client certificate secret never delivered when it is exclusively referenced by a SecurityPolicy jwt/oidc Backend (#8711)

Signed-off-by: zirain <zirain2009@gmail.com>
(cherry picked from commit 95c3a79)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix: helm secrets rbac for gateway namespace with watch list of namespaces (#8706)

* fix: helm secrets rbac for gateway namespace with watch list of namespaces

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* add release notes

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* review update

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

---------

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Co-authored-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
(cherry picked from commit c48a346)
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* add release notes

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix lint

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

* fix lint

Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>

---------

Signed-off-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Signed-off-by: Karol Szwaj <karol.szwaj@gmail.com>
Signed-off-by: zirain <zirain2009@gmail.com>
Signed-off-by: Adam Buran <aburan@roblox.com>
Signed-off-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Signed-off-by: stekole <stefan@sandnetworks.com>
Signed-off-by: stekole <30674956+stekole@users.noreply.github.com>
Signed-off-by: Felipe Sabadini Facina <fsabadini@hotmail.com>
Signed-off-by: Isaac Wilson <isaac.wilson514@gmail.com>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
Signed-off-by: kkk777-7 <kota.kimura0725@gmail.com>
Co-authored-by: Rudrakh Panigrahi <rudrakh97@gmail.com>
Co-authored-by: zirain <zirain2009@gmail.com>
Co-authored-by: aburanrbx <aburan@roblox.com>
Co-authored-by: Arko Dasgupta <arkodg@users.noreply.github.com>
Co-authored-by: stekole <30674956+stekole@users.noreply.github.com>
Co-authored-by: Felipe Sabadini <fsabadini@hotmail.com>
Co-authored-by: Isaac Wilson <isaac.wilson514@gmail.com>
Co-authored-by: Adrian Cole <64215+codefromthecrypt@users.noreply.github.com>
Co-authored-by: Kota Kimura <86363983+kkk777-7@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Follow ups for #8438

4 participants