Completely remove all resources for any type (listener, route, secret etc)

[pravinpushkar]

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: One line description

Description:

How to update configurations when all resources are removed for any particular type or all types.

[optional Relevant Links:]

I was reading this issues which is related to SDS only - #24373

The behavior that I am getting -

When I remove all the resources of all types or any particular types. The configuration does not get updated to envoy. I check the go-control-plane logs it seems the CP is sending the right data for removed resources but not sure why it is getting ignored in envoy.

I above issue regarding SDS seems relevant and it looks like envoy ignores the complete removal of resources for any types.

From Documentation of DeltaDiscoveryResponse - It says "Removed resources for missing resources can be ignored" What does it mean exactly?

[RyanTheOptimist]

CC: @adisuissa @htuch

[adisuissa]

Not sure I fully understand your problem, so I'll try to answer your question, and feel free to provide more details.

From Documentation of DeltaDiscoveryResponse - It says "Removed resources for missing resources can be ignored" What does it mean exactly?

Envoy uses the xDS-protocol to subscribe to resources. There are essentially 2 types of subscriptions: collections/wildcards (CDS, LDS in Envoy work like that) and singletons (non-collections, such as EDS, SDS, RDS). Specifically in Envoy, the collections' resources can exist without having anything
In the context of a secret provided by SDS, it needs some other resource that "points to it", for example, a cluster. If that cluster is removed (say by a CDS update), and no other resource needs that secret, the cluster and the secret are removed.
Envoy will send an unsubscribe from the secret request, but the response is moot, as the server cannot override the request to unsubscribe.

[adisuissa]

Might be worth mentioning 2 recently related PRs: #32961 and #33362.

[pravinpushkar]

@adisuissa If I understood you then In context of SDS - If I just remove a secret but that secret is being pointed by other resource like LDS/CDS then in this case secret data will not be removed from envoy configurations. Is it correct ?

To give more details of the issue that I am facing -

I am trying to use gateway project and this installation guide - https://gateway.envoyproxy.io/latest/tasks/quickstart/

It just create resources like HTTPRoute , gateway and gatewayclass. These things gets converted to envoy proxy configurations.
To apply these resources to k8s cluster - kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default
Now, to remove all these configurations from k8s cluster and thus from envoy proxy, I am doing -

kubectl delete -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default

But to my surprise, envoy config does not get updated instead and it keeps showing the older config. I was debugging the go-control-plane server.go. and it seems it is correctly detecting which resources are to be deleted but envoy side there is effect.

few logs from go-control-plane, these are my logs -

 send getResources []
 send getRemovedResources [httproute/default/backend/rule/0]

[adisuissa]

@adisuissa If I understood you then In context of SDS - If I just remove a secret but that secret is being pointed by other resource like LDS/CDS then in this case secret data will not be removed from envoy configurations. Is it correct ?

The other way around, say cluster_0 was removed and that cluster had a secret, secret_0. When cluster_0 is removed, Envoy should automatically remove secret_0. Envoy will also send a discovery request to the xDS server stating that it wants to unsubscribe from the resource. The xDS server will probably send a response that removes secret_0, but it is essentially moot, as the resource was already removed.

[pravinpushkar]

@adisuissa Thanks for the response on secret/SDS update. It is indeed helpful.

Could you please also clarify a bit on the below behavior, this does not involve any secrets. and Since I am removing everything from cluster, which includes CDS, LDS, should this not remove all related configs like RDS ?

I am trying to use gateway project and this installation guide - https://gateway.envoyproxy.io/latest/tasks/quickstart/

It just create resources like HTTPRoute , gateway and gatewayclass. These things gets converted to envoy proxy configurations. To apply these resources to k8s cluster - kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default Now, to remove all these configurations from k8s cluster and thus from envoy proxy, I am doing -

kubectl delete -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default

But to my surprise, envoy config does not get updated instead and it keeps showing the older config. I was debugging the go-control-plane server.go. and it seems it is correctly detecting which resources are to be deleted but envoy side there is effect.

few logs from go-control-plane, these are my logs -

 send getResources []
 send getRemovedResources [httproute/default/backend/rule/0]

[adisuissa]

Sorry, I'm not very familiar with this.
Can you post the Envoy trace logs (hopefully we'll see what updates Envoy receives)?

[pravinpushkar]

Sure , Let me provide the traces.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.