NSM interface deleting periodically

[denis-tingaikin]

Logs

stderr F Jun 17 10:50:48.232 [ERRO] [cmd:[/bin/app]] [healServer:processHeal] Failed to heal connection alpine-cl-0: Error returned from sdk/pkg/networkservice/common/authorize/authorizeClient.Request: rpc error: code = PermissionDenied desc = no sufficient privileges

Steps to reproduce

1. Run webhook example: https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/features/webhook
2. don't cleanup
3. wait for 20-30 min

Actual:

nsm-toggling.webm.zip

Expected:
NSM interface should not be deleted if data plane and control plane are fine

[denis-tingaikin]

Tested on kind. The issue is reproducible.

[denis-tingaikin]

Logs from my local running:

nsc-init.txt
nsc.txt

[denis-tingaikin]

@edwarnicke Can we consider this issue ASAP?

[edwarnicke]

@denis-tingaikin Yes

[denis-tingaikin]

Root cause:

1. Spire gives a certificate for 1h
2. NSM schedules refresh after 1h * 1/3
3. On refreshing spire updates certificates for all applications
4. At refresh request moment nsmgr has a new certificate from spire, but authInfo from gRPC keeps the old certificate from step1.
5. nsmgr updates token with new certificate
6. client can not validate the token from nsmgr because authInfo from gRPC keeps the old certificate from step1. (failure here)

Currently, I'm not found a good solution for this issue, started to look into gRPC source code.

[denis-tingaikin]

Tested today two workarounds:

1. https://golang.org/pkg/crypto/tls/#RenegotiationSupport -- it is not helped
2. remove connection caching in connect -- it is working

Still looking for other solutions.

[denis-tingaikin]

@edwarnicke

I've asked spire guys about the issue and got the next answer:

Andrew Harding 14 hours ago
This is expected. gRPC will reuse the existing connection when you issue RPCs unless you redial. Since no new TLS handshake takes place, the new client credential is never communicated.
white_check_mark
eyes
raised_hands

Andrew Harding 14 hours ago
The x509source returns a channel from Updated() that callers can use to know when the SVID has been updated so they can re-establish a connection with the new credential.

Question: Can we modify connect chain elements to wait for update SVID to make re-dial?

Note: we can just pass option to wait to channel to not depend on spire functions

[denis-tingaikin]

Currently we have the next options to fix the issue:

1. Do redial as suggested spire guys on svid updating NSM interface deleting periodically  #1929 (comment)
2. Remove policy last token signed.
3. Keep and use first certificates for client and server on token generating.
4. Your option.

For me option 1 looks good.

@edwarnicke Please share your thoughts on these options.

[edwarnicke]

I'm curious... are they saying that GRPC won't close existing connections that have a TLS certificate that has expired after the connection was established?

[denis-tingaikin]

Yes, as I got it, a handshake is doing once per dial.

[edwarnicke]

@denis-tingaikin It looks like we need to do something that involves option 1 above... but lets try to keep it simple and natural :)

[denis-tingaikin]

The root cause is fixed in networkservicemesh/sdk#1005

But I found that the issue can be reproduced via unstable healing. This reproducing periodically.
Tested a fix networkservicemesh/sdk#1005 without heal and it working fine in 100% cases.