xDS: Use fallback server credentials if no certificate_providers in bootstrap

[halvards]

What version of gRPC are you using?

v1.59.0

What version of Go are you using (go version)?

go1.20.8

What operating system (Linux, Windows, …) and version?

Linux, macOS

What did you do?

I have an xDS-enabled gRPC server, configured to use xDS server-side credentials, with insecure credentials as the fallback:

xdscredentials.NewServerCredentials(xdscredentials.ServerOptions{FallbackCreds: insecure.NewCredentials()})

What did you expect to see?

I expected the xDS-enabled gRPC server to use fallback credentials if the xDS bootstrap config did not contain any credential providers.

I'm aware that xDS server credentials are not required when using xDS, but I wanted the ability to switch to using the fallback credentials by only making changes to the gRPC xDS bootstrap config, without making source code changes.

This is the behavior of gRPC-Java as of v1.59.0 (and also in v1.57.0), and I expected gRPC-Go to behave the same way.

What did you see instead?

When there was no credential_providers section in the bootstrap config, the gRPC server failed to start with this error:

xds credentials are passed to the user, but certificate_providers config is missing in the bootstrap configuration

Caused by this section of code:

grpc-go/xds/server.go

Lines 138 to 147 in 7765221

	// If xds credentials were specified by the user, but bootstrap configs do
	// not contain any certificate provider configuration, it is better to fail
	// right now rather than failing when attempting to create certificate
	// providers after receiving an LDS response with security configuration.
	if s.xdsCredsInUse {
	if len(cfg.CertProviderConfigs) == 0 {
	xdsClientClose()
	return nil, fmt.Errorf("xds credentials are passed to the user, but certificate_providers config is missing in the bootstrap configuration")
	}
	}

Additional comments

I found these statements in gRFC A29 (https://github.com/grpc/proposal/blob/deaf1bcf248d1e48e83c470b00930cbd363fab6d/A29-xds-tls-security.md#programming-api):

"The fallback credentials are used in the following cases: [...] xDS is in use, but the control plane does not provide security configuration."

"Note that the fallback credentials is not used in case of errors. For example, if there is an error encountered while using the xDS provided security configuration a connection will be terminated rather than using the fallback credentials."

With the current implementation, no certificate providers in the bootstrap config is considered an error and we fail early, but no security configuration from the control plane, is not considered an error, and fallback credentials are used.

Regardless of the what constitutes an error, not using the fallback credentials in this situation violates the principle of least surprise IMO. If I have an xDS-enabled gRPC server set up to use xDS server credentials, and then decide to temporarily use insecure credentials instead (say, because of a problem with the PKI configuration), right now it is insufficient for me to update the xDS bootstrap config to remove the certificate provider block. I also need to make a source code change and recompile my binary.

In my gRPC server, in order to enable switching xDS server credentials on and off with only a bootstrap config change, I ended up replicating part of the bootstrap parsing to detect if credential providers were present in the bootstrap config:

https://github.com/GoogleCloudPlatform/solutions-workshops/blob/499cb0f90c47ce0c761ce94c8f41413f34a59f45/grpc-xds-workshop/greeter-go/pkg/config/xds.go#L39-L54

Having to replicate this parsing logic from an internal grpc-go package felt onerous, but the alternative of exposing a flag or controlling with an envvar would mean xDS server-side credential configuration would be controlled in two places (bootstrap and flag), and that seemed suboptimal.

gRPC-Java behavior

Note also the inconsistent behavior between the Go and Java implementations. gRPC-Java uses fallback credentials for xDS server credentials if the xDS bootstrap config does not contain any certificate_providers, because there won't be a DownstreamTlsContext in the Listener's filter chain:

https://github.com/grpc/grpc-java/blob/ae49d275b3fdc143e6f2203747c96fd94146439b/xds/src/main/java/io/grpc/xds/EnvoyServerProtoData.java#L224-L226

and

https://github.com/grpc/grpc-java/blob/ae49d275b3fdc143e6f2203747c96fd94146439b/xds/src/main/java/io/grpc/xds/internal/security/SecurityProtocolNegotiators.java#L288-L302

[easwars]

Thank you for the detailed bug report. I'm investigating this and consulting with other grpc language implementators to determine what should be the correct behavior in this case.

[easwars]

I had a discussion with other language implementors and will be making a change where I will remove this check. But grpc-go will not use fallback credentials if the certificate provider instance name specified by the management server is not found in the bootstrap configuration.

In fact, both C-core and Java also will NACK the resource under this circumstance. The gRFC does specify this in https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#cds-for-client-side-security: "If this field is not present, or if it specifies a certificate provider instance that is not configured in the bootstrap file, gRPC will NACK the CDS update."

And it says the same for LDS in https://github.com/grpc/proposal/blob/master/A29-xds-tls-security.md#lds-for-server-side-security.

If you think the behavior of grpc-java does not match the ones specified in the gRFC, please file a bug with them.

[halvards]

Thanks for looking into it @easwars!

Agree that unrecognized certificate providers in an LDS response should trigger a NACK. My scenario was LDS responses that didn't contain any certificate providers.

Am I correct in stating that if:

1. the xDS-enabled gRPC server is configured to use xDS server-side credentials, with insecure credentials as the fallback, AND
2. the bootstrap file does not contain any certificate providers, AND
3. the LDS response from the management server does not contain any certificate providers

then the insecure credentials specified as the fallback credentials will be used?

[easwars]

Yes, your statement is correct.