Instructions on how to configure ALTS

[Alos]

There is a way you can configure Application Layer Transport Security as per: https://grpc.io/docs/languages/java/alts/#grpc-server-with-alts-transport-security-protocol

Is there a way to do this using Spring gRPC?

[dsyer]

The APIs are provided by grpc-java, and I don't think there's anything stopping you from using them, but with Spring gRPC you would need a custom GrpcChannelFactory (on the client) and a corresponding GrpcServerFactory (on the server). If you can make it work it might be a useful addition, but it might be better under Spring Cloud GCP (https://github.com/GoogleCloudPlatform/spring-cloud-gcp), if I understand correctly.

You can use regular mTLS via Spring Secuirity (maybe that isn't helpful if you don't control the client and the server).

[Alos]

Is the GrpcServerFactory a bean that is autowrired where I can configure the security properties? It's unclear to me how to create a server from the gRPC's AltsServerBuilder.

Alternatively, is there a way to grab the existing Server bean so that I can configure its security?

[dsyer]

I don't think the existing Server is going to be an ALTS one. It's not available as a bean either but you would find it by implementing your own GrpcServerFactory and registering that as a @Bean. If you do that I think you can just copy one of the existing implementations and use the AltsServerBuilder instead of the default (there's a DefaultGrpcServerFactory with a protected method you can override).

[Alos]

I see! Can you point me to one of the existing implementations so I can use as a reference?

I think I found them! Something like this:

spring-grpc/spring-grpc-core/src/main/java/org/springframework/grpc/server/DefaultGrpcServerFactory.java

Line 58 in 3fb4e2f

public class DefaultGrpcServerFactory<T extends ServerBuilder<T>> implements GrpcServerFactory {

[dsyer]

It looks to me like all you need to do is override the credentials() method in NettyGrpcServerFactory.

[Alos]

I'm actually not sure how to extend the NettyGrpcServerFactory and then instantiate and start the server.

Here is my attempt at extending it and creating a AltsEnabledNettyGrpcFactory. I tried to create a bean with it here.

The application boots up but then immediately terminates with no errors. I assume my bean is not magically picked up and the sever is never started.

[dsyer]

I see an error (logged as a WARN):

java.io.IOException: Your default credentials were not found. To set up Application Default Credentials for your environment, see https://cloud.google.com/docs/authentication/external/set-up-adc.

So your bean is being created and then the app fails to start because the server cannot be created when credentials are missing. I'm not in GCP, and I don't have access to a Google environment right now, so it's hard to test. I guess you do have access and therefore your error is different than mine.

Maybe you can run your test with "debug=true" and see if the autconfigurations you expect are triggered.

[Alos]

Aha! You are totally right! Let me run it in a real GKE cluster and see what happens!

[Alos]

When running in a GKE cluster, or with the correct permissions the app starts up and dies. The only error I see is related to a missing Context Storage.

 > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] java.lang.ClassNotFoundException: io.grpc.override.ContextStorageOverride
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source) ~[na:na]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ~[na:na]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at java.base/java.lang.Class.forName(Unknown Source) ~[na:na]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at java.base/java.lang.Class.forName(Unknown Source) ~[na:na]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.grpc.Context$LazyStorage.createStorage(Context.java:129) ~[grpc-api-1.76.0.jar:1.76.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.grpc.Context$LazyStorage.<clinit>(Context.java:118) ~[grpc-api-1.76.0.jar:1.76.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.grpc.Context.storage(Context.java:106) ~[grpc-api-1.76.0.jar:1.76.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.grpc.Context.current(Context.java:172) ~[grpc-api-1.76.0.jar:1.76.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.opencensus.trace.unsafe.ContextManagerImpl.currentContext(ContextManagerImpl.java:30) ~[opencensus-api-0.31.1.jar:0.31.1]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.opencensus.trace.unsafe.ContextHandleUtils.currentContext(ContextHandleUtils.java:56) ~[opencensus-api-0.31.1.jar:0.31.1]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.opencensus.trace.CurrentSpanUtils.getCurrentSpan(CurrentSpanUtils.java:37) ~[opencensus-api-0.31.1.jar:0.31.1]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at io.opencensus.trace.Tracer.spanBuilder(Tracer.java:308) ~[opencensus-api-0.31.1.jar:0.31.1]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:865) ~[google-http-client-2.0.2.jar:2.0.2]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.auth.oauth2.ComputeEngineCredentials.pingComputeEngineMetadata(ComputeEngineCredentials.java:586) ~[google-auth-library-oauth2-http-1.40.0.jar:1.40.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.auth.oauth2.ComputeEngineCredentials.isOnGce(ComputeEngineCredentials.java:532) ~[google-auth-library-oauth2-http-1.40.0.jar:1.40.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.auth.oauth2.DefaultCredentialsProvider.tryGetComputeCredentials(DefaultCredentialsProvider.java:314) ~[google-auth-library-oauth2-http-1.40.0.jar:1.40.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentialsUnsynchronized(DefaultCredentialsProvider.java:219) ~[google-auth-library-oauth2-http-1.40.0.jar:1.40.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:120) ~[google-auth-library-oauth2-http-1.40.0.jar:1.40.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:192) ~[google-auth-library-oauth2-http-1.40.0.jar:1.40.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:164) ~[google-auth-library-oauth2-http-1.40.0.jar:1.40.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.api.gax.core.GoogleCredentialsProvider.getCredentials(GoogleCredentialsProvider.java:70) ~[gax-2.71.0.jar:2.71.0]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.cloud.spring.core.DefaultCredentialsProvider.<init>(DefaultCredentialsProvider.java:101) ~[spring-cloud-gcp-core-7.4.1.jar:7.4.1]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.google.cloud.spring.autoconfigure.core.GcpContextAutoConfiguration.googleCredentials(GcpContextAutoConfiguration.java:56) ~[spring-cloud-gcp-autoconfigure-7.4.1.jar:7.4.1]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) ~[na:na]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.lambda$instantiate$0(SimpleInstantiationStrategy.java:172) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiateWithFactoryMethod(SimpleInstantiationStrategy.java:89) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:169) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:489) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1375) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1205) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:569) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:529) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:339) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:373) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:337) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.instantiateSingleton(DefaultListableBeanFactory.java:1228) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingleton(DefaultListableBeanFactory.java:1194) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:1130) ~[spring-beans-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:990) ~[spring-context-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:627) ~[spring-context-6.2.12.jar:6.2.12]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:752) ~[spring-boot-3.5.7.jar:3.5.7]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:439) ~[spring-boot-3.5.7.jar:3.5.7]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:318) ~[spring-boot-3.5.7.jar:3.5.7]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1361) ~[spring-boot-3.5.7.jar:3.5.7]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1350) ~[spring-boot-3.5.7.jar:3.5.7]
      > [grpc-demo-service-5d9f776764-dlgfd grpc-demo-service] 	at com.example.grpcdemo.GrpcDemoApplication.main(GrpcDemoApplication.java:23) ~[classes/:na]

I guess it might be related to all those nulls I passed in the constructor here. Tryin to understand more how should I correctly pass these values.

[Alos]

I started debugging and it looks like the ContextStorageOverride might be a red herring. It looks like the implementation falls back to a default (plus I also see this on the "regular" server without my changes). Looking to see if its because the Registered gRPC service: Simple does not happen with my implementation.

Update: Added registration of services here. Still the app just stops with no errors.

Update 2: Debugging through the start up sequence it looks like normally there's a GrpcServerLifecycle that starts the server, but it doesn't appear to be called when I add my own implementation.

Update 3: Got the app to run now with the following changes:

• I had to change the io.grpc.netty.shaded.io.grpc.netty
• I added a loop to register the services using the GrpcServiceDiscoverer and the GrpcServiceConfigurer
• I had to add a new GrpcServerLifecycle bean configured with my AltsEnabledNettyGrpcFactory like so:

   @Bean
   GrpcServerLifecycle altsServerLifecycle(AltsEnabledNettyGrpcFactory factory,
                                                    GrpcServerProperties properties, ApplicationEventPublisher eventPublisher) {
       return new GrpcServerLifecycle(factory, properties.getShutdownGracePeriod(), eventPublisher);
   }

Now I'm stuck trying to configure the health endpoints to work in GKE! But at least the server starts!

[dsyer]

The lifecycle should be there automatically (so that might be a bug). I'm sorry I should have said - it's not enough just to create a bean of type GrpcServerFactory - you need to copy the existing @Bean definition from GrpcServerFactoryConfigurations because it has some logic in it, including the registration of services.