How to debug rpc failures?

[GiantPluto]

I am trying to use the reclient with chromium where its already integrated. Modified the server and instance details to point it to the local BuildGrid server I have setup with no auth. There is no GCP involved here. On running the build. I am seeing the below failure. How can I debug RPC errors and what could be the reason?

rpc error: code = Unavailable desc = rpc error: code = Unavailable desc = retry budget exhausted (6 attempts): all SubConns are in TransientFailure, authentication type (identity) used="no authentication"

[gkousik]

If you need no auth between the RBE server and reproxy, you need to set RBE_service_no_auth=true when starting reproxy. Do you see this error message with that option set?

[GiantPluto]

service_no_auth=true is set in the config itself. I am getting the error with the same config

[gkousik]

Could you give more details about your server config too? Which RBE server implementation are you using? Can you share your reproxy.INFO / reproxy.WARNING / reproxy.ERROR and reproxy_outerr.log files?

[GiantPluto]

@gkousik Thanks for the response. Please find the attached zip containing all the logs you have asked and the reproxy.cfg .
REProxy_logs.zip

Btw I am using the BuildGrid

[gkousik]

Thanks! From the reclient config, I'm not seeing a lot that's amiss. The service_no_auth flag is getting passed in as it should be but the error message doesn't really tell me what's wrong.

I've never worked with Buildgrid before but from https://buildgrid.build/user/configuration.html#authorization-section, can you confirm that authorization is set to none in your server config?

If you can confirm that the server config is also correct, can you also disable TLS with RBE_service_no_security=true and see if that works? (https://github.com/bazelbuild/remote-apis-sdks/blob/042d9851eb2847d088094bba28b4ea065be83c27/go/pkg/flags/flags.go#L45)

[GiantPluto]

@gkousik In BuildGrid, authorization is not configured which defaults to none. I have had goma working before with these servers. Started facing this issue during the migration to reclient. I will try setting RBE_service_no_security=true and will update you. Could you let me know which RBE backend servers you have worked with?

[gkousik]

Hi @GiantPluto we test with a Google-internal RBE implementation which implements the RBE API specification in https://github.com/bazelbuild/remote-apis.

Did RBE_service_no_security work for your use case?

[ap4ss3rby]

I would like to chime in, as this issue also affected me while trying to build android using RBE. Referencing the settings the chromium kajiya reference server uses (referenced in the readme.md remote-apis) , I ended up with this config in a source script I use before sourcing build/envsetup. The compiler specific environment variables are irrelevent to the auth issue, but are included for reproducability. Afterwards, I was able to confirm that the CPU usage was not from native build tools but from bazel buildfarm itself. This appears to be due to autoauth, which is not documented whatsoever in the readme.md

export USE_RBE=1
export RBE_R8_EXEC_STRATEGY=remote_local_fallback
export RBE_CXX_EXEC_STRATEGY=remote_local_fallback
export RBE_D8_EXEC_STRATEGY=remote_local_fallback
export RBE_JAVAC_EXEC_STRATEGY=remote_local_fallback
export RBE_JAVAC=1
export RBE_R8=1
export RBE_D8=1
export RBE_service="localhost:8980"
export RBE_service_no_security=true
export RBE_service_no_auth=true
export RBE_automatic_auth=false