When LBs throw, we go into panic mode and the channel is permanently broken. Instead of throwing, AutoConfiguredLoadBalancerFactory could go into some failure mode where it fails RPCs, but still keeps the channel alive in case things improve.
However, some times there are throws scare me a lot, in that they are failing the channel due to invalid service config. Different logic is probably necessary there. Related discussion: starting at grpc/grpc#15896 (comment)