Stackdriver exception trying to publish metrics during shutdown #4353
Comments
|
FYI - locally I changed the two methods and do not get this exception anymore (I get a different one - something with header size too large but it seems to be a different issue): |
|
Is this happening all the time or occasionally? If the latter, how frequently? |
jonatan-ivanov
added
waiting for feedback
|
@jonatan-ivanov this happens only when the application is being shut down - e.g. a new revision is being deployed and all old revision instances are being shut down. The bigger impact is actually that it also happens in cloudrun jobs when they finish and as a result the metrics are not pushed when they finish. |
|
@jonatan-ivanov i changed the micrometer.version maven property in the reproducer to use a different micrometer version (and verified that it is being used) and it still fails both with 1.11.6 and with 1.12.0 |
|
Hey there, it seems we are facing the same issue. To get more ideas of what changes lead to the observed behaviour I will write this comment. We started to face this issue when we migrated from Springboot 2.7 to 3.1. def micrometer_version_pin = '1.10.6'
dependencies {
// [...]
implementation "io.micrometer:micrometer-commons:${micrometer_version_pin}!!"
implementation "io.micrometer:micrometer-core:${micrometer_version_pin}!!"
implementation "io.micrometer:micrometer-observation:${micrometer_version_pin}!!"
implementation "io.micrometer:micrometer-registry-stackdriver:${micrometer_version_pin}!!"
// [...]
} |
shakuzen
added
bug
Thank you for the details. That sounds like it was #3759, included in 1.10.7, that caused this to start happening, which makes sense. The proposed fix sounds reasonable. I'd like to see if there is any way we can effectively test the behavior locally with a stub of the stackdriver metrics service. That could help us avoid reintroducing this issue in the future and make sure it stays fixed. |
|
@shakuzen I could not find an emulator or anything remotely helpful to integration test it :( |
|
So for @Override
public void stop() {
if (client != null)
client.shutdownNow();
super.stop();
}to @Override
public void stop() {
if (client != null && !client.isShutdown())
client.shutdownNow();
super.stop();
}fix the problem? I understand that we're closing an already closed client? We would need to check if the metrics are still being sent on stop. Still no luck with providing some stub for Stackdriver metrics server? |
|
@marcingrzejszczak would that not mean that no metrics will be pushed upon shutdown? We do not want to get the exception, but we also want to push the metrics one last time upon shutdown. |
|
From my understanding the suggested change would shut down the client only if it hasn't been already shut down but maybe I'm missing sth. |
marcingrzejszczak
added
the
waiting for feedback
|
For me it looks like the most fundamental change is to move shutdown of the client from |
shakuzen
removed
the
waiting for feedback
|
Could folks affected by this issue please try with the latest snapshots to make sure the fix is working and not causing any other issues? Specifically, it's in 1.10.14-SNAPSHOT, 1.11.9-SNAPSHOT, and 1.12.3-SNAPSHOT available from the snapshot repo. |
|
1.12.3-SNAPSHOT -> Fix seems to work properly, metrics are being published and client is shutdown gracefully afterwards: No other issue recognized so far. |
|
I was a bit slow to respond - we have this fix already rolled out to multiple applications and it works. Thank you for merging it |
Describe the bug
Whenever an application is shut down, the stack driver registry tries to publish its metrics and throws an exception with the error UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Channel shutdown invoked
Environment
Google Cloud Run
Spring Boot 3.1.5
micrometer-registry-stackdriver 1.11.5
spring-cloud-gcp 4.8.2
To Reproduce
Expected behavior
Additional context
I have not debugged through the code. However, the io.micrometer.core.instrument.push.PushMeterRegistry#close seems to call stop(), publish(), close() when the context is closed. The io.micrometer.stackdriver.StackdriverMeterRegistry#stop calls client.shutdownNow() which I guess is not correct. Maybe this should be rather done in close() and not in stop(). Just a wild guess.
The text was updated successfully, but these errors were encountered: