fix: health check exception with opentelemetry tracing interceptors #5392
Conversation
github-actions
bot
added
size/S
area/core
| @@ -60,6 +60,8 @@ def telemetry_wrapper(behavior, request_streaming, response_streaming): | |||
| return _wrap_rpc_behavior(next_handler, telemetry_wrapper) | |||
|
|
|||
| def _intercept_aio_server_unary(self, behavior, handler_call_details): | |||
| from jina.helper import iscoroutinefunction | |||
There was a problem hiding this comment.
Not needed if we decide to use the aio.HealthServicer approach. I was just experimenting if the interceptors can be easily modified to accept any method.
|
The latest commit (b6ddf2e) uses the hidden Although I'm not sure why this issue is masked when running the Flow locally and in integration tests without using docker. The other potential solution is to adapt the |
Codecov Report
@@ Coverage Diff @@
## master #5392 +/- ##
==========================================
- Coverage 86.94% 86.93% -0.02%
==========================================
Files 101 101
Lines 6611 6611
==========================================
- Hits 5748 5747 -1
- Misses 863 864 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
girishc13
force-pushed
the
debug-jinahub-docker
branch
from
356377b
to
771cae9
Compare
girishc13
changed the title
jina/serve/runtimes/asyncio.py
Outdated
| @@ -182,6 +182,7 @@ def is_ready(ctrl_address: str, **kwargs) -> bool: | |||
| :return: True if status is ready else False. | |||
| """ | |||
|
|
|||
| # print('--->address', ctrl_address) | |||
The
asynctracing interceptors expect the RPC stubs to be acoroutinefunction otherwise an exception is thrown for the RPC. When tracing was enabled, the health checks with docker or k8s containers failed silently and the Flow timed out eventually. The underlying issue was the usage of the defaultgrpc_health.HealthServicerimplementation with anasyncgrpc server.Goals:
aio.HealthServicerandawaitkeyword when calling the health servicer.